A sand mining behavior monitoring method, device, equipment and medium

Abstract

The present application discloses a method, device, equipment and medium for monitoring sand mining activities, which are used to detect illegal sand mining activities of ships and solve the problem of illegal sand mining in rivers. In the present application, the electronic device can detect ships in the water area image by one or more conditions, such as the length of time the ship stays in the current water area exceeds the set length of time, the water ripple characteristics of the ship meet the set water ripple characteristics, and the water surface color of the ship meets the set water surface color characteristics, to determine whether the ship in the monitoring screen has carried out illegal sand mining, thereby solving the problem of illegal sand mining in rivers. In addition, in the embodiments of the present application, the detection of sand mining activities is not only performed by distinguishing between ordinary ships and sand mining ships, but one or more detection conditions are provided for detection, which can improve the detection accuracy of illegal sand mining activities of ships.

Description

Technical Field

[0001] The present application relates to the field of video surveillance technology, and in particular to a method, device, equipment and medium for monitoring sand mining activities. Background Art

[0002] Rivers are important paths of the hydrological cycle on Earth. Water is a basic natural resource and a strategic economic resource, and is a controlling factor in ecology and the environment. Rivers are important carriers and components of water resources, and are channels for sediment, salts, and chemical elements to enter lakes and oceans.

[0003] As the demand for basic building materials such as sand continues to increase in sectors such as the construction and transportation industries, the phenomenon of indiscriminate sand mining has become increasingly serious. Sand and gravel in the river are important barriers to buffer river flow, conserve water sources, and protect embankments. Long-term sand mining has intensified the scouring of embankments and other water conservancy projects by river water, increased the risk of embankments and other flood control projects being destroyed during flood season, and posed a serious threat to the safety of life and property of people on both sides of the river.

[0004] Therefore, solving the problem of illegal sand mining in rivers and protecting the safety of river embankments and the status of water sources are issues that need to be addressed urgently. Summary of the invention

[0005] The present application provides a sand mining behavior monitoring method, device, equipment and medium to detect illegal sand mining behavior by ships and solve the problem of illegal sand mining in rivers.

[0006] The present application provides a method for monitoring sand mining behavior, the method comprising:

[0007] Performing target detection on the water area image to determine the ships in the water area image;

[0008] Determine whether the vessel meets the set conditions, where the set conditions include one or more of the following: the vessel stays in the current waters for longer than a set time, the water ripple characteristics of the vessel meet the set water ripple characteristics, and the water surface color of the vessel meets the set water surface color characteristics;

[0009] If yes, it is determined that the vessel in question has engaged in illegal sand mining.

[0010] Furthermore, the set condition includes that the water ripple characteristics of the vessel meet the set water ripple characteristics, and the vessel meets the set condition including:

[0011] The first water ripple vibration frequency within a first set range centered on the vessel exceeds the second water ripple vibration frequency within a second set range centered on the non-sand mining vessel, and the difference between the first water ripple vibration frequency and the second water ripple vibration frequency is greater than the set vibration frequency; and / or

[0012] The first water ripple vibration amplitude within the first setting range exceeds the second water ripple vibration amplitude within the second setting range, and the difference between the first water ripple vibration amplitude and the second water ripple vibration amplitude is greater than the set vibration amplitude.

[0013] Furthermore, the set condition includes that the water surface color of the vessel meets the set water surface color characteristics, and the vessel meets the set condition including:

[0014] The distance between the red, green, blue (RGB) and yellow (RGB) images of the water surface image within a first set range centered on the vessel exceeds a set distance.

[0015] Furthermore, the performing target detection on the water area image to determine the vessel in the water area image includes:

[0016] Perform target detection on the water image of the current frame and use detection boxes to mark all detected ships;

[0017] If the current frame of the water area image is the first frame of the image, all the detection frames are numbered in sequence according to a set order;

[0018] If the current frame of the water area image is not the first frame of the water area image, the detection frame in the current frame of the water area image is used to update the detection frame in the previous frame of the water area image of the current frame of the water area image; if there is a detection frame of a newly entered ship, the detection frame of the newly entered ship continues to be numbered based on the numbering of all detection frames in the previous frame of the water area image.

[0019] Furthermore, the performing target detection on the water area image to determine the vessel in the water area image includes:

[0020] If there are multiple overlapping detection frames, for the multiple overlapping detection frames, the number of each corresponding detection frame in the previous frame of water area image is retained.

[0021] Furthermore, before retaining the serial number of each detection frame corresponding to the previous frame of the water area image, the method further includes:

[0022] It is determined whether the area of ​​the overlapping region between the multiple detection frames exceeds a set area threshold. If so, a step of retaining the number of each detection frame corresponding to the previous frame of the water area image is performed.

[0023] Furthermore, after performing target detection on the water area image and determining the vessel in the water area image, the method further includes:

[0024] Determine whether the water area in the water area image prohibits sand mining ships from entering;

[0025] If yes, an alarm is triggered when the vessels in the water area image include a sand mining vessel.

[0026] Further, after determining that the vessel has engaged in illegal sand mining, the method further includes:

[0027] Obtain a detailed image of the vessel captured at a variable magnification.

[0028] On the other hand, the present application provides a sand mining behavior monitoring device, the device comprising:

[0029] A detection unit, used to perform target detection on the water area image and determine the ships in the water area image;

[0030] A judging unit, used to judge whether the vessel meets a set condition, wherein the set condition includes one or more of the following: the vessel stays in the current waters for a time period exceeding a set time period, the water ripple characteristics of the vessel meet the set water ripple characteristics, and the water surface color of the vessel meets the set water surface color characteristics;

[0031] A determination unit is used to determine that the vessel has engaged in illegal sand mining when the result of the judgment unit is yes.

[0032] Furthermore, the detection unit is specifically used to perform target detection on the current frame water image, and use detection frames to mark all detected ships; if the current frame water image is the first frame image, all the detection frames are numbered in sequence according to a set order; if the current frame water image is not the first frame image, the detection frame in the current frame water image is used to update the detection frame in the previous frame water image of the current frame water image; if there is a detection frame of a newly entered ship, the detection frame of the newly entered ship continues to be numbered based on the numbering of all the detection frames in the previous frame water image.

[0033] Furthermore, the detection unit is specifically configured to retain the serial number of each corresponding detection frame in the previous frame of the water area image for the multiple overlapping detection frames if there are multiple overlapping detection frames.

[0034] Furthermore, the detection unit is also used to determine whether the area of ​​the overlapping region between the multiple detection frames exceeds a set area threshold. If it exceeds, the step of retaining the number of each corresponding detection frame in the previous frame of the water area image is executed.

[0035] On the other hand, the present application provides an electronic device, which includes a processor and a memory, and the processor is used to implement the steps of any of the above-mentioned sand mining behavior monitoring methods when executing a computer program stored in the memory.

[0036] On the other hand, the present application provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps of any of the above-mentioned sand mining behavior monitoring methods.

[0037] Because in this application, the electronic device can detect the ships in the water area image by one or more conditions, such as the length of time the ship stays in the current water area exceeds the set time, the water ripple characteristics of the ship meet the set water ripple characteristics, and the water surface color of the ship meets the set water surface color characteristics, to determine whether the ship in the monitoring screen has carried out illegal sand mining, thereby solving the problem of illegal sand mining in the river. In addition, in the embodiment of the present application, the sand mining behavior is not only detected by distinguishing between ordinary ships and sand mining ships, but one or more detection conditions are provided for detection, which can improve the accuracy of detecting illegal sand mining behavior by ships. BRIEF DESCRIPTION OF THE DRAWINGS

[0038] In order to more clearly illustrate the technical solutions in the present application or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0039] Figure 1 A schematic diagram of a sand mining behavior monitoring process provided in some embodiments of the present application;

[0040] Figure 2 A schematic diagram of a water ripple image provided in some embodiments of the present application;

[0041] Figure 3 A schematic diagram of water ripple characteristics provided for some embodiments of the present application;

[0042] Figure 4 A schematic diagram of ship detection provided for some embodiments of the present application;

[0043] Figure 5 A schematic diagram of a sand mining behavior monitoring process provided in some embodiments of the present application;

[0044] Figure 6 A schematic diagram of the structure of a monitoring system provided in some embodiments of the present application;

[0045] Figure 7 A schematic diagram of the structure of a sand mining behavior monitoring device provided in some embodiments of the present application;

[0046] Figure 8 A schematic diagram of the structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0047] In order to make the purpose, technical solutions and advantages of the present application clearer, the present application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0048] In order to detect illegal sand mining by ships and solve the problem of illegal sand mining in rivers, the embodiments of the present application provide a sand mining behavior monitoring method, device, equipment and medium.

[0049] Embodiment 1:

[0050] Figure 1 A schematic diagram of a sand mining behavior monitoring process provided in an embodiment of the present application, the process includes the following steps:

[0051] S101: Performing target detection on the water area image to determine the ships in the water area image.

[0052] The sand mining behavior monitoring method provided in the embodiment of the present application is applied to an electronic device, which can be a server, or an image acquisition device or other device with sand mining behavior monitoring capabilities. The computing power of the server is stronger than that of the image acquisition device, so the electronic device is preferably a server, and in the embodiment of the present application, the electronic device is mainly used as a server for description. The server can communicate with the image acquisition device to obtain the water area image acquired by the image acquisition device.

[0053] After the image acquisition device is installed, it can capture images of the water area in the monitoring screen and the ships traveling on the water surface. Generally, the image acquisition device can be installed near the water area where river sand mining may occur. The water area image captured by the image acquisition device includes the water area in the monitoring screen and the ships traveling on the water surface (if any). In the embodiment of the present application, the case where there are ships on the water surface is mainly described.

[0054] The water image may include one or more ships, and the electronic device performs target detection on the water image, and can detect all ships in the water image. In some possible implementations, the electronic device can mark the detection frame of each ship in the water image, and can mark the number assigned to each ship, so as to facilitate the distinction of all detected ships and more accurately monitor the sand mining behavior of each ship.

[0055] Optionally, in S101, the electronic device may classify the detected vessel, for example, identifying whether the vessel is a sand mining vessel or a non-sand mining vessel (also referred to as an ordinary vessel or a normal vessel).

[0056] S102: Determine whether the vessel meets the set conditions, which include one or more of the following: the vessel stays in the current waters for longer than the set time, the water ripple characteristics of the vessel meet the set water ripple characteristics, and the water surface color of the vessel meets the set water surface color characteristics. If yes, proceed to S103, if not, proceed to S104.

[0057] In S102, a ship in a water area image is used for illustration. In the case where there are multiple ships in the water area image, the electronic device can detect whether each ship in the water area image meets the set conditions, determine whether the ship that meets the set conditions has carried out illegal sand mining, and determine whether the ship that does not meet the set conditions has not carried out illegal sand mining.

[0058] Some illegal sand mining ships stay on the water surface during the day and start to mine sand at night, or some illegal sand mining ships carry out illegal sand mining during the period of staying on the water surface. Therefore, through the stay time of the ship in the current waters, such ships can be targeted for detection. The specific value of the set time can be arbitrary and is not limited in the embodiment of the present application. For example, the set time can be 1 hour. For ships that stay in the current waters for more than the set time, an overtime warning mark can be added.

[0059] During the operation of illegal sand mining ships, their water ripple characteristics are different from those of ships sailing or staying normally (i.e., non-sand mining ships). Therefore, the water ripple characteristics of the ships can be used to determine whether illegal sand mining has occurred. For ships whose water ripple characteristics meet the set water ripple characteristics, water ripple warning marks can be added.

[0060] During the operation of illegal sand mining ships, the water surface color of the ships is different from that of non-sand mining ships. Therefore, the water surface color of the ships can be used to determine whether illegal sand mining has occurred. For ships whose water surface color meets the set water surface color characteristics, a water surface color warning mark can be added.

[0061] S103: It was determined that the vessel had engaged in illegal sand mining.

[0062] In one possible implementation, if one or more of a timeout warning mark, a water ripple warning mark, and a water surface color warning mark is added to a ship, it can be determined that the ship has engaged in illegal sand mining.

[0063] In addition, the electronic device can also trigger an alarm for ships that engage in illegal sand mining, such as informing relevant personnel that illegal sand mining activities exist (or that illegal sand mining ships exist). The alarm method is not limited in the embodiment of the present application. For example, the alarm can be sent by email or text message, or by sounding a whistle.

[0064] S104: Determine that the vessel is not engaged in illegal sand mining.

[0065] Electronic equipment may not trigger alarms for ships that are not engaging in illegal sand mining.

[0066] Because in this application, the electronic device can detect the ships in the water area image by one or more conditions, such as the length of time the ship stays in the current water area exceeds the set time, the water ripple characteristics of the ship meet the set water ripple characteristics, and the water surface color of the ship meets the set water surface color characteristics, to determine whether the ship in the monitoring screen has carried out illegal sand mining, thereby solving the problem of illegal sand mining in the river. In addition, in the embodiment of the present application, the sand mining behavior is not only detected by distinguishing between ordinary ships and sand mining ships, but one or more detection conditions are provided for detection, which can improve the accuracy of detecting illegal sand mining behavior by ships.

[0067] Embodiment 2:

[0068] In order to further improve the detection accuracy of illegal sand mining by ships, based on the above embodiments, in the embodiment of the present application, the setting conditions include that the water ripple characteristics of the ship meet the set water ripple characteristics, and the ship meets the set conditions including:

[0069] The first water ripple vibration frequency within a first set range centered on the vessel exceeds the second water ripple vibration frequency within a second set range centered on the non-sand mining vessel, and the difference between the first water ripple vibration frequency and the second water ripple vibration frequency is greater than the set vibration frequency; and / or

[0070] The first water ripple vibration amplitude within the first setting range exceeds the second water ripple vibration amplitude within the second setting range, and the difference between the first water ripple vibration amplitude and the second water ripple vibration amplitude is greater than the set vibration amplitude.

[0071] The water ripple characteristics may include the water ripple vibration frequency and / or the water ripple vibration amplitude.

[0072] The values ​​of the first setting range and the second setting range are not limited in the embodiment of the present application. The first setting range and the second setting range may be the same or different.

[0073] Here, the first set range centered on the ship is used as the detection area. For example, the water ripple image in the detection area is as follows: Figure 2 The electronic device can extract edge features of water ripples in the detection area, and the extracted water ripple features can be Figure 3As shown, the distance d1 between the two wave peaks can be used to represent the water ripple vibration frequency, and the size of d1 is inversely proportional to the water ripple vibration frequency. The smaller d1 is, the higher the water ripple vibration frequency is, and the larger d1 is, the lower the water ripple vibration frequency is. The distance d2 from the wave peak to the wave trough can be used to represent the water ripple vibration amplitude. The size of d2 is proportional to the water ripple vibration amplitude. The larger d2 is, the higher the water ripple vibration amplitude is, and the smaller d2 is, the lower the water ripple vibration amplitude is.

[0074] The electronic device stores water ripple characteristics within a second set range centered on the non-sand mining ship, such as the second water ripple vibration frequency and / or the second water ripple vibration amplitude. Generally speaking, the second water ripple vibration frequency and the second water ripple vibration amplitude of the non-sand mining ship are relatively low during normal navigation or during the stay process, while the first water ripple vibration frequency and the first water ripple vibration amplitude of the ship in the illegal sand mining operation are relatively high. Therefore, this feature can be used to detect the ship and combine the changes in the water ripple frequency and amplitude around the ship as one of the bases for judging whether illegal sand mining occurs.

[0075] When the first water ripple vibration frequency exceeds the second water ripple vibration frequency, and the difference between the first water ripple vibration frequency and the second water ripple vibration frequency is greater than the set vibration frequency, it means that the first water ripple vibration frequency is significantly higher than the second water ripple vibration frequency and exceeds a certain range, and a water ripple warning mark can be added to the ship. The specific value of the set vibration frequency is arbitrary and is not limited in the embodiments of the present application.

[0076] When the first water ripple vibration amplitude exceeds the second water ripple vibration amplitude, and the difference between the first water ripple vibration amplitude and the second water ripple vibration amplitude is greater than the set vibration amplitude, it means that the first water ripple vibration amplitude is significantly higher than the second water ripple vibration amplitude and exceeds a certain range, and a water ripple warning mark can be added to the ship. The specific value of the set vibration amplitude is arbitrary and is not limited in the embodiments of the present application.

[0077] In an embodiment of the present application, the electronic device can detect the ship based on water ripple characteristics such as water ripple vibration frequency and / or water ripple vibration amplitude, and can also reduce false alarms caused by irregular operations of sand mining ships, and can further improve the detection accuracy of illegal sand mining activities by ships.

[0078] Embodiment 3:

[0079] In order to further improve the detection accuracy of illegal sand mining by ships, based on the above embodiments, in the embodiment of the present application, the setting condition includes that the water surface color of the ship meets the set water surface color characteristics, and the ship meets the set conditions including:

[0080] The distance between the red, green, blue (RGB) of the water surface image and the yellow image RGB within the first set range centered on the ship exceeds a set distance.

[0081] The water surface color near non-sand mining ships is generally bluish, and when a ship stays on the water for a period of time, the water surface color changes from bluish to yellowish. It is possible that the ship is illegally mining sand, causing the riverbed sediment to be stirred up and the water color to turn yellow.

[0082] Here, the first set range centered on the ship is used as the detection area, and the RGB of the yellow image is (255, 255, 0). The distance between the RGB of the water surface image and the RGB of the yellow image within the first set range can be calculated according to the distance formula d = sqrt ((x1-x2)^2+(y1-y2)^2+(z1-z2)^2), where (x1, y1, z1) can be substituted for the RGB value of the water surface image, and (x2, y2, z2) can be substituted for the value of the yellow RGB.

[0083] When the distance between the RGB of the water surface image and the RGB of the yellow image exceeds the set distance, it means that the water color near the ship is more yellow and exceeds a certain color range, and a water surface color warning mark can be added to the ship. The specific value of the set distance is arbitrary and is not limited in the embodiments of the present application.

[0084] In a possible implementation, the electronic device may also store the water surface color within a second set range not centered on the sand mining ship. For example, the water surface color within the second set range is blue, and the RGB of the blue image is (0, 0, 255). The first electronic device may calculate the distance between the RGB of the water surface image and the RGB of the blue image within the first set range according to the above distance formula, where (x1, y1, z1) may be substituted for the RGB value of the water surface image, and (x2, y2, z2) may be substituted for the value of the blue RGB. If the distance between the RGB of the water surface image and the RGB of the yellow image is less than the distance between the RGB of the water surface image and the RGB of the blue image, it may be determined that the RGB of the water surface image is more yellowish. For example, the distance between the RGB of the water surface image and the RGB of the yellow image is 1000, and the distance between the RGB of the water surface image and the RGB of the blue image is 2000. 1000 is less than 2000, so the water surface image is more yellowish.

[0085] In an embodiment of the present application, the electronic device can detect ships based on the color of the water surface, reduce false alarms caused by irregular operations of sand mining ships, and further improve the accuracy of detecting illegal sand mining activities by ships.

[0086] Embodiment 4:

[0087] In order to further improve the accuracy of detecting illegal sand mining by ships, based on the above embodiments, in the embodiment of the present application, target detection is performed on the water image, and it is determined that the ships in the water image include:

[0088] Perform target detection on the water image of the current frame and use detection boxes to mark all detected ships;

[0089] If the current frame of water area image is the first frame of image, all detection frames are numbered in sequence according to the set order;

[0090] If the current frame of the water image is not the first frame, the detection frame in the current frame of the water image is used to update the detection frame in the previous frame of the water image; if there is a detection frame of a newly entered ship, the detection frame of the newly entered ship continues to be numbered based on the numbering of all detection frames in the previous frame of the water image.

[0091] When there are multiple ships traveling in the monitoring screen, since there is no characteristic information such as license plate and model to distinguish the ships, the multiple ships in the monitoring screen can be numbered separately, such as assigning an identity (ID) to the ship. In the actual monitoring scene, when the ships are staggered, or the image acquisition device zooms in and captures and returns to the preset point within a few seconds, each ship may have moved relative to the background. At this time, the ID may be confused or the ID may jump, resulting in an unstable situation such as incorrect identification of the ship. In view of this situation, each ship in the water image can be numbered separately.

[0092] When numbering each ship in the water area image, the detection frame of each ship can be numbered in sequence according to the set order. For example, the set order can be from left to right or from right to left in the monitoring screen. Figure 4 As shown, the electronic device can number the ships from left to right in the monitoring screen. All ship number ids are 1 to 5. The ship numbers are always valid while they are in the monitoring screen. For ships that have newly entered the monitoring screen, the electronic device can number the ship based on the existing ship numbers. For example, the ship that has newly entered the monitoring screen is assigned a number id of 6.

[0093] The vessel may be stationary or traveling on the water, so the image acquisition device can continuously acquire images of the water area in the monitoring screen.

[0094] If the current frame of the water area image is the first frame of the image, the electronic device may number all the detected detection frames in sequence according to a set order. For example, the electronic device numbers all the detected detection frames in sequence according to a set order. There may be multiple overlapping detection frames in the current frame of the water area image. For the multiple detection frames, the electronic device may number the multiple detection frames respectively. For example, when the area of ​​the overlapping region between the multiple detection frames (hereinafter referred to as the overlapping area) does not exceed the area threshold, the electronic device numbers the multiple detection frames respectively.

[0095] In the traditional target detection process, when the overlapping area of ​​the detection frames of two targets exceeds the set area threshold, the detection frames of the two targets will be displayed as one detection frame, and only the one detection frame will be identified, which may cause the ID to be confused or jump, which is not conducive to detecting whether illegal sand mining occurs by the ship. Therefore, in this embodiment, when the overlapping area of ​​multiple detection frames exceeds the set area threshold, the electronic device still numbers the multiple detection frames separately, that is, as long as the detection frames of the ship overlap, regardless of whether the overlapping area of ​​the multiple overlapping detection frames exceeds the area threshold, the electronic device will number the multiple detection frames separately, which is conducive to distinguishing the ships in the monitoring screen.

[0096] The vessel may be in a moving state. If the current frame of the water image is not the first frame of the image, the electronic device can determine the correspondence between the vessel in the current frame of the water image and the vessel in the previous frame of the water image (of the current frame of the water image). Therefore, the electronic device can use the detection frame in the current frame of the water image to update the detection frame in its previous frame of the water image. Based on the changes in the positions of the detection frames in the previous frame of the water image and the current frame of the water image, the driving trajectory of the vessel in the monitoring screen can be analyzed.

[0097] Based on the correspondence between the vessel in the current frame of the water image and the vessel in the previous frame of the water image, the electronic device can also number the corresponding detection frame in the current frame of the water image according to the number of the detection frame in the previous frame of the water image. In a possible implementation, for each detection frame in the current frame of the water image, the number of the corresponding detection frame in the previous frame of the water image is used to number the detection frame in the current frame of the water image, that is, in the current frame of the water image, the number of the corresponding detection frame in the previous frame of the water image is retained.

[0098] There may also be overlapping detection frames in the current frame of the water image. If there are multiple overlapping detection frames, the number of each corresponding detection frame in the previous frame of the water image is retained for the multiple overlapping detection frames. In some embodiments, when the area of ​​the overlapping region between multiple detection frames does not exceed the set area threshold, the detection frame in the current frame of the water image is used to update the detection frame in the previous frame of the water image of the current frame of the water image, or in other embodiments, when the area of ​​the overlapping region between multiple detection frames exceeds the set area threshold, the number of each corresponding detection frame in the previous frame of the water image is retained. The detection frame ID management strategy based on the overlapping area judgment can ensure the matching of the detection frame position and the target area (such as the area of ​​the ship in the water image) to a certain extent, and can also prevent missed detection caused by occlusion. In some other embodiments, during the driving process of a ship, regardless of whether it overlaps (or is occluded) with other ships, the number of the ship and the overlapping ship is retained unchanged. In other words, regardless of whether the overlapping area of ​​the detection frames of the two overlapping targets exceeds the set area threshold, the numbers of the detection frames of the two targets can be kept, and the ID will not be reallocated because of the overlap of the two ships. That is to say, in Figure 4 In the above example, no matter whether the overlapping area between the ship identified by id4 and the ship identified by id5 exceeds the set area threshold, the two ids id4 and id5 are kept unchanged.

[0099] In a possible implementation, the area of ​​the overlapping region between the detection frames can be represented by the numerical value of the overlapping area, or by the overlapping rate of the overlapping area. Accordingly, the set area threshold can be the numerical value of the area, or can be the set overlap rate threshold. The value of the set area threshold is arbitrary and is not limited in the embodiment of the present application. For example, the set area threshold is 90%.

[0100] If there is a new ship entering the monitoring screen, the electronic device can detect the detection frame of the new ship, and based on the numbers of all detection frames in the previous frame of the water area image, continue to number the detection frames of the new ship and assign a new ID to the new ship.

[0101] If there is a vessel that leaves the monitoring screen, the electronic device can delete the number and detection frame of the vessel that leaves.

[0102] It is worth noting that in the embodiment of the present application, the numbering is performed in an ascending numbering strategy. In actual application, there is no limitation on the numbering strategy, as long as the numbering can distinguish different ships.

[0103] The process is explained with a specific flow: 1) The electronic device assigns target IDs to all ships detected in the first frame image, and saves all detection frames of all ships. 2) The electronic device compares the detection frame detected in the second frame image with the detection frame detected in the previous frame (i.e., the first frame) image in sequence, and calculates the overlap rate between the detection frames. 3) The electronic device uses the detection frame of the second frame image to update the detection frame in the previous frame image. 4) The electronic device calculates the overlap rate of each ship, and when the overlap rate reaches the set overlap rate threshold, continues to use the ID originally assigned to each ship in the first frame image. 5) When a ship leaves the monitoring screen, the saved detection frame of the ship is deleted, and the ship is output to the outside as being in a deleted state.

[0104] Also Figure 4 For example, the overlap ratio of the overlapping area to the detection frame area of ​​the ship identified by id4 is ROI_id4=Siou / S4, where Siou is the overlapping area of ​​the ship identified by id4 and the ship identified by id5, and S4 is the detection frame area of ​​the ship identified by id4.

[0105] The overlap ratio of the overlapping area to the detection frame area of ​​the ship identified by id5 is ROI_id5=Siou / S5, where S5 is the detection frame area of ​​the ship identified by id5.

[0106] The overlapping area Siou of the ship identified by id4 and the ship identified by id5 is (Ret5_lu–Ret4_rl)*(Ret4_rl–Ret5_lu), where Ret5 is the rectangular detection box of the ship identified by id5, Ret4 is the rectangular detection box of the ship identified by id4, lu is the upper left (left up), that is, the upper left coordinate point of the rectangular detection box, and rl is the lower right (right low), that is, the lower right coordinate point of the rectangular detection box.

[0107] In this embodiment, the ship IDs are managed based on a specific ID management strategy to avoid ID confusion or jumps when the detection frames of the ships overlap. Therefore, the ships in the monitoring screen can be numbered stably and accurately, thereby further improving the detection accuracy of illegal sand mining activities by ships.

[0108] Embodiment 5:

[0109] On the basis of the above embodiments, in the embodiment of the present application, after performing target detection on the water image and determining the vessel in the water image, the method further includes:

[0110] Determine whether the water area in the water area image prohibits sand mining ships from entering;

[0111] If yes, an alarm is triggered when the vessels in the water area image include a sand mining vessel.

[0112] If the waters in the monitoring image prohibit sand mining ships from entering, that is, the waters in the monitoring image do not allow all types of ships to enter, the electronic device can identify whether the ship in the water image is a sand mining ship or a non-sand mining ship, and if the ships in the water image include sand mining ships, an alarm can be triggered. The alarm method may include sending an email, text message or whistle to issue an alarm.

[0113] In a possible implementation, if the water area in the monitoring image allows all types of ships to enter, the electronic device can also locate and capture a detailed image (i.e., a close-up image) of each ship in the monitoring image, so as to observe the details to detect whether the ship in the water area image meets the set conditions, and the detailed image can also be used as the collected evidence chain.

[0114] Optionally, after determining that the vessel has engaged in illegal sand mining, the method further includes: obtaining a detailed image of the vessel captured by zooming. Generally, the image acquisition device uses a small magnification to monitor the entire water surface at a preset point. Through zooming, detailed images of distant ships can be clearly captured, which is convenient for observing details and collecting evidence chains. The preset point refers to the (optimal) pan / tilt position set in combination with the monitoring scene and lens magnification after the image acquisition device is installed. For example, the electronic device can control the image acquisition device to zoom in and capture detailed images of the ship.

[0115] The above embodiments are described below with a specific embodiment. Figure 5 As shown, the following steps are included:

[0116] The electronic device acquires the image acquired by the image acquisition device in real time, performs target detection in the acquired image, determines the detection frames of all ships, and determines the stable ID and ship type of all detection frames (i.e., distinguishing between ordinary ships and sand mining ships) according to the above-mentioned ID management strategy.

[0117] The electronic device determines whether all types of vessels are allowed to enter the waters (i.e., whether sand mining vessels are not strictly prohibited from entering). If not, an alarm is triggered when a sand mining vessel is present in the image, and the real-time collected images continue to be acquired.

[0118] If yes, it means that there are no restrictions on ships in the waters, and long-term monitoring and detection can be carried out for all types of ships. The electronic equipment controls the image acquisition equipment to locate and capture a detailed image of each ship. Once illegal sand mining is detected, a chain of evidence can be formed. The electronic equipment performs water ripple feature detection, water surface color detection, and stay time detection on each ship. The electronic equipment determines whether an alarm is needed based on the comprehensive detection results. If not, it continues to obtain the real-time collected images. If yes, the electronic equipment controls the image acquisition equipment to capture a detailed image again to obtain the chain of evidence, and then triggers an alarm.

[0119] It can be seen that in the embodiments of the present application, targeted detection can be carried out on illegal sand mining ships disguised as ordinary ships and sand mining ships that dock on the water during the day and operate at night, and the combined ID management strategy can effectively improve the detection accuracy and alarm accuracy.

[0120] For example, the embodiments of the present application can be applied to Figure 6 In the monitoring system shown, the monitoring system includes a network camera, a monitoring box, a positioning module, a server and monitoring equipment.

[0121] The network camera is used as an image acquisition device to monitor the water surface and the sand mining ship in the monitoring screen. The positioning module, for example, a global positioning system (GPS) positioning module, can be used to locate the water area monitored by the network camera, and the monitoring box is used to power the network camera and the positioning module.

[0122] The network camera can communicate with the server through the Internet and send the collected water area images to the server. The server can use the sand mining behavior monitoring method improved by the above-mentioned embodiments of the present application to detect the sand mining behavior of the ship in the monitoring screen, and send the detection results to the monitoring device, so that relevant personnel can view the monitoring screen and the detection results through the application system installed on the monitoring device.

[0123] On the basis of the above-mentioned embodiments, based on the same technical concept as the method embodiment, the present application also provides a sand mining behavior monitoring device. Figure 7 A schematic diagram of the structure of a sand mining behavior monitoring device 700 provided in this application, such as Figure 7 As shown, the sand mining behavior monitoring device 700 includes:

[0124] The detection unit 701 is used to perform target detection on the water area image and determine the ships in the water area image;

[0125] The judging unit 702 is used to perform target detection on the water area image and determine the ships in the water area image;

[0126] The determination unit 703 is used to determine that the vessel has carried out illegal sand mining when the result of the judgment unit 702 is yes.

[0127] In a possible implementation manner, the set condition includes that the water ripple characteristics of the vessel meet the set water ripple characteristics, and the vessel meets the set condition including:

[0128] The first water ripple vibration frequency within a first set range centered on the vessel exceeds the second water ripple vibration frequency within a second set range centered on the non-sand mining vessel, and the difference between the first water ripple vibration frequency and the second water ripple vibration frequency is greater than the set vibration frequency; and / or

[0129] The first water ripple vibration amplitude within the first setting range exceeds the second water ripple vibration amplitude within the second setting range, and the difference between the first water ripple vibration amplitude and the second water ripple vibration amplitude is greater than the set vibration amplitude.

[0130] In a possible implementation, the set condition includes that the water surface color of the vessel meets the set water surface color characteristics, and the vessel meets the set condition including:

[0131] The distance between the red, green, blue (RGB) and yellow (RGB) images of the water surface image within a first set range centered on the ship exceeds a set distance.

[0132] In a possible implementation, the detection unit 701 is specifically used to perform target detection on the current frame water image, and use detection frames to mark all detected ships; if the current frame water image is the first frame image, all detection frames are numbered in sequence according to a set order; if the current frame water image is not the first frame image, the detection frame in the current frame water image is used to update the detection frame in the previous frame water image of the current frame water image; if there is a detection frame of a newly entered ship, the detection frame of the newly entered ship continues to be numbered based on the numbering of all detection frames in the previous frame water image.

[0133] In a possible implementation, the detection unit 701 is specifically configured to retain the serial number of each corresponding detection frame in the previous frame of the water area image for the multiple overlapping detection frames.

[0134] In a possible implementation, the detection unit 701 is further used to determine whether the area of ​​the overlapping region between multiple detection frames exceeds a set area threshold. If so, a step of retaining the number of each corresponding detection frame in the previous frame of the water area image is executed.

[0135] In a possible implementation, the device further includes:

[0136] The warning unit is used to determine whether the water area in the water area image prohibits sand mining ships from entering; if so, when the ships in the water area image include sand mining ships, an alarm is triggered.

[0137] In a possible implementation, the device further includes:

[0138] The acquisition unit is used to acquire the detailed image of the ship captured by zooming.

[0139] Figure 8 This is a schematic diagram of the structure of an electronic device provided by the present application. Based on the above embodiments, the present application embodiment further provides an electronic device, such as Figure 8 As shown, it includes: a processor 801, a communication interface 802, a memory 803 and a communication bus 804, wherein the processor 801, the communication interface 802, and the memory 803 communicate with each other through the communication bus 804;

[0140] The memory 803 stores a computer program. When the program is executed by the processor 801, the processor 801 performs the following steps:

[0141] Perform target detection on water images to identify ships in the water images;

[0142] Determine whether the vessel meets the set conditions, where the set conditions include one or more of the following: the vessel stays in the current waters for longer than the set time, the water ripple characteristics of the vessel meet the set water ripple characteristics, and the water surface color of the vessel meets the set water surface color characteristics;

[0143] If yes, it is determined that the vessel has engaged in illegal sand mining.

[0144] Furthermore, the set condition includes that the water ripple characteristics of the vessel meet the set water ripple characteristics, and the vessel meets the set condition including:

[0145] The first water ripple vibration frequency within a first set range centered on the vessel exceeds the second water ripple vibration frequency within a second set range centered on the non-sand mining vessel, and the difference between the first water ripple vibration frequency and the second water ripple vibration frequency is greater than the set vibration frequency; and / or

[0146] The first water ripple vibration amplitude within the first setting range exceeds the second water ripple vibration amplitude within the second setting range, and the difference between the first water ripple vibration amplitude and the second water ripple vibration amplitude is greater than the set vibration amplitude.

[0147] Furthermore, the set condition includes that the water surface color of the vessel meets the set water surface color characteristics, and the vessel meets the set condition including:

[0148] The distance between the red, green, blue (RGB) and yellow (RGB) images of the water surface image within a first set range centered on the ship exceeds a set distance.

[0149] Furthermore, the processor 801 is specifically configured to perform target detection on the water area image of the current frame, and mark all detected vessels with a detection frame;

[0150] If the current frame of water area image is the first frame of image, all detection frames are numbered in sequence according to the set order;

[0151] If the current frame of the water image is not the first frame, the detection frame in the current frame of the water image is used to update the detection frame in the previous frame of the water image; if there is a detection frame of a newly entered ship, the detection frame of the newly entered ship continues to be numbered based on the numbering of all detection frames in the previous frame of the water image.

[0152] Furthermore, the processor 801 is specifically configured to retain the serial number of each corresponding detection frame in the previous frame of the water area image for the multiple overlapping detection frames.

[0153] Furthermore, the processor 801 is also used to determine whether the area of ​​the overlapping region between multiple detection frames exceeds a set area threshold. If it exceeds, the step of retaining the number of each corresponding detection frame in the previous frame of the water area image is executed.

[0154] Furthermore, the processor 801 is also used to determine whether the water area in the water area image prohibits sand mining ships from entering; if so, when the ships in the water area image include sand mining ships, an alarm is triggered.

[0155] Furthermore, the processor 801 is also used to obtain a detailed image of the vessel captured by zooming.

[0156] The communication interface 802 is used for communication between the above electronic device and other devices.

[0157] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0158] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0159] Based on the above embodiments, an embodiment of the present application provides a computer-readable storage medium, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor implements the following steps when executing:

[0160] Perform target detection on water images to identify ships in the water images;

[0161] Determine whether the vessel meets the set conditions, where the set conditions include one or more of the following: the vessel stays in the current waters for longer than the set time, the water ripple characteristics of the vessel meet the set water ripple characteristics, and the water surface color of the vessel meets the set water surface color characteristics;

[0162] If yes, it is determined that the vessel has engaged in illegal sand mining.

[0163] In a possible implementation manner, the set condition includes that the water ripple characteristics of the vessel meet the set water ripple characteristics, and the vessel meets the set condition including:

[0164] The first water ripple vibration frequency within a first set range centered on the vessel exceeds the second water ripple vibration frequency within a second set range centered on the non-sand mining vessel, and the difference between the first water ripple vibration frequency and the second water ripple vibration frequency is greater than the set vibration frequency; and / or

[0165] The first water ripple vibration amplitude within the first setting range exceeds the second water ripple vibration amplitude within the second setting range, and the difference between the first water ripple vibration amplitude and the second water ripple vibration amplitude is greater than the set vibration amplitude.

[0166] In a possible implementation manner, the set condition includes that the water surface color of the vessel meets the set water surface color characteristics, and the vessel meets the set condition including:

[0167] The distance between the red, green, blue (RGB) and yellow (RGB) images of the water surface image within a first set range centered on the ship exceeds a set distance.

[0168] In a possible implementation, performing target detection on the water image to determine the vessel in the water image includes:

[0169] Perform target detection on the water image of the current frame and use detection boxes to mark all detected ships;

[0170] If the current frame of water area image is the first frame of image, all detection frames are numbered in sequence according to the set order;

[0171] If the current frame of the water image is not the first frame, the detection frame in the current frame of the water image is used to update the detection frame in the previous frame of the water image; if there is a detection frame of a newly entered ship, the detection frame of the newly entered ship continues to be numbered based on the numbering of all detection frames in the previous frame of the water image.

[0172] In a possible implementation, performing target detection on a water image to determine a vessel in the water image includes:

[0173] If there are multiple overlapping detection frames, for the multiple overlapping detection frames, the number of each corresponding detection frame in the previous frame of the water area image is retained.

[0174] In a possible implementation manner, before retaining the number of each detection frame corresponding to the previous frame of the water area image, the method further includes:

[0175] It is determined whether the area of ​​the overlapping region between the multiple detection frames exceeds the set area threshold. If it exceeds, the step of retaining the number of each detection frame corresponding to the previous frame of the water area image is executed.

[0176] In a possible implementation manner, after performing target detection on the water image and determining the vessel in the water image, the method further includes:

[0177] Determine whether the water area in the water area image prohibits sand mining ships from entering;

[0178] If yes, an alarm is triggered when the vessels in the water area image include a sand mining vessel.

[0179] In a possible implementation manner, after determining that the vessel has engaged in illegal sand mining, the method further includes:

[0180] Get detailed images of the ship captured at variable magnification.

[0181] Since the principle of solving the problem by the computer-readable medium provided above is similar to that of the sand mining behavior monitoring method, after the processor executes the computer program in the computer-readable medium above, the steps implemented can refer to the other embodiments above, and the repeated parts will not be repeated.

[0182] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0183] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0184] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0185] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0186] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A sand mining behavior monitoring method, characterized in that: The method comprises: Performing target detection on the water area image to determine the ships in the water area image; Determining whether the vessel meets a set condition, wherein the set condition at least includes that the water ripple characteristics of the vessel meet the set water ripple characteristics, and the set condition also includes one or more of the following: the vessel stays in the current waters for longer than a set time, and the water surface color of the vessel meets the set water surface color characteristics; If yes, it is determined that the vessel in question has engaged in illegal sand mining; The set condition includes that the water ripple characteristics of the vessel meet the set water ripple characteristics, and the vessel meets the set condition including: The first water ripple vibration frequency within the first setting range centered on the vessel exceeds the second water ripple vibration frequency within the second setting range centered on the non-sand mining vessel, and the difference between the first water ripple vibration frequency and the second water ripple vibration frequency is greater than the set vibration frequency; and / or the first water ripple vibration amplitude within the first setting range exceeds the second water ripple vibration amplitude within the second setting range, and the difference between the first water ripple vibration amplitude and the second water ripple vibration amplitude is greater than the set vibration amplitude.

2. The method according to claim 1, characterized in that The performing target detection on the water area image to determine the vessel in the water area image comprises: Perform target detection on the water image of the current frame and use detection boxes to mark all detected ships; If the current frame of the water area image is the first frame of the image, all the detection frames are numbered in sequence according to a set order; If the current frame of the water area image is not the first frame of the water area image, the detection frame in the current frame of the water area image is used to update the detection frame in the previous frame of the water area image of the current frame of the water area image; if there is a detection frame of a newly entered ship, the detection frame of the newly entered ship continues to be numbered based on the numbering of all detection frames in the previous frame of the water area image.

3. The method according to claim 1, characterized in that The performing target detection on the water area image to determine the vessel in the water area image comprises: If there are multiple overlapping detection frames, for the multiple overlapping detection frames, the number of each corresponding detection frame in the previous frame of water area image is retained.

4. The method according to claim 3, characterized in that Before retaining the number of each detection frame corresponding to the previous frame of the water area image, the method further includes: It is determined whether the area of ​​the overlapping region between the multiple detection frames exceeds a set area threshold. If so, a step of retaining the number of each detection frame corresponding to the previous frame of the water area image is performed.

5. The method according to claim 1, characterized in that After performing target detection on the water area image and determining the vessel in the water area image, the method further includes: Determine whether the water area in the water area image prohibits sand mining ships from entering; If yes, an alarm is triggered when the vessels in the water area image include a sand mining vessel.

6. The method according to claim 1, characterized in that After determining that the vessel has carried out illegal sand mining, the method further comprises: Obtain a detailed image of the vessel captured at a variable magnification.

7. A sand mining behavior monitoring device, characterized in that: The device comprises: A detection unit, used to perform target detection on the water area image and determine the ships in the water area image; A judging unit, configured to judge whether the vessel satisfies a set condition, wherein the set condition includes at least that the water ripple characteristics of the vessel meet the set water ripple characteristics, and the set condition also includes one or more of the following: the vessel stays in the current waters for a time period exceeding a set time period, and the water surface color of the vessel meets the set water surface color characteristics; a determination unit, configured to determine that the vessel has carried out illegal sand mining when the result of the judgment unit is yes; The set condition includes that the water ripple characteristics of the vessel meet the set water ripple characteristics, and the vessel meets the set condition including: The first water ripple vibration frequency within the first setting range centered on the vessel exceeds the second water ripple vibration frequency within the second setting range centered on the non-sand mining vessel, and the difference between the first water ripple vibration frequency and the second water ripple vibration frequency is greater than the set vibration frequency; and / or the first water ripple vibration amplitude within the first setting range exceeds the second water ripple vibration amplitude within the second setting range, and the difference between the first water ripple vibration amplitude and the second water ripple vibration amplitude is greater than the set vibration amplitude.

8. An electronic device, characterized in that: The electronic device includes a processor and a memory, and the processor is used to implement the steps of the sand mining behavior monitoring method as described in any one of claims 1-6 when executing the computer program stored in the memory.

9. A computer-readable storage medium, characterized in that: It stores a computer program, which, when executed by a processor, implements the steps of the sand mining behavior monitoring method described in any one of claims 1 to 6.

Images