Fishing behavior detection method under high-distance monitoring view angle, electronic equipment and storage medium

Abstract

The invention discloses a fishing behavior detection method under a high-distance monitoring view angle, electronic equipment and a storage medium, and the method comprises the steps: extracting an external rectangular region of a water surface region in an image, employing a sliding window to traverse an original image, and when the overlapping area of the sliding window and the external rectangular region of the water surface region exceeds a preset threshold value, executing the operation; and segmenting an image area in the sliding window, performing personnel detection, performing merging and redundancy elimination on detected target detection frames to obtain a personnel detection result, performing distance calculation on the personnel detection result and the edge contour of the water surface area, and if the distance exceeds a preset threshold, considering that the current personnel has a fishing behavior and performing early warning. According to the method, human participation is not needed, the problems of difficulty in personnel detection under a high and far monitoring view angle, multiple application scenes, time and labor waste in manual division of the water surface area and inaccuracy are solved, and compared with a scheme of directly adopting a sliding window to segment a whole image for detection, the detection time is shortened by about 50%, and the detection efficiency is improved. And the accuracy of the logic judgment of the abnormal fishing behavior of the personnel is ensured.

Description

Technical Field

[0001] The present invention belongs to the technical field of image processing, and is a method for detecting fishing behavior under a high-distance monitoring viewing angle, an electronic device and a storage medium. Background Art

[0002] Substations and transmission lines are usually surrounded by water bodies such as rivers, lakes, and reservoirs, which are gathering places for fishing enthusiasts. Therefore, the water surface area around the substation needs to be monitored. If there are people around the water area, it is necessary to determine whether the people are fishing and issue an early warning. However, the cameras in these scenes are generally located high and far away, and the proportion of human pixels in the images taken by the cameras is extremely small, which is a small target. Therefore, whether it relies on manual recognition of whether there are people with abnormal fishing behavior in the image or deep learning algorithms directly detect people in the image, the effect is extremely poor and seriously affected by background interference.

[0003] There are some existing methods for detecting small targets under high-distance monitoring perspectives: cropping the original image, detecting the cropped sub-images and merging the detection results of the sub-images. Although such methods increase the accuracy of small target detection under high-distance monitoring perspectives and provide technical support for the logical judgment of whether there is fishing behavior in the subsequent personnel, such methods require detection of each cropped sub-image, which increases the additional computational cost compared to directly detecting the original image. At the same time, since all areas in the image are covered, the redundant information in the output results is highly disturbed, and the algorithm cannot effectively focus on the water surface and its surrounding areas that need to be paid attention to, thereby increasing the workload of manual verification of warnings. Therefore, it is necessary to screen unnecessary areas to avoid invalid warnings. The existing technology can separate the water surface and its surrounding areas by manually dividing the ROI area, and then crop and detect them, but due to the irregularity of the water surface contour, it is still a challenge to select the appropriate segmentation position and size. Dividing the ROI area too large may introduce redundant information and interfere with subsequent processing; while segmenting too small may lead to missed detection. In addition, this method requires manual calibration of the ROI area for each scene and the operation of dividing the ROI area lacks flexibility. If it needs to be re-divided, it will add additional workload. Therefore, it is necessary to develop a small target detection method based on images taken from a high-distance monitoring perspective, focusing on the water surface and its surrounding areas and being able to flexibly adjust the ROI area according to specific needs, with high accuracy and low computational cost for abnormal fishing behavior detection. Summary of the invention

[0004] The problem to be solved by the present invention is that the existing cameras around the substation are located at a high and far distance, and the detection effect of the original image is extremely poor, which is not conducive to the subsequent logical judgment of the abnormal fishing behavior of the personnel detected in the image. Although the existing small target detection method improves the accuracy of small target detection by cropping the original image, detecting the cropped sub-image and merging the detection results of the sub-image, it is necessary to detect each cropped sub-image, which increases the additional calculation cost. At the same time, the method covers all areas in the image, and cannot effectively focus the attention of the algorithm on the water surface and its surrounding areas that need to be paid attention to. Although the artificial division of the ROI area can achieve the purpose of screening unnecessary areas, it is also impossible to select the appropriate segmentation position and size, and the manual calibration of the ROI area is time-consuming and laborious, and the re-division will increase the additional workload. Under the premise that it is impossible to directly judge whether there is abnormal fishing behavior by detecting tools such as fishing rods, a method for detecting abnormal fishing behavior of personnel that focuses on the water surface and its surrounding areas, can flexibly adjust the ROI area according to specific needs, and takes into account accuracy and low calculation cost is needed.

[0005] The technical solution of the present invention is: a method for detecting fishing behavior under a high-distance monitoring perspective, comprising the following steps:

[0006] Step 1: Construct a binary image dataset of water surface areas in different scenes and obtain the network weights of the image semantic segmentation network through training;

[0007] Step 2: Build a personnel data set and obtain the network weights of the target detection network with global perception capabilities through training;

[0008] Step 3: Get the image taken by the surveillance camera, establish a rectangular coordinate system with the upper left corner of the image as the coordinate origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis;

[0009] Step 4: Use the image semantic segmentation network to detect the water surface area in the image, obtain the edge contour of the water surface area and the set of coordinates of the edge points of the water surface area [(x 0 ,x 1 ,……x k ), (y 0 ,u 1 ,……y k )], get the maximum coordinate value of the edge point (x water_min ,x water_max ) and (y water_min ,y water_max ), thereby deriving the maximum enclosing rectangle of the edge contour of the water surface area and saving the coordinates of the upper left corner of the enclosing rectangle (x water_min ,y water_min ) and the lower right corner coordinate (x water_max ,ywater_max );

[0010] Step 5: Use the sliding window strategy to traverse the original image and cut it. Before cutting, calculate the overlapping area S of the maximum circumscribed rectangle of the original image area in the sliding window and the edge contour of the water surface area derived in Step 4. overlap , if S overlap If it is greater than the set threshold, it means that the area of ​​the sliding window contains the water surface and the area adjacent to the water surface that needs to be paid attention to. The original image area in the sliding window is cut and saved. Otherwise, it means that the area in the sliding window is not adjacent to the water surface and is a non-focus area. The sliding window moves and judges the next area.

[0011] Step 6: Use the target detection network with global perception capability to detect people in the sub-image area cut from the original image in Step 5, save the detection results of each sub-image, and obtain the target detection frame of the people in the sub-image area by merging and eliminating redundancies of the detection results, and save the coordinates of the detection frame:

[0012] [(x 0_min ,y 0_min ,x 0_max ,y 0_max ), (x 1_min ,y 1_min ,x 1_max ,y 1_max )…(x k_min ,y k_min ,x k_max ,y k_max )];

[0013] Step 7: Calculate the local minimum center point coordinates of each detected person using the detection box saved in Step 6:

[0014] x k_minimum_center =(x k_min +x k_max ) / 2

[0015] y k_minimum_center =y k_max

[0016] Calculate the local minimum center point (x k_minimum_center ,y k_minimum_center ) to the shortest distance w from the edge contour of the water surface area obtained in Step 4 minimum , if w minimum If the value is less than or equal to the set threshold, the target is considered to be suspected of phishing, an early warning is issued, and the target detection frame is displayed in the original image, waiting for further manual verification; if w minimumIf it is greater than the set threshold, it is considered that there is no phishing behavior on the target, and no warning or detection result display will be issued;

[0017] Step 8: Get the image captured by the surveillance camera and repeat the operations of Step 3-7 to achieve continuous detection of whether there are people engaging in abnormal fishing behavior under a high-angle surveillance perspective.

[0018] The present invention also provides an electronic device, comprising a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the above-mentioned fishing behavior detection method under the high-distance monitoring perspective.

[0019] The present invention also provides a computer-readable storage medium, in which at least one instruction or at least one program is stored. When the at least one instruction or at least one program is executed, the above-mentioned fishing behavior detection method under the high-distance monitoring perspective is implemented.

[0020] The present invention uses an improved Deeplabv3+ network to extract the edge contour of the water surface area in the image captured by the surveillance camera and derive the maximum circumscribed rectangle of the water surface area. On this basis, a sliding window is used to traverse the original image. If and only if the overlapping area of ​​the image area in the sliding window and the maximum circumscribed rectangle of the water surface area exceeds a preset threshold, it indicates that the area in the sliding window is adjacent to the water surface. If there is a person who may be fishing, the image area in the sliding window will be segmented and saved, and a target detection network with global perception capability will be used to perform personnel detection on the segmented and saved area. The improved IOU calculation method minIOU is used to realize the merging and redundant elimination of the personnel target detection frame detected in each sub-image area. Finally, the final personnel detection result is calculated with the edge contour of the pre-extracted water surface area. If it exceeds the preset threshold, it is considered that the current person is suspected of fishing behavior, and an early warning is issued and the target detection frame is displayed in the original image, waiting for further manual verification. The present invention does not require human intervention, and solves the problems of difficulty in detecting small target personnel under high-distance monitoring perspective, multiple application scenarios, complex scenarios, and time-consuming, labor-intensive and inaccurate manual division of water surface areas. Compared with direct detection of the results of sliding window cutting, the present invention shortens the detection time by about 50% while ensuring the accuracy of the logical judgment of abnormal fishing behavior of personnel.

[0021] Compared with the prior art, the present invention has the following advantages:

[0022] (1) Under the premise that there are a large number of scenes to be monitored and the water surface areas in different scenes to be monitored are quite different, compared with the artificial division of ROI areas, the water surface area can be adaptively segmented according to the scene captured by the camera, and the range of the area around the water surface to be monitored can be flexibly adjusted according to the segmentation result. Since the segmentation result directly affects the accuracy of subsequent processing, the present invention improves the network structure of Deeplabv3+, improves the network's ability to extract features, and makes full use of the channel information and spatial information of the feature map to ensure the accuracy of water surface and edge area segmentation.

[0023] (2) Suitable for high- and far-field monitoring perspectives. By introducing an improved image semantic segmentation algorithm, the semantic information in the image is fully utilized to optimize the traditional small target detection method of segmenting the original image and detecting it while ensuring segmentation accuracy. It only needs to cut, save and detect the sub-images that include the water surface and the area around the water surface that need to be focused on, which shortens the detection time by about 50% and greatly reduces the computing cost. It solves the difficulty of detecting small targets under high- and far-field monitoring perspectives while avoiding the detection of sub-images after the entire image is segmented.

[0024] (3) Under the premise that the proportion of human pixels in the image is small and it is impossible to determine whether there is fishing behavior through tools such as fishing rods, the present invention can flexibly configure the detection standard according to the specific scenario, and has wide applicability to various monitoring scenarios, ensuring that the targets finally output to the detection frame of the original image only include targets suspected of fishing behavior that require further manual verification, effectively filtering out invalid warnings and reducing the workload of subsequent manual verification. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Figure 1 The present invention is a flow chart of the method for detecting fishing behavior under a high-altitude monitoring perspective.

[0026] Figure 2 The network structure diagram of the improved MobilNetv2 of the present invention, (a) is the improvement of the inverted residual module with residual edges, and (b) is the improvement of the inverted residual module without using residual edges.

[0027] Figure 3 This is the structure diagram of the Deeplabv3+ model improved by the present invention.

[0028] Figure 4 A schematic diagram of a monitoring scenario applied in an embodiment of the present invention.

[0029] Figure 5 A schematic diagram of the maximum circumscribed rectangle of the water surface area is derived by using the improved Deeplabv3+ network to extract the edge contour of the water surface area in the scene.

[0030] Figure 6This is a schematic diagram of a sub-image including a water surface and adjacent areas obtained by sliding cropping in an embodiment of the present invention.

[0031] Figure 7 This is the final detection result of the embodiment of the present invention. Specific implementation methods

[0032] The present invention proposes a method for detecting fishing behavior under a high-distance monitoring perspective. The edge contour of the water surface area is adaptively extracted according to the image taken by the camera and the maximum circumscribed rectangle of the water surface area is derived. The sub-image containing the water surface and the surrounding area in the image is cut with a sliding window and detected. The person target detection frames detected in each sub-image area are merged and redundantly eliminated. The distance between the final person detection result and the edge contour of the water surface area extracted in advance is calculated. This method can better distinguish, display and warn whether a person in the high-distance monitoring perspective has fishing behavior.

[0033] The present invention relates to deep learning, image semantic segmentation algorithm and target detection algorithm with global perception capability. The overall steps are as follows: Figure 1 As shown, the specific implementation process is as follows.

[0034] Step 1: Construct a dataset of binary images of water surface areas in different scenes, mark the edges of the binary images of water surface areas in different scenes by tracing and drawing points, create training sets and test sets, train the image semantic segmentation network, and obtain the network weights of the image semantic segmentation network.

[0035] As a preferred embodiment of the present invention, the image semantic segmentation network adopts the improved Deeplabv3+, including improvements to the backbone network and improvements to the decoder part.

[0036] The Backbone part of the DeepLabv3+ network is improved as follows: As the Backbone of the DeepLabv3+ network, the MobileNetV2 network is composed of stacked inverted residual modules, such as Figure 2As shown in (a) and (b), for the inverted residual module using residual edges, that is, the inverted residual module with shortcut connection, the CBAM attention module is introduced after the depthwise separable convolution of the original branch; for the inverted residual module that does not use residual edges, three branches are expanded on the basis of the original branches to obtain four branches, and the four branches have receptive fields of different scales: among the three expanded branches, one branch uses 1×1 convolution, and the other two branches, based on the original branch structure, set the boundary padding and receptive field expansion factor of the depthwise separable convolution to 2 or 3 at the same time, that is, the boundary padding and receptive field expansion factor of the depthwise separable convolution of one branch are both 2, and the boundary padding and receptive field expansion factor of the depthwise separable convolution of the other branch are both 3. The four branches are integrated, and the features of different scales extracted by the receptive fields of different scales are fused as the output of the inverted residual module that does not use residual edges. The features extracted by the improved Backbone are richer in spatial scale than those extracted by the original Backbone, which is beneficial to the subsequent pixel classification in the image.

[0037] In the Deeplabv3+ network structure, only one high-level feature and low-level feature with the same size after processing are fused. The feature extraction of the fused feature map is insufficient, and it contains few edge details, and the boundary information of the water surface area cannot be accurately segmented. In addition, the decoder structure is simple and lacks low-level feature information. Therefore, the present invention makes the following improvements to the decoder part of the Deeplabv3+ network: Deeplabv3+ only inputs the feature map output by the first 4 layers of the MobileNetv2 network into the decoder as a shallow feature map. The present invention passes the two feature maps output by the first 6 layers and the first 14 layers of the MobileNetv2 network through the CBAM attention module, adjusts the size of the feature map, and splices them with the feature map output by the first 4 layers of the MobileNetv2 network as a new shallow feature map, which is sent to the decoder, focusing on learning the water surface information, so that the model can better restore the edge detail information of the target; the original deep feature map obtained by the MobileNetv2 network and the ASPP module of the Deeplabv3+ network is passed through the CBAM attention module as a new deep feature map, and the new deep feature map is upsampled and sent to the decoder. The CBAM attention module adds weight information to each feature channel and makes full use of the channel information and spatial information of the feature map. It focuses on the edge area of ​​the water surface on the basis of obtaining the overall features, reduces the transmission of useless information, and improves the network feature extraction capability.

[0038] Finally, the shallow and deep feature maps fed into the decoder are fused to increase feature information, such as Figure 3The improved Deeplabv3+ model shown in the figure, by comprehensively utilizing feature information at different levels and depths, the improved image semantic segmentation network of the present invention can more accurately identify the boundaries and details of the water surface area, reduce the mis-segmentation problem caused by factors such as lighting changes and weather conditions, ensure the accuracy of water surface area segmentation, and has strong robustness.

[0039] Step 2: Construct a personnel data set, and obtain the network weights of the target detection network with global perception capabilities through training. As a preferred method, after the image is processed by data enhancement methods such as flipping and rotating, the people in the image are marked as positive samples. Since the camera position of the image is relatively high and far away, the proportion of human pixels is small, which is a small target. In order to reduce the interference of the background on personnel detection, the sub-image of the cropped part of the original image that only contains the background class is introduced as a negative sample, and a training set and a test set are made to train the target detection network. The present invention preferably uses the end-to-end target detection framework DETR as the target detection network.

[0040] Step 3: Get the image taken by the surveillance camera, establish a rectangular coordinate system with the upper left corner of the image as the coordinate origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis. Figure 4 This is a surveillance image obtained by an embodiment of the present invention.

[0041] Step 4: Use the image semantic segmentation network to detect the water surface area in the image, obtain the edge contour of the water surface area and the set of coordinates of the edge points of the water surface area [(x 0 ,x 1 ,……x k ), (y 0 ,y 1 ,……y k )], get the maximum coordinate value of the edge point: from (x 0 ,x 1 ,……x k ) is derived from (x water_min ,x water_max ), from (y 0 ,y 1 ,……y k ) is derived from (y water_min ,y water_xax ), thereby deriving the maximum enclosing rectangle of the edge contour of the water surface area and saving the coordinates of the upper left corner of the enclosing rectangle (x water_min ,y water_min ) and the lower right corner coordinate (x water_xax ,y water_xax ), the edge contour of the water surface area and its maximum circumscribed rectangle are as follows Figure 5 shown.

[0042] Step 5: Use the sliding window strategy to traverse the original image and cut it into multiple sub-images using the sliding window cropping method, such as Figure 6 As shown, the sub-image overlap rate is generally set to no less than 50%. The overlapping area S of the maximum circumscribed rectangle of the original image area in the sliding window and the edge contour of the water surface area derived in Step 4 is calculated. overlap , if S overlap If S is greater than the set threshold, it means that the area in the sliding window contains the water surface and the area adjacent to the water surface. If there are people in it, there may be fishing behavior, which needs to be paid special attention to. The original image area in the sliding window at this time is cut and saved; if S overlap If the value is smaller than the set threshold, it indicates that the area is far away from the water surface in the scene and is a background area where it is impossible for fishermen to appear. There is no need to cut and save the original image area in the sliding window. The sliding window moves to judge the next area.

[0043] The implementation of step 5 is described below through an example.

[0044] Step 5.1: Set the number of rows to 4, the number of columns to col to 6, and the length of the original image to Image Height 、WidthImage Width , the overlap ratio between the sliced ​​subgraphs of the sliding window operation Overlap=0.5, and the length of the sliding window Window Height , Width Window Width .

[0045] Step 5.2: According to the length of the original image Height 、WidthImage Width , the overlap ratio Overlap between the cut subgraphs, the length of the sliding window Window Height , Width Window Width , calculate the coordinate set of all points that the coordinates of the upper left corner of the sliding window will traverse in the original image:

[0046] [(x 0 ,y 0 ),(x 1 ,y 0 )…(x 5 ,y 0 ),(x 0 ,y 1 ),(x 1 ,y 1 )…(x 5 ,y 1 )…(x 0 ,y 3 ),(x 1 ,y3 )…(x 5 ,y 3 )]

[0047] Step 5.3: Take out (x p ,y q ), through (x p ,y q ) deduce (x p ',y q '):

[0048] x p '=x p +Window Width

[0049] y q '=y q +Window Height

[0050] (x p ,y q ), (x p ',y q ') represent the coordinates of the upper left corner and the lower right corner of the sliding window respectively, and the original image area within the sliding window is cropped. In this embodiment, the original image is pre-divided into 4 rows and 6 columns, with a total of 24 sub-images. p, q represent the current traversal of the sliding window to the pth row and the qth column. The coordinates of the sliding window are compared with the coordinates of the maximum circumscribed rectangle (x water_min ,y water_min ,x water_max ,y water_max ), calculate the overlapping area of ​​the maximum circumscribed rectangle of the original image area in the current sliding window and the edge contour of the water surface area. The area is calculated as follows:

[0051] S overlap =(min(x water_max ,x p ')-max(x water_min ,x p ))*(min(y water_max ,y p ')-max(y water_min ,y p ))

[0052] Step 5.4: Determine S overlap and set the threshold value. In this embodiment, the threshold value is set to 0. If S overlap >0, the area in the sliding window is considered to be adjacent to the water surface and needs to be paid special attention to. The original image area in the sliding window is cut and saved. overlap≤0, it is considered that the area is far away from the water surface in the scene and is a background area where it is impossible for fishermen to appear. The sliding window moves to judge the next area.

[0053] The present invention believes that only people who appear around the monitored waters are likely to engage in fishing behavior. By adjusting the corresponding S in Step 5.4 overlap The threshold size is set to adjust the selection judgment of the focus area, and the sub-image containing the area adjacent to the water surface is cut and saved, so as to pre-exclude the people who are not around the water surface area in the image.

[0054] Step 6: Use the target detection network with global perception capability to detect people in the sub-image area cut from the original image in Step 5, save the detection results of each sub-image, merge and redundantly remove the target detection frames for the same person detected in each area and save the coordinates of the detection frames:

[0055] [(x 0_min ,y 0_min ,x 0_max ,y 0_max ), (x 1_min ,y 1_min ,x 1_max ,y 1_max )…(x k_min ,y k_min ,x k_max ,y k_max )]

[0056] Step 6.1: Use a target detection network with global perception capabilities to perform personnel detection on the sub-image area cut from the original image in Step 5. According to the number of rows and columns in the 24 sub-images into which the original image is pre-divided, convert the relative coordinates of the person detection frame in the sub-image into absolute coordinates in the original image and save them.

[0057] Step 6.2: Sort the detection frames stored in step 6.1 in descending order according to the confidence level. Take the detection frame with the largest confidence level each time as the detection frame to be matched. Calculate the iou between the detection frame to be matched and all other detection frames using the improved method of calculating iou minIOU. The iou calculation formula is as follows:

[0058]

[0059] Where S A Represents the area of ​​the detection box to be matched, S BIndicates the area of ​​the detection frame that calculates the iou of all other detection frames and the detection frame to be matched. Determine the size of the iou and the set threshold. If it is greater than the threshold, it is considered that the current detection frame matches the detection frame to be matched and is for the same target. The detection frame to be matched and the detection frame that matches it are stored as a set of matching pairs, and the matched detection frames are removed from the list of stored detection frames, otherwise it is considered that the current two detection frames are detection frames for two different targets. Finally, repeat the operation of Step 6.2 from the remaining detection frames until all the detection frames stored in Step 6.1 are stored in the form of matching pairs. If there is a detection frame to be matched that does not have a matching detection frame, it is considered that the probability of a false detection background class is high and it is directly removed.

[0060] Step 6.3: Merge and remove redundant detection frames belonging to the same target, obtain the largest detection frame that covers the detection frames of the same target in the current matching pair as the final detection result, and save the upper left corner coordinates (x result_min ,y result_min ) and the lower right corner coordinate (x result_max ,x result_max ), corresponding to multiple target detection boxes [(x 0_min ,y 0_min ,x 0_max ,y 0_max ), (x 1_min ,y 1_min ,x 1_max ,y 1_max )…(x k_min ,y k_min ,x k_max ,y k_max )].

[0061] Since the image is cut in a sliding window manner, there may be an incomplete target in a sub-image, resulting in an incomplete detection frame. The traditional method of calculating IOU cannot effectively eliminate incomplete detection frames for the same target. The present invention improves the traditional IOU calculation method and combines the algorithm logic of merging and redundantly eliminating detection frames for the same target to effectively eliminate incomplete detection frames for the same target, thereby ensuring the accuracy of detection.

[0062] Step 7: The coordinates of the local lowest center point representing each detected person are derived from the coordinates of the detected persons saved in Step 6:

[0063] x k_minimum_center =(x k_min +x k_max ) / 2

[0064] y k_minimum_center=y k_max

[0065] Calculate the local minimum center point (x k_minimum_center ,y k_minimum_center ) to the shortest distance w from the water surface area obtained in Step 4 minimum , if w minimum If the value is less than or equal to the set threshold, the target is considered to be suspected of phishing, an early warning is issued, and the target detection frame is displayed in the original image. Figure 7 As shown, waiting for further manual verification. minimum If the value is greater than the set threshold, it is considered that the target does not have any fishing behavior, and no warning or detection result display is performed. The preferred threshold of the present invention is 30.

[0066] Step 8: Continue to obtain images captured by the surveillance camera and repeat the operations of Steps 3-7 to achieve continuous detection of whether there are people engaging in abnormal fishing behavior under a high-angle surveillance perspective.

[0067] The present invention firstly realizes the elimination of people who do not appear around the water surface area in the image by adjusting the threshold of the overlapping area between the original image area and the water surface area in the sliding window. However, this threshold alone cannot guarantee that the people detected in the image are fishing. Therefore, based on the improvement of the traditional IOU calculation method, the threshold is adjusted to remove redundant frames to ensure the detection accuracy. The threshold in Step 7 is further introduced by the present invention. By adjusting the size of the threshold, it can be ensured that the detected people displayed in the original image are very close to the water surface, and the possibility of fishing behavior is very high, thereby issuing an early warning.

[0068] The present invention can be implemented based on a computer program. Based on this, the present invention also provides an electronic device, the electronic device includes a processor and a memory, the memory stores at least one instruction or at least one program, the at least one instruction or at least one program is loaded and executed by the processor to implement the above-mentioned method for detecting whether a person has abnormal fishing behavior under a high-distance monitoring perspective. A computer-readable storage medium is also provided, the computer-readable storage medium stores at least one instruction or at least one program, and when the at least one instruction or one program is executed, the above-mentioned method for detecting whether a person has abnormal fishing behavior under a high-distance monitoring perspective is implemented.

Claims

1. The method for detecting phishing behavior from a high-level monitoring perspective is characterized by: It includes the following steps: Step 1: Construct a binary image dataset of water surface areas in different scenes and obtain the network weights of the image semantic segmentation network through training; Step 2: Build a personnel data set and obtain the network weights of the target detection network with global perception capabilities through training; Step 3: Get the image taken by the surveillance camera, establish a rectangular coordinate system with the upper left corner of the image as the coordinate origin, the horizontal direction as the x-axis, and the vertical direction as the y-axis; Step 4: Use the image semantic segmentation network to detect the water surface area in the image, obtain the edge contour of the water surface area and the set of coordinates of the edge points of the water surface area [(x0, x1, ... x k ), (y0,y1,…y k )], get the maximum coordinate value of the edge point (x water_min ,x water_max ) and (y water_min ,y water_max ), thereby deriving the maximum enclosing rectangle of the edge contour of the water surface area and saving the coordinates of the upper left corner of the enclosing rectangle (x water_min ,y water_min ) and the lower right corner coordinate (x water_max ,y water_max ); Step 5: Use the sliding window strategy to traverse the original image and cut it. Before cutting, calculate the overlapping area S of the maximum circumscribed rectangle of the original image area in the sliding window and the edge contour of the water surface area derived in Step 4. overlap , if S overlap If it is greater than the set threshold, it means that the area of ​​the sliding window contains the water surface and the area adjacent to the water surface that needs to be paid attention to. The original image area in the sliding window is cut and saved. Otherwise, it means that the area in the sliding window is not adjacent to the water surface and is a non-focus area. The sliding window moves and judges the next area. Step 6: Use the target detection network with global perception capability to detect people in the sub-image area cut from the original image in Step 5, save the detection results of each sub-image, and obtain the target detection frame of the people in the sub-image area by merging and eliminating redundancies of the detection results, and save the coordinates of the detection frame: [(x 0_min ,and 0_min ,x 0_max ,and 0_max ),(x 1_min ,and 1_min ,x 1_max ,and 1_max )…(x k_min ,and k_min ,x k_max ,and k_max )]; Step 7: Calculate the local minimum center point coordinates of each detected person using the detection box saved in Step 6: x k_minimum_center =(x k_min +x k_max ) / 2 and k_minimum_center =and k_max Calculate the local minimum center point (x k_minimum_center ,y k_minimum_center ) to the shortest distance w from the edge contour of the water surface area obtained in Step 4 minimum , if w minimum If the value is less than or equal to the set threshold (no modification), the target is considered to be suspected of phishing, an early warning is issued, and the target detection frame is displayed in the original image, waiting for further manual verification; if w minimum If it is greater than the set threshold, it is considered that there is no phishing behavior on the target, and no warning or detection result display will be issued; Step 8: Get the image captured by the surveillance camera and repeat the operations of Step 3-7 to achieve continuous detection of whether there are people engaging in abnormal fishing behavior under a high-angle surveillance perspective.

2. The method for detecting fishing behavior under high-distance monitoring viewing angle according to claim 1 is characterized in that In Step 1, the edges of the binary images of the water surface areas in different scenes are marked by tracing and drawing points to create training sets and test sets for training the image semantic segmentation network.

3. The method for detecting fishing behavior under high-distance monitoring viewing angle according to claim 1 is characterized in that In Step 2, after the image is processed by the data enhancement method, the people in the image are annotated as positive samples, and the sub-image containing only the background class after cropping the original image is introduced as a negative sample to create a training set and a test set for training a target detection network with global perception capabilities.

4. The method for detecting fishing behavior under high-distance monitoring viewing angle according to claim 1 is characterized in that The image semantic segmentation network uses the improved Deeplabv3+. The following improvements are made to the Backbone part of the Deeplabv3+ network: The Backbone of the DeepLabv3+ network adopts the MobileNetV2 network, which is composed of stacked inverted residual modules. For the inverted residual module that uses residual edges, the CBAM attention module is introduced after the depthwise separable convolution of the original branch; for the inverted residual module that does not use residual edges, three branches are expanded on the basis of the original branches to obtain four parallel branches. Among the three expanded branches, one branch uses 1×1 convolution, and the other two branches set the boundary filling and receptive field expansion factor of the depthwise separable convolution to 2 or 3 respectively based on the structure of the original branches. The four branches extract features of different scales through receptive fields of different scales, and the features of the four branches are fused as the output of the inverted residual module that does not use residual edges; The following improvements are made to the decoder part of the Deeplabv3+ network: the two feature maps output by the first 6 layers and the first 14 layers of the MobileNetv2 network are passed through the CBAM attention module, and after the feature map size is adjusted, they are spliced ​​with the feature maps output by the first 4 layers of the MobileNetv2 network as shallow feature maps and sent to the decoder; the deep feature map obtained by the MobileNetv2 network and the ASPP module is passed through the CBAM attention module as a new deep feature map, and the new deep feature map is upsampled and sent to the decoder; the shallow and deep feature maps sent to the decoder are fused, and the boundaries and details of the water surface area are decoded and output.

5. The method for detecting fishing behavior under high-distance monitoring viewing angle according to claim 1 is characterized in that Step 5 is as follows: Step 5.1: Set the number of rows to 4, the number of columns to col to 6, and the length of the original image to Image Height 、WidthImage Width , the overlap ratio between the subgraphs Overlap=0.5, the length of the sliding window Window Height , Width Window Width ; Step 5.2: According to the length of the original image Height 、WidthImage Width , the overlap ratio Overlap between the cut subgraphs, the length of the sliding window Window Height , Width Window Width , calculate the coordinate set of all points that the coordinates of the upper left corner of the sliding window will traverse in the original image: [(x0,y0),(x1,y0)…(x5,y0),(x0,y1),(x1,y1)…(x5,y1)…(x0,y3),(x1,y3)…(x5,y3)] Step 5.3: Take out (x p ,y q ), through (x p ,y q ) deduce (x p' ,y q' ): x p' =x p +Window Width and q' =and q +Window Height (x p ,y q ), (x p' ,y q' ) represent the coordinates of the upper left corner and lower right corner of the sliding window, p, q represent that the sliding window is currently traversed to the pth row and qth column, and the original image area within the sliding window is cropped; the coordinates of the sliding window are compared with the coordinates of the maximum circumscribed rectangle (x water_min ,y water_min ,x water_max ,y water_max ), calculate the overlapping area of ​​the maximum circumscribed rectangle of the original image area in the current sliding window and the edge contour of the water surface area. The area is calculated as follows: S overlap =(min(x water_max ,x p' )-max(x water_min ,x p ))*(min(y water_max ,y p' )-max(y water_min ,y p )) Step 5.4: Determine S overlap and set the threshold value. If it is greater than the threshold, the area in the sliding window is considered to be adjacent to the water surface and is an area that needs attention. The original image area in the sliding window is cut and saved. If it is less than or equal to the threshold, the area is considered to be a background area where it is impossible for fishermen to appear, and the sliding window moves to judge the next area.

6. The method for detecting fishing behavior under high-distance monitoring viewing angle according to claim 1 is characterized in that The target detection network with global perception capability is an end-to-end target detection framework DETR.

7. The method for detecting fishing behavior under high-distance monitoring viewing angle according to claim 1 is characterized in that Step 6 is as follows: Step 6.1: Use the target detection network with global perception capability to detect people in the sub-image area cut from the original image in Step 5, convert the relative coordinates of the person detection frame in the sub-image area into absolute coordinates in the original image and save them; Step 6.2: Sort the detection frames in descending order according to their confidence. Take the detection frame with the largest confidence each time as the detection frame to be matched. Calculate the iou between the detection frame to be matched and all remaining detection frames using the minIOU method. The iou calculation formula is as follows: Determine the size of iou and the set threshold. If it is greater than the threshold, it is considered that the current detection frame matches the detection frame to be matched, and they are for the same target. The detection frame to be matched and the detection frame matched to the same target are stored as a set of matching pairs, and the matched detection frame is removed. Otherwise, it is considered to be a detection frame for two different targets. Finally, repeat the operation of Step 6.2 from the remaining detection frames until all the stored detection frames are stored in the form of matching pairs. If there is a detection frame to be matched that does not have a matching detection frame, it is considered to be a falsely detected background class and is directly removed; Step 6.3: Merge and eliminate redundant detection frames of the same target, and obtain the largest detection frame in the current matching pair as the final target detection result.

8. The method for detecting fishing behavior under high-distance monitoring viewing angle according to claim 1 is characterized in that In Step 7, if w minimum ≤30, the target is considered to be suspected of phishing, an early warning is issued, and the target detection frame is displayed in the original image, waiting for further manual verification; if w minimum >30, it is considered that there is no phishing behavior on the target, and no warning or detection result display will be issued.

9. An electronic device, characterized in that The electronic device includes a processor and a memory, wherein the memory stores at least one instruction or at least one program, and the at least one instruction or at least one program is loaded and executed by the processor to implement the fishing behavior detection method under the high-distance monitoring perspective described in any one of claims 1-8.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores at least one instruction or at least one program. When the at least one instruction or the program is executed, the method for detecting fishing behavior under a high-distance monitoring perspective as described in any one of claims 1-8 is implemented.

Images