Phishing detection method and device, computer device and storage medium

Abstract

The application discloses a fishing detection method and device, computer equipment and a storage medium, and the method comprises the following steps: obtaining a target detection frame comprising a pedestrian and a fishing rod; performing fishing behavior identification on the target detection frame through a fishing behavior detection model, and outputting a first result of an identification result being "fishing behavior"; performing fishing behavior identification on the target detection frame through a human body fishing rod key point detection model, and outputting a second result of the identification result being "fishing behavior"; and voting the first result and the second result to output a final identification result. The fishing detection method provided by the application can detect and manage the behavior of stealing a fishing rod in a day and night scene for 24 hours a day, can greatly save labor costs, can identify illegal fishing behavior in a large water area for 24 hours a day at low cost, improves the management level of water areas where fishing is prohibited, and improves the detection accuracy by voting the mean value of the first result and the second result to output the final identification result.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computer vision recognition, and particularly relates to a phishing detection method and device, computer equipment and a storage medium. BACKGROUND

[0002] Fishing is a popular outdoor activity, and we can often see anglers at the edge of rivers, lakes or reservoirs. However, in order to manage needs and commercial interests, fishing is not allowed in many places, such as lakes in scenic spots and privately contracted reservoirs. However, there are still some cases of illegal fishing from time to time, which not only brings hidden dangers to the safety management of the relevant water area, but also may cause personal injury to the illegal fisherman, such as accidental falling into water, drowning, or touching high-voltage electricity, etc. In addition, the waste generated during the illegal fishing process may also cause water pollution, etc.

[0003] The existing means for prohibiting fishers from illegally fishing in the controlled water area mainly include setting up warning signs in the corresponding area, or arranging special management personnel for patrol. However, in practice, it is found that setting up warning signs cannot effectively drive away illegal fishers, and arranging special management personnel for patrol greatly wastes human resources, and cannot achieve 24-hour all-weather patrol and monitoring.

[0004] In view of this, some existing technologies also provide a means based on computer vision recognition to detect and identify illegal fishers. Object detection is one of the basic tasks in the field of computer vision, and has been studied for nearly twenty years in the academic field. In recent years, with the hot development of deep learning technology, object detection algorithms have also shifted from traditional algorithms based on handcrafted features to detection techniques based on deep neural networks.

[0005] However, the existing computer illegal fishing identification technology mainly relies on the principle of traditional frame pixel comparison method. The principle of pixel comparison method is to collect real-time images in the detection area, and compare the front and back frame data of the collected images. Specifically, the pixel difference value of k and k+1 time is calculated to detect the image difference between the front and back time. Then the probability distribution of the difference pixel point value is calculated and compared with the existing phishing image database to determine whether there is phishing behavior. Because the available phishing image data is limited, that is, the standard data distribution that can be compared is limited, and the pixel difference value distribution collected and calculated is random and diverse, this method has two problems: (1) pedestrians are often mistaken for phishers; (2) the phishing behavior cannot be detected. At the same time, the image data collected by the existing computer illegal fishing identification technology is mainly visible light image, which is sensitive to light. Therefore, many fishing enthusiasts like to fish at night, so under the cover of night, these illegal fishermen can still escape from monitoring, and the real-time monitoring of the illegal fishing behavior cannot be realized for 24 hours a day. SUMMARY

[0006] To solve the above technical problems, the fishing detection method, device, computer equipment and storage medium provided by the embodiments of the present application can detect and manage the illegal fishing behavior for 24 hours a day in day and night scenes, and the detection accuracy is high.

[0007] A fishing detection method, comprising:

[0008] obtaining a target detection frame, wherein the target detection frame includes pedestrians and fishing rods;

[0009] performing fishing behavior identification on the target detection frame through a fishing behavior detection model, and outputting a first result of the identification result as "fishing behavior";

[0010] performing fishing behavior identification on the target detection frame through a human-fishing rod key point detection model, and outputting a second result of the identification result as "fishing behavior";

[0011] voting the first result and the second result to output a final identification result.

[0012] Preferably, in the fishing detection method described above, the target detection frame comprises:

[0013] extracting a pedestrian frame and a fishing rod frame in the to-be-detected image;

[0014] matching the fishing rod frame with the pedestrian frame in the same to-be-detected image respectively;

[0015] if the intersection-over-union value obtained by matching is greater than a preset intersection-over-union threshold value, it is determined that the pedestrian in the pedestrian frame is associated with the fishing rod in the fishing rod frame at this time.

[0016] generate a target detection frame according to the pedestrian and the fishing rod.

[0017] Preferably, in the fishing detection method, a pre-trained target detection model is used to extract a pedestrian frame and a fishing rod frame in a to-be-detected image, and before the extraction of the pedestrian frame and the fishing rod frame in the to-be-detected image, the method further comprises:

[0018] obtaining preprocessed sample data;

[0019] inputting the preprocessed sample data into a preset initial target detection model to obtain an output result;

[0020] adjusting a sample type weight in the initial target detection model according to a preset focal loss function and the output result to obtain a parameter-adjusted target detection model;

[0021] training the parameter-adjusted target detection model by a batch stochastic gradient descent algorithm to obtain the pre-trained target detection model.

[0022] Preferably, in the fishing detection method, the obtaining of the preprocessed sample data comprises:

[0023] obtaining a first original image and a second original image in sample data;

[0024] performing mixed enhancement processing on the first original image and the second original image according to a mixing weight to obtain enhanced sample data.

[0025] Preferably, in the fishing detection method, before the fishing behavior detection model is used to perform fishing behavior recognition on the target detection frame and output a first result of the recognition result being “fishing behavior”, the method further comprises:

[0026] extracting a fishing rod frame in the preprocessed sample data, wherein the fishing rod frame comprises a sample pedestrian and a sample fishing rod;

[0027] training a preset classification model according to the fishing rod frame and a batch stochastic gradient descent algorithm;

[0028] if a training result obtained by training reaches a preset training threshold, using the trained classification model as the fishing behavior detection model.

[0029] Preferably, in the fishing detection method, before the human-fishing rod key point detection model is used to perform fishing behavior recognition on the target detection frame and output a second result of the recognition result being “fishing behavior”, the method further comprises:

[0030] The human frame is marked with key point features to obtain human body key points corresponding to the sample pedestrian and fishing rod key points corresponding to the sample fishing rod;

[0031] The preset segmentation model is trained according to the human body key points, the fishing rod key points and a batch stochastic gradient descent algorithm, and the trained segmentation model is used as the human body fishing rod key point detection model.

[0032] Preferably, in the fishing detection method, the voting of the first result and the second result to output the final recognition result comprises:

[0033] The mean value of the first result and the confidence of the second result are obtained respectively;

[0034] The final recognition result is output by voting calculation according to the mean value and the confidence.

[0035] A fishing detection device comprises:

[0036] An RGB-IR image acquisition module is configured to acquire a to-be-detected image in a monitored water area scene;

[0037] A target detection module is configured to extract a pedestrian frame and a fishing rod frame in the to-be-detected image, and generate a target detection frame including the pedestrian and the fishing rod according to an intersection-over-union threshold;

[0038] A fishing behavior detection module is configured to perform fishing behavior recognition on the target detection frame, and output a first result of the recognition result as “fishing behavior”;

[0039] A human body fishing rod key point detection module is configured to perform fishing behavior recognition on the target detection frame, and output a second result of the recognition result as “fishing behavior”;

[0040] A voting module is configured to perform voting calculation according to the mean value of the first result and the confidence of the second result, and output a final recognition result.

[0041] A computer device comprises a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the fishing detection method when executing the computer program.

[0042] A computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the fishing detection method.

[0043] The beneficial effects of the present application are that the fishing detection method provided by the present application can detect and manage the behavior of stealing fishing in the night scene for 24 hours a day, and the final recognition result is output by voting the mean value of the first result and the second result, which improves the detection accuracy. Through 24 hours of automatic detection, the labor cost is greatly saved, so that the illegal fishing behavior in large water areas can be identified for 24 hours a day, low cost and high accuracy, and the management level of the water area where fishing is prohibited is improved. BRIEF DESCRIPTION OF DRAWINGS

[0044] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings needed to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments of the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0045] Figure 1 The flowchart of the fishing detection method described in the present application;

[0046] Figure 2 The flowchart for obtaining the target detection frame;

[0047] Figure 3 The pre-training flowchart of the target detection model described in the present application;

[0048] Figure 4 The training flowchart of the fishing behavior detection model described in the present application;

[0049] Figure 5 The training flowchart of the human fishing rod key point detection model described in the present application;

[0050] Figure 6 The structural schematic diagram of the fishing detection device described in the present application;

[0051] Figure 7 The internal structure schematic diagram of an embodiment of the computer device described in the present application;

[0052] Figure 8 The internal structure schematic diagram of another embodiment of the computer device described in the present application. DETAILED DESCRIPTION

[0053] The technical solutions of the embodiments of the present application will be described in detail below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some embodiments of the present application, not all embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor are within the scope of protection of the present application.

[0054] Referring to Figure 1 As shown in the figure, the fishing detection method proposed by the embodiment of the application comprises:

[0055] In step S100, a target detection frame is acquired, wherein the target detection frame comprises a pedestrian and a fishing rod.

[0056] Specifically, the to-be-detected images can be acquired in real time by an image sensor arranged in a monitored water area scene, which can be time sequence frame images or time sequence frame images in a streaming media video. The time sequence frame images taken from the streaming media video can be continuous frame images or interval frame images taken according to an interval frame taking rule in a time sequence frame image set. For example, if M frame to-be-detected images are included in the streaming media video, at least one frame to-be-detected image is acquired every N frames. It should be noted that the frame rate of the to-be-detected images in the streaming media video is generally above 25 frames per second. If each to-be-detected image is detected, the operation amount will be increased, and the response and timeliness of the fishing behavior recognition detection will be reduced. In the embodiment, multiple to-be-detected images are acquired from the streaming media video at intervals, which can reduce the operation amount of image processing and improve the speed of fishing behavior recognition detection, so that real-time detection result feedback can be achieved, and illegal fishing poachers can be effectively cracked down.

[0057] The to-be-detected images acquired in real time on the spot can be transmitted to a monitoring center server in a wired or wireless manner, and then the server detects and recognizes the pedestrians and fishing rods in the to-be-detected images, and marks and frames the pedestrians on the to-be-detected images with pedestrian frames and marks and frames the fishing rods on the to-be-detected images with fishing rod frames. When marking and framing the pedestrians and fishing rods, the to-be-detected images are scanned and detected in an automatic sliding window manner.

[0058] Specifically, by traversing the pedestrian frames on each frame to-be-detected image, the pedestrian frames associated and matched with the fishing rod frames on the frame to-be-detected image are determined according to a preset intersection-over-union threshold, and a target detection frame containing the pedestrian and the fishing rod is generated.

[0059] Specifically, the pedestrian frames and the fishing rod frames on the to-be-detected images detected in step S100 can be mutually irrelevant and have no matching relationship, or can be associated and matched. The association and matching referred to herein means that the pedestrian frame and the fishing rod frame are a feature set of a poacher, that is, the pedestrian frame reflects the human body features of the poacher, and the fishing rod frame reflects the fishing rod features of the poacher, that is, there is a poacher at this time. By associating and matching the pedestrian frames and the fishing rod frames, the target detection frame containing the pedestrian and the fishing rod is generated, which can avoid identifying passing strangers as poachers and improve the accuracy of the identification and detection.

[0060] Step S200, the target detection frame is recognized by the phishing behavior detection model, and the first result of the recognition result is "phishing behavior".

[0061] Specifically, the target detection frame is a label frame containing a pedestrian and a fishing rod, that is, the two associated matching features of a fisherman detected and recognized in step S200, that is, the pedestrian feature and the fishing rod feature. By recognizing the target detection frame containing the pedestrian and the fishing rod, one or more fisherman in the to-be-detected image can be detected, and one target detection frame represents one fisherman. In a to-be-detected image, there can be no target detection frame, or one or more target detection frames. In the case where there is no target detection frame, the to-be-detected image does not enter the phishing behavior recognition process of step S200. In the case where there is one target detection frame or multiple target detection frames, the to-be-detected image enters step S200 for phishing behavior recognition and outputs the first result of the recognition result as "phishing behavior".

[0062] Step S300, the target detection frame is recognized by the human fishing rod key point detection model, and the second result of the recognition result is "phishing behavior".

[0063] Specifically, when detecting the human fishing rod key points, the target detection frame of the multiple frames of images in the same time sequence as the frame image outputting the first result is loaded, that is, the target detection frames in the multiple frames of images in the same time sequence as the frame image outputting the first result are loaded at the same time, and then the human fishing rod key point features in the target detection frames in the multiple frames of images are recognized for phishing behavior recognition. It should be noted that the phishing behavior recognition here is different from the phishing behavior recognition in step S200. Here, the human key point features and the fishing rod key point features of the fisherman are recognized, rather than the target detection frame. By obtaining the multi-frame human fishing rod key point feature data, then using STGCN (spatial-temporal graph convolutional network model) to extract the key point feature information for phishing behavior recognition, the recognition and detection result of the phishing behavior can be more accurately obtained.

[0064] Step S400, voting the first result and the second result to output the final recognition result.

[0065] Specifically, the first result of the multiple frames of images in the same time sequence is averaged, and then the second result output based on the human fishing rod key point feature information of the multiple frames of images in the same time sequence is voted to output the final result, further improving the detection and recognition accuracy of the phishing behavior.

[0066] In some embodiments of the present application, the image to be detected is acquired by an RGB-IR image sensor arranged under the monitored water area scene. The RGB-IR image sensor is an image sensor that can simultaneously sense visible light and infrared light signals. The infrared light signal is used to provide scene brightness information in a low-illumination environment. The object of interest can be presented in the case of poor night lighting by using the infrared feature map. Here, the object of interest includes a human target and a fishing rod target. Since fishing at night usually involves projecting light, the fishing rod can reflect infrared features under the light, and the infrared features of a poacher or a pedestrian can be reflected by the human body's own infrared signal. In the case of normal illumination, the R, G, and B visible light components can achieve good imaging results in the image sensor, mainly applied to good daytime illumination conditions.

[0067] Further, in some embodiments of the present application, the image to be detected is a plurality of real-time static images in a unit time sequence acquired from a stream media video based on an interval frame taking rule by an RGB-IR image sensor arranged under the monitored water area scene. Specifically, as another example, the interval frame taking rule can be 1+(n-1), where n is the frame taking period, that is, only 1 frame of the image to be detected is acquired within the frame taking period n. In the 1+(n-1) mode, the detection time is 1 / n of the frame taking period. For example, if n is 5, the number of frames of the image to be detected accounts for 20% of the number of frames of the video to be detected, which can further improve the video processing speed and the number of video access channels. The interval acquisition of a plurality of images to be detected from the stream media video can improve the processing speed of the image to be detected and improve the response timeliness of the fishing behavior recognition detection, thereby reducing the delay of the fishing behavior recognition detection.

[0068] Further, in some embodiments of the present application, the image to be detected is a plurality of real-time static images in a unit time sequence acquired from a stream media video based on an interval frame taking rule by an RGB-IR image sensor arranged under the monitored water area scene. Specifically, as another example, the interval frame taking rule can be 1+(n-1), where n is the frame taking period, that is, only 1 frame of the image to be detected is acquired within the frame taking period n. In the 1+(n-1) mode, the detection time is 1 / n of the frame taking period. For example, if n is 5, the number of frames of the image to be detected accounts for 20% of the number of frames of the video to be detected, which can further improve the video processing speed and the number of video access channels. The interval acquisition of a plurality of images to be detected from the stream media video can improve the processing speed of the image to be detected and improve the response timeliness of the fishing behavior recognition detection, thereby reducing the delay of the fishing behavior recognition detection.

[0069] Further, in some embodiments of the present application, as Figure 2As shown, the steps of obtaining the target detection frame include:

[0070] Step S110, extracting a pedestrian frame and a fishing rod frame from the image to be detected;

[0071] Step S120, matching the fishing rod frame with the pedestrian frame in the same image to be detected;

[0072] Step S130: If the IoU value obtained by matching is greater than a preset IoU threshold, it is determined that the pedestrian in the pedestrian frame is associated with the fishing rod in the fishing rod frame;

[0073] Step S140: Generate a target detection frame based on the pedestrian and the fishing rod.

[0074] Specifically, in the embodiment of the present invention, the calculation rule of the intersection-over-union ratio is as follows:

[0075] The upper left corner coordinates and lower right corner coordinates of the pedestrian box a are marked as bbox a =[(x a1 ,y a1 ),(x a2 ,y a2 )], where the coordinate of the upper left corner of the pedestrian frame a is marked as (x a1 ,y a1 ), the coordinate of the lower right corner of the pedestrian frame a is marked as (x a2 ,y a2 );

[0076] The upper left corner coordinate and the lower right corner coordinate of the fishing rod frame b are marked as bbox b =[(x 1b ,y 1b ),(x2,y 2b )], where the coordinate of the upper left corner of the fishing rod frame b is (x 1b ,y 1b ), the coordinates of the lower right corner of the fishing rod frame b are marked as (x2,y 2b );

[0077] The intersection over union (IoU) is the intersection over union of the pedestrian frame a and the fishing rod frame b, which can be expressed as:

[0078]

[0079] Further, the preset intersection-over-union threshold in the embodiment of the present application is set to 0.1-0.3. Wherein, as a preferred embodiment of the present application, the preset intersection-over-union threshold is set to 0.2. It is found in actual test that when the intersection-over-union of a fishing rod frame and a pedestrian frame is greater than 0.2, the areas contained by the two frames can be considered as a person and a fishing rod associated with him; when the intersection-over-union threshold is less than 0.2, the fishing rod and the pedestrian of the two frames are considered to be irrelevant, so the intersection-over-union threshold is set to 0.2, which is more appropriate. Under the premise of not increasing the amount of operation, the high detection and recognition accuracy of the fishing behavior is ensured.

[0080] In the preferred embodiment of the present application, the pedestrian frame and the fishing rod frame on the image to be detected are extracted by a pre-trained target detection model. Specifically, as shown in Figure 3 the training process of the target detection model includes:

[0081] Step S1101, obtaining preprocessed sample data;

[0082] Step S1102, inputting the preprocessed sample data into a preset initial target detection model to obtain an output result;

[0083] Step S1103, adjusting the sample type weight in the initial target detection model according to a preset focal loss function and the output result to obtain a target detection model with adjusted parameters;

[0084] Step S1104, training the target detection model with adjusted parameters by a batch stochastic gradient descent algorithm to obtain a pre-trained target detection model.

[0085] Specifically, in the preferred embodiment of the present application, the initial target detection model is a YOLOv3 model, and the sample types include positive samples and negative samples.

[0086] Wherein, the preprocessed sample data is specifically obtained by:

[0087] Step S11011, obtaining a first original image and a second original image in sample data;

[0088] Step S11012, performing mixed enhancement processing on the first original image and the second original image according to a mixing weight to obtain enhanced sample data.

[0089] Specifically, the method of data enhancement processing can be expressed by the expression:

[0090] image mix =lambda*image a +(1-lambda)*imageb ;

[0091] bbox mix ∈bboxes mix ;

[0092] bbox a ∈bboxes a ;

[0093] bbox b ∈bboxes b ;

[0094] bboxes mix =bboxes a ∪bboxes b ;

[0095] Wherein, image mix represents the mixed enhanced image, image a represents the first original image, image b represents the second original image, bboxes mix represents the feature bounding box of the mixed enhanced image, bboxes a represents the feature bounding box of the first original image, bboxes b represents the feature bounding box of the second original image, and lambda is the mixing weight. In the preferred embodiment of the present application, the lambda mixing weight is 0.5, so that the first original image and the second original image have the same weight in the mixed enhanced image. Through this data enhancement processing method, the training samples can be greatly enriched, and the detection efficiency and accuracy of the pedestrian / fishing rod detection model for pedestrian and fishing rod features are improved, so as to quickly generate corresponding pedestrian boxes and fishing rod boxes. Specifically, the pedestrian box includes pedestrian features, the fishing rod box includes fishing rod features, and the target detection box includes both pedestrian features and fishing rod features, that is, the feature label of a fishing behavior identified in the present application.

[0096] In the embodiment of the present application, the sample data is derived from a plurality of visible light images and infrared images corresponding to the existing stealing fishing behavior in the monitored water area scene. Specifically, the visible light images or infrared images labeled with pedestrian boxes and fishing rod boxes are used as sample data for model training.

[0097] In the embodiments of the present application, in order to improve the accuracy of model training, 1000 frames of corresponding visible light images and infrared images of fishing behaviors can be used for repeated training and iterative training. In order to improve the reading speed of the images and the accuracy of the recognition, the visible light images and the infrared images need to be aligned before the pedestrians or the fishing rods on the infrared images are labeled. Specifically, the image alignment includes feature-based image alignment and data-based alignment. The feature-based image alignment is to find a spatial transformation to map the floating image to the reference image, so that the points corresponding to the same position in space in the two images are one-to-one corresponding, so as to achieve the purpose of information fusion, and facilitate the positioning, extraction and detection and recognition of features. In the embodiments of the present application, preferably, the visible light image is taken as the reference image, and the infrared image is taken as the floating image, the infrared image is mapped to the visible light image through the spatial transformation, so that the pedestrian features and the fishing rod features on the infrared image correspond to and align with the pedestrian features and the fishing rod features on the visible light image at a time node. Specifically, the feature-based image alignment can be processed by using the Homography (homography) algorithm, the Mesh Warps (mesh warps) algorithm or the Optical flow (optical flow) algorithm in the prior art. The data-based alignment is to arrange the infrared image data and the visible light image data in space according to certain rules, rather than sequentially one after another. Because the processing of the storage space is very different on different hardware platforms, some platforms can only access some specific types of data from some specific addresses. If the data storage is not aligned according to the requirements of the platform, the access efficiency will be affected. For example, some platforms start reading from even addresses every time. If an int type (assuming a 32-bit system) is stored in an even address, one read cycle can read it out. If it is stored in an odd address, it may need 2 read cycles, and the high and low bytes of the read results need to be spliced to get the int data. Obviously, the reading efficiency is greatly reduced. Therefore, the data-based alignment of the infrared image data and the visible light image data realizes the game in space and time, improves the data reading efficiency, and improves the operation amount and the response timeliness. Specifically, the data alignment is a common prior art in the data access processing process, and four-byte alignment is usually used, which will not be described here.

[0098] Specifically, in the embodiments of the present application, the mixed enhancement processing of the sample data in step S11012 is specifically weighted processing of difficult samples in positive and negative samples using a focal loss function, which can improve sample quality, reduce loss, and improve the accuracy of difficult sample judgment in the model training process, so as to weaken the problems of sample class imbalance and sample classification difficulty imbalance. Specifically, in the embodiments of the present application, the input data is processed by data enhancement based on the YOLOv3 model.

[0099] Further, in some embodiments of the present application, as shown in Figure 4 The training process of the phishing behavior detection model includes:

[0100] Step S210, extracting the human rod frame in the preprocessed sample data, wherein the human rod frame includes sample pedestrians and sample fishing rods;

[0101] Step S220, training a preset classification model according to the human rod frame and the batch random gradient descent algorithm;

[0102] Step S230, if the training result obtained by training reaches a preset training threshold, the classification model trained at this time is used as the phishing behavior detection model.

[0103] Specifically, in the preferred embodiments of the present application, the classification model is a mobilenetv1 (a kind of lightweight neural network) model.

[0104] Further, in some embodiments of the present application, as shown in Figure 5 The training process of the human-fishing rod key point detection model includes:

[0105] Step S310, marking the human rod frame with key point features to obtain human key points corresponding to sample pedestrians and fishing rod key points corresponding to sample fishing rods;

[0106] Step S320, training a preset segmentation model according to the human key points, fishing rod key points, and batch random gradient descent algorithm, and using the trained segmentation model as the human-fishing rod key point detection model.

[0107] Specifically, in the preferred embodiments of the present application, the segmentation model is an improved Unet model.

[0108] Specifically, in the preferred embodiment of the present application, the human fishing rod key point feature information includes 17 human key point features and 1 fishing rod key point feature corresponding to human joints. Among them, the 17 human key point features include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The 1 fishing rod key point feature includes the end of the fishing rod connected to the fishing line.

[0109] In step 320, since the fishing rod is relatively thin and difficult to identify in the image, the improved Unet (segmentation network) model of the present application replaces the ordinary convolution layer with the empty convolution on the basis of the existing Unet model, enhances the receptive field during convolution calculation, and obtains the context (background) information around the fishing rod to represent the existence of the fishing rod, thereby improving the accuracy of the fishing rod key point detection. Specifically, the Unet model is a well-known network used in the field of medical image segmentation. Since its bottom feature and high-level feature fusion idea can well preserve the detailed feature information of the image, the Unet model is used as the base network for key point detection in the embodiment of the present application. In order to further improve the detection rate of small details of the key point, i.e. the fishing rod key point, the convolution layer in the Unet model is replaced with the empty convolution layer to enhance the receptive field during convolution calculation. In step S320, multiple frames of human fishing rod key point data can be obtained, and then STGCN (spatial-temporal graph convolution network model) is used to extract key point feature information for fishing behavior recognition, so as to obtain the final result more accurately.

[0110] Further, in some embodiments of the present application, the voting on the first result and the second result to output the final recognition result specifically includes:

[0111] Step S410, respectively obtaining the mean value of the first result and the confidence of the second result;

[0112] Step S420, voting calculation according to the mean value and the confidence to output the final recognition result.

[0113] Specifically, the mean value of the first result refers to the mean value of the first result of multiple frames of images under one time sequence, and the confidence of the second result refers to the final fishing behavior recognition confidence result obtained by using multiple frames of images under one time sequence to extract human fishing rod key point feature information.

[0114] Among them, the voting rule can be represented as:

[0115]

[0116] Among them, score STGCNis a final fishing behavior recognition confidence result obtained by using multi-frame time series data for the STGCN (Spatial Temporal Graph Convolutional Networks, spatial-temporal graph convolutional network model); score imagemodeli is a result of the i-th frame obtained in step S300; lambda is a weight parameter.

[0117] In a preferred embodiment of the present application, the voting weight of the mean value of the first result is 0.4 < n < 0.6, and the voting weight of the fishing behavior recognition confidence result is 0.4 < m < 0.6, where m + n = 1.

[0118] Specifically, as a preferred embodiment of the present application, the weight parameter lambda is set to 0.5 here, indicating that the detection and recognition results of the two models of the average reference fishing behavior detection model and the human fishing rod key point detection model are averaged. Wherein, n is the number of frames of the multi-frame used, which is set to 5 here, which can achieve a good recognition rate and also has a good performance in speed.

[0119] Further, in the embodiment of the present application, the to-be-detected image is acquired by an RGB-IR image sensor arranged in a monitored water area scene, which includes:

[0120] The human body sensing signal is received, and after the sensing trigger, the RGB-IR image sensor in the dormant state is awakened to collect real-time pictures in the current field of view;

[0121] The collected real-time pictures are returned to the monitoring center server.

[0122] The RGB-IR image sensor in the dormant state is awakened by triggering the human body sensing signal to collect real-time pictures in the current field of view, which can greatly save data storage space and reduce the power consumption of the terminal RGB-IR image sensor.

[0123] On the other hand, another embodiment of the present application also proposes a fishing detection device, which corresponds one-to-one to the fishing detection method in the above-mentioned embodiment. Specifically, as shown in Figure 6 The fishing detection device includes an RGB-IR image acquisition module 10, a target detection module 20, a fishing behavior detection module 30, a human fishing rod key point detection module 40, and a voting module 50.

[0124] The RGB-IR image acquisition module 10 is configured to acquire a to-be-detected image in a monitored water area scene. The target detection module 20 is configured to extract a pedestrian frame and a fishing rod frame in the to-be-detected image, and generate a target detection frame including both the pedestrian and the fishing rod according to an intersection-over-union threshold. The fishing behavior detection module 30 is configured to perform fishing behavior recognition on the target detection frame, and output a first result of the recognition result as "fishing behavior". The human-fishing rod key point detection module 40 is configured to perform fishing behavior recognition on the target detection frame, and output a second result of the recognition result as "fishing behavior". The voting module 50 is configured to perform voting calculation according to a mean value of the first result and a confidence of the second result, and output a final recognition result.

[0125] Specifically, the target detection module 20 completes the following work: first, detecting and recognizing pedestrians and fishing rods in the loaded to-be-detected image to generate corresponding pedestrian frames and fishing rod frames; then, traversing the pedestrian frames on each frame of the to-be-detected image, and matching the pedestrian frames with the fishing rod frames on the frame of the to-be-detected image to determine the pedestrian frames associated with the fishing rod frames according to a preset intersection-over-union threshold; and finally, generating a target detection frame including both the pedestrian and the fishing rod according to the associated matching result.

[0126] The specific working principle of the fishing detection device based on RGB-IR image data can be referred to the working process of the fishing detection method described above, and will not be described here. Each module in the fishing detection device described above can be realized by software, hardware, or a combination thereof. Each module described above can be embedded in or independent of the processor in the computer device in hardware form, or can be stored in the memory in the computer device in software form, so as to be called and executed by the processor to perform the operations corresponding to each module.

[0127] On the other hand, in one embodiment, as shown in Figure 7 , the present application also proposes a computer device, which can be a server, and the internal structure diagram thereof is shown in Figure 7 . The computer device includes a data processor, a memory, a network interface, and a database connected through a device bus. The computer device is provided with a plurality of data processors for providing computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores operating devices, computer programs, and databases. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage medium. The database of the device is used to store data related to image processing. The network interface of the device is used to communicate with external terminals through network connection.

[0128] The memory stores a computer program executable on the processor, and the processor executes the computer program to implement the fishing detection method described above.

[0129] In one embodiment, the present application also provides a computer device, which can be a terminal, a schematic diagram of the internal structure of which is shown in Figure 8 The computer device includes a data processor, a memory, a network interface, a display screen and an input device connected through a system bus. The computer device is provided with a plurality of data processors for providing computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operating system and the computer program in the non-volatile storage medium to run. The network interface of the computer device is used to communicate with external terminals through network connection.

[0130] The memory stores a computer program capable of running on the processor, and the computer program is executed by the processor to implement the phishing detection method.

[0131] Specifically, the display screen of the computer device can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer device can be a touch layer overlaid on the display screen, or a key, trackball or touchpad arranged on the shell of the computer device, or an external keyboard, touchpad or mouse, etc.

[0132] Those skilled in the art can understand that Figure 7 and Figure 8 The structures shown in the above embodiments are only block diagrams of part of the structures related to the scheme of the present application, and do not constitute a limitation on the computer device to which the scheme of the present application is applied. The specific device can include more or fewer components than those shown in the diagram, or combine certain components, or have a different arrangement of components.

[0133] In another aspect, the present application also provides a computer readable storage medium storing a computer program, and the computer program is executed by the processor to implement the phishing detection method.

[0134] It should be understood that the size of the serial number of each step in the above embodiments does not mean the order of execution, and the execution order of each process should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0135] For example, the computer program can be divided into one or more modules / units, which are stored in the memory and executed by the processor to complete the present application. One or more modules / units can be a series of computer program instruction segments capable of completing a specific function, which are used to describe the execution process of the computer program in the computer device.

[0136] The computer device can be a desktop computer, a notebook computer, a palm computer, a cloud server, and the like. The computer device can include, but is not limited to, a processor, a memory. Those skilled in the art can understand that Figure 7 and Figure 8 are merely examples of the computer device, and do not constitute a limitation on the computer device, and can include more or fewer components than shown, or combine some components, or different components, for example, the computer device can also include an input / output device, a network access device, a bus, and the like.

[0137] The processor can be a central processing unit (CPU), and can also be other general-purpose processors, a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic device, discrete gate or transistor logic, discrete hardware components, and the like. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor.

[0138] The memory can be an internal storage unit of the computer device, such as a hard disk or a memory of the computer device. The memory can also be an external storage device of the computer device, such as a plug-in hard disk, a smart media card (SMC), a secure digital (SD) card, a flash card, and the like. Further, the memory can include both the internal storage unit and the external storage device of the computer device. The memory is used to store computer programs and other programs and data required by the terminal device. The memory can also be used to temporarily store data that has been output or will be output.

[0139] Those skilled in the art can clearly understand that, for the convenience and brevity of description, only the division of the above functional units and modules is exemplified, and in actual application, the above functions can be completed by different functional units and modules according to needs, that is, the internal structure of the device is divided into different functional units or modules to complete all or part of the functions described above.

[0140] In addition, each function unit in each embodiment of the present application can be integrated in one processing unit, or each unit can be physically present separately, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software function unit.

[0141] The integrated module / unit, if realized in the form of a software function unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the above-mentioned embodiment methods can also be completed by a computer program instructing related hardware, and the computer program can be stored in a computer-readable storage medium. When the processor executes the computer program, the steps of each method embodiment described above can be implemented. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or some intermediate forms, etc. The computer-readable medium can include any entity or device capable of carrying the computer program code, recording medium, U disk, mobile hard disk, magnetic disk, optical disk, computer memory, read-only memory (ROM), random access memory (RAM), electrical carrier signal, telecommunication signal, and software distribution medium, etc. It should be noted that the content included in the computer-readable medium can be appropriately increased or decreased according to the requirements of legislation and patent practice in the jurisdiction, for example, in some jurisdictions, according to legislation and patent practice, the computer-readable medium does not include electrical carrier signals and telecommunication signals.

[0142] The above-described embodiments are only used to illustrate the technical solutions of the present application, rather than limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that: it can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be included in the protection scope of the present application.

Claims

1. A fishing detection method, characterized in that: include: Obtaining a target detection frame, wherein the target detection frame includes a pedestrian and a fishing rod; Performing fishing behavior recognition on the target detection frame using a fishing behavior detection model, and outputting a first result of the recognition result being "fishing behavior"; Performing fishing behavior recognition on the target detection frame through the human body fishing rod key point detection model, and outputting a second result of the recognition result being "fishing behavior"; Voting on the first result and the second result to output a final recognition result; Voting on the first result and the second result to output a final recognition result includes: Obtaining a mean of the first result and a confidence level of the second result, respectively, where the mean of the first result refers to the mean of the first result of multiple frames of images in a time sequence, and the confidence level of the second result refers to extracting key feature information of the human fishing rod using the multiple frames of images in a time sequence, to obtain a final fishing behavior recognition confidence level result representing the second result; Voting calculation is performed based on the mean and the confidence level to output a final recognition result.

2. The fishing detection method according to claim 1, wherein: The acquiring of the target detection frame comprises: Extract the pedestrian frame and fishing rod frame in the image to be detected; Matching the fishing rod frame with the pedestrian frame in the same image to be detected; If the intersection-over-union ratio obtained by matching is greater than a preset intersection-over-union ratio threshold, it is determined that the pedestrian in the pedestrian frame at this time is associated with the fishing rod in the fishing rod frame; An object detection frame is generated based on the pedestrian and the fishing rod.

3. The fishing detection method according to claim 2, wherein: A pre-trained target detection model is used to extract a pedestrian frame and a fishing rod frame from the image to be detected. Before extracting the pedestrian frame and the fishing rod frame from the image to be detected, the method further includes: Get preprocessed sample data; Inputting the preprocessed sample data into a preset initial target detection model to obtain an output result; Adjusting the sample type weights in the initial target detection model according to a preset focus loss function and the output result to obtain a parameter-adjusted target detection model; The parameter adjustment target detection model is trained by a batch stochastic gradient descent algorithm to obtain a pre-trained target detection model.

4. The fishing detection method according to claim 3, wherein: The obtaining of pre-processed sample data includes: Acquire a first original image and a second original image in the sample data; The first original image and the second original image are subjected to mixed enhancement processing according to a mixed weight to obtain enhanced sample data.

5. The fishing detection method according to claim 3 or 4, characterized in that: Before performing fishing behavior recognition on the target detection frame using the fishing behavior detection model and outputting a first result in which the recognition result is "fishing behavior", the method further includes: Extracting a person-rod frame from the preprocessed sample data, wherein the person-rod frame includes a sample pedestrian and a sample fishing rod; Training a preset classification model according to the human rod framework and batch stochastic gradient descent algorithm; If the training result obtained through training reaches the preset training threshold, the classification model trained at this time will be used as the fishing behavior detection model.

6. The fishing detection method according to claim 5, characterized in that: Before performing fishing behavior recognition on the target detection frame using the human body fishing rod key point detection model and outputting the second result of the recognition result being "fishing behavior", the method further includes: Marking the key point features of the human-rod frame to obtain the human body key points corresponding to the sample pedestrian and the fishing rod key points corresponding to the sample fishing rod; The preset segmentation model is trained according to the human body key points, fishing rod key points and batch stochastic gradient descent algorithm, and the trained segmentation model is used as the human body fishing rod key point detection model.

7. A fishing detection device, characterized in that: include: RGB-IR image acquisition module, used to obtain the image to be detected in the monitored water scene; The target detection module extracts the pedestrian and fishing rod frames in the image to be detected, and generates a target detection frame that includes both the pedestrian and the fishing rod based on the intersection-over-union ratio threshold; a fishing behavior detection module, performing fishing behavior recognition on the target detection frame and outputting a first result of the recognition result being "fishing behavior"; The human fishing rod key point detection module performs fishing behavior recognition on the target detection frame and outputs a second result of the recognition result being "fishing behavior"; A voting module, performing voting calculation based on the mean of the first results and the confidence level of the second results, and outputting a final recognition result; Voting on the first result and the second result to output a final recognition result includes: Obtaining a mean of the first result and a confidence level of the second result, respectively, where the mean of the first result refers to the mean of the first result of multiple frames of images in a time sequence, and the confidence level of the second result refers to extracting key feature information of the human fishing rod using the multiple frames of images in a time sequence, to obtain a final fishing behavior recognition confidence level result representing the second result; Voting calculation is performed based on the mean and the confidence level to output a final recognition result.

8. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein: When the processor executes the computer program, the fishing detection method according to any one of claims 1 to 6 is implemented.

9. A computer-readable storage medium storing a computer program Sequence, characterized in that When the computer program is executed by a processor, the fishing detection method according to any one of claims 1 to 6 is implemented.

Images