A method for controlling dangerous fishing behaviors based on reinforcement learning and deformable Transformer

Abstract

A method for controlling dangerous fishing behaviors based on reinforcement learning and a deformable transformer belongs to the technical field of dangerous behavior control and solves the problem of inaccurate detection of small, slender fishing rods. The present invention uses a sample fitness evaluation model based on reinforcement learning to perform fitness evaluation on each training sample, selects samples with high fitness and sends them to a deformable transformer fishing rod detection model for training. The deformable transformer fishing rod detection model is then used to extract multi-scale spatial features of slender objects for fishing rod detection. Finally, the sample fitness evaluation model based on reinforcement learning is updated through a maximum entropy reinforcement learning algorithm, thereby improving the detection accuracy of the model. The method of the present invention can effectively extract the slender structural features of fishing rods, improve the model's ability to recognize slender features, and can detect dangerous behaviors in high-voltage environments, thereby ensuring the safety of people in high-voltage environments.

Description

Technical Field

[0001] The present invention belongs to the technical field of dangerous behavior control, and relates to a method for controlling dangerous fishing behaviors based on reinforcement learning and a deformable Transformer. Background Art

[0002] With the development of power systems, high-voltage power lines have become a ubiquitous part of our daily lives. However, while high voltage electricity brings convenience, it also carries dangers. Despite various warning signs in high-voltage environments, safety accidents have continued to occur in recent years due to negligence. The causes of electric shock are often due to dangerous behavior by people entering high-voltage environments, such as contact with high-voltage power lines or accidental contact with high-voltage power lines through conductors. Therefore, it is necessary to monitor potential dangerous behavior of people entering high-voltage environments. Among these, electric shocks are common when fishing rods contact high-voltage power lines while fishing near high voltages. Analysis of electric shock behaviors reveals that most anglers are most susceptible to electric shock when lifting or swinging their rods. Therefore, to effectively prevent electric shocks among anglers, it is necessary to detect dangerous behavior in high-voltage environments and issue timely warnings when such behavior is detected. Therefore, a dangerous behavior management algorithm that can detect and make decisions in real time is crucial.

[0003] With the development of computers, the application of computer vision has solved this problem. It uses various imaging systems to replace visual organs as input, and uses computers to replace the brain to complete processing and judgment by using measures such as target detection. As a basic task of computer vision, target detection has received extensive research attention in recent years. The document "Research on Target Detection Method for Transmission Line Anti-External Damage Based on Improved Mask R-CNN" (Power System Protection and Control, Wei Xianzhe et al.) published in December 2021 detects dangerous factors that invade the monitoring area of ​​high-voltage overhead transmission lines; however, the algorithm used in this document has the problem of slow detection speed and is not suitable for real-time detection in the process of identifying dangerous behaviors in high-voltage environments. The document "Detection of Transmission Line Insulator Defects Based on an Improved Lightweight YOLOv4 Model" (Applied Sciences, Qiu Z, Zhu X, Liao C, et al.) published in December 2022 proposed an improved lightweight YOLOv4 model, which was applied to the detection of transmission line insulator defects, improving the detection accuracy and speed. However, due to the limitations of the network model, the detection effect of slender objects with low resolution and small size is not ideal. Therefore, a detection method that can detect objects with slender structural characteristics such as fishing rods is needed.

[0004] This paper is dedicated to the problem of detecting slender structural objects such as fishing rods. From an academic research perspective, the unique characteristics of slender objects bring special challenges, resulting in research topics of scientific value. From an application perspective, once slender objects can be well handled, the practicality of the object detection system will become higher. The document "End-to-end Lane Shape Prediction with Transformers" (Workshop on Applications of Computer Vision, IEEE, Liu R, Yuan Z, Liu T, et al) published in November 2020 proposed a spatial structured deep convolutional neural network (Spatial CNN, SCNN). This SCNN is particularly suitable for long continuous shape structures or large objects with strong spatial relationships but fewer appearance clues, such as lanes, telephone poles, and walls. It can also effectively maintain the continuity of slender structures. The paper "Spatial As Deep: Spatial CNN for TrafficScene Understanding" (Pan X, Shi J, Luo P, et al.), published in December 2017, proposed an end-to-end method by applying the Transformer to feature extraction of slender objects such as lane lines. This method directly outputs parameters that represent the lane shape model and uses a network built with the Transformer to learn a richer structure and global context. However, the color characteristics of lane lines are distinct and differ significantly from the background. The Chinese invention patent application "A Power Defect Detection Method Based on Reinforcement Learning and Transformer," with application publication number CN114821368A and publication date July 29, 2022, proposes a feature extraction module based on dilated convolution, which improves the perception of small objects. However, the feature extraction module based on dilated convolution is not suitable for extracting the narrow and long features of fishing rods; in addition, the comparative document addresses the problem that irrelevant background information in electric cruise images occupies a large amount of computing resources during detection, and adopts a DQN-based reinforcement learning algorithm to reduce the computational proportion of irrelevant background information in the Transformer network, thereby reducing the impact of irrelevant information on the final detection results.

[0005] In summary, the primary target for dangerous behavior detection in fishing scenarios is the fishing rod. Fishing rods, which have slender structural features and little difference from background information such as aquatic plants and ponds, result in mediocre performance of existing networks in fishing rod detection tasks, which easily leads to missed detections and makes it difficult to meet the safety protection requirements of anglers in high-voltage environments. Most existing methods train models directly on large datasets, but not all data samples are useful for the model. For example, bad samples or samples with few features will cause the model's performance to deteriorate. Therefore, selecting samples with high reliability for training will improve the model's recognition effect. Therefore, a dangerous behavior control method that can accurately detect fishing rod targets is needed. Summary of the Invention

[0006] The purpose of this invention is to design a method for controlling dangerous fishing behaviors based on reinforcement learning and deformable Transformer to solve the problem of inaccurate detection of small and slender fishing rods in the existing technology.

[0007] The present invention solves the above technical problems through the following technical solutions:

[0008] A method for controlling dangerous fishing behaviors based on reinforcement learning and deformable Transformers includes the following steps:

[0009] S1. Obtain a dangerous behavior control dataset, manually annotate it, and divide it into a dangerous behavior control training set, a dangerous behavior control validation set, and a dangerous behavior control test set according to a certain ratio;

[0010] S2. Build a deformable Transformer network fishing rod detection model for detecting fishing rods and a reinforcement learning-based sample fitness evaluation model for evaluating the fitness of training samples;

[0011] S3. Initialize the sample fitness evaluation model based on reinforcement learning, and use the sample fitness evaluation model based on reinforcement learning to evaluate the sample fitness of the dangerous behavior control training set;

[0012] S4. Send the sampled training sample set to the deformable Transformer network fishing rod detection model for training, and perform performance evaluation on the dangerous behavior control validation set, thereby obtaining the trained deformable Transformer network fishing rod detection model, and perform performance evaluation on the trained deformable Transformer network fishing rod detection model on the dangerous behavior control validation set;

[0013] S5. Update the sample fitness evaluation model based on reinforcement learning through the maximum entropy reinforcement learning algorithm. When the number of update iterations reaches the set threshold, save the parameters of the deformable Transformer network fishing rod detection model for dangerous behavior identification.

[0014] Furthermore, the deformable Transformer network fishing rod detection model described in step S2 includes: a multi-scale spatial depth convolution module, a deformable Transformer encoder, a deformable Transformer decoder, and a feedforward network; the multi-scale spatial depth convolution module includes: ResNet-50, an SCNN network, and a controllable direction filter; the operating mechanism of the deformable Transformer network fishing rod detection model is as follows: multi-scale features are extracted from the sample ResNet-50 trained with the fishing rod detection model output by the sample fitness evaluation model based on reinforcement learning to obtain a multi-scale feature map, and the multi-scale feature map is subjected to spatial depth convolution through the SCNN network, and then multi-directional spatial features are extracted through a controllable direction filter to obtain a multi-scale spatial feature map; encoding is performed through the deformable Transformer encoder to obtain an encoded feature vector, which is then sent to the deformable Transformer decoder for decoding to obtain a decoded feature vector and classified and regressed into coordinate frames through two feedforward networks respectively. At the same time, the parameters of the fishing rod detection model are updated based on the loss calculated according to the predicted coordinate frame and the actual coordinate frame, and the reward is calculated based on the performance evaluation results on the validation set as the parameter update of the reinforcement learning network based on maximum entropy.

[0015] Furthermore, the sample fitness evaluation model based on reinforcement learning in step S2 includes: an actor network, a critic network, a target critic network, and an experience pool. The operation mechanism of the sample fitness evaluation model based on reinforcement learning is as follows:

[0016] 1) Extract feature maps s containing candidate straight line features from training samples based on line detector;

[0017] 2) The Actor network evaluates the sample fitness of the feature graph s of each training sample, outputs the selected probability action a, and samples the training sample corresponding to s to obtain a training sample with linear feature information;

[0018] 3) The selected training samples are trained using the deformable Transformer network fishing rod detection model, and performance evaluation is performed on the dangerous behavior control validation set. The performance evaluation results are compared with the moving average of the previous performance evaluation results, and the change in the performance evaluation results is obtained as a reward r. The maximum entropy-based Actor network is updated using the reward r until the Actor network can correctly select samples trained by the fishing rod detection model to obtain the optimal detection results.

[0019] Furthermore, the method of initializing the sample fitness evaluation model based on reinforcement learning in step S3 and performing sample fitness evaluation on the dangerous behavior control training set through the sample fitness evaluation model based on reinforcement learning is specifically as follows:

[0020] 1) Initialize the Actor network parameter φ, two Critic network parameters θ1, θ2, and two target Critic network parameters Initialize experience pool

[0021] 2) The dangerous behavior control training set is evaluated for sample fitness through the Actor network to obtain the sample fitness evaluation probability and sample it through a sampler based on multinomial distribution to obtain the sampled training sample set, which is recorded as the current action a.

[0022] Furthermore, the method described in step S4 is to feed the sampled training sample set into the deformable Transformer network fishing rod detection model for training, and to perform performance evaluation using the dangerous behavior control validation set, thereby obtaining the trained deformable Transformer network fishing rod detection model as follows:

[0023] 1) The dangerous behavior control dataset is fed into a multi-scale spatial deep convolutional deformable self-attention module to extract multi-scale spatial features and obtain the multi-scale spatial features of the fishing rod target;

[0024] 2) Inputting the multi-scale spatial features into an encoder for encoding to obtain encoded fishing rod features;

[0025] 3) Inputting the encoded fishing rod features into a decoder for decoding to obtain output fishing rod features;

[0026] 4) The output fishing rod features are classified and regressed through a feedforward network to obtain the fishing rod category and fishing rod bounding box, which are used as the output results of fishing rod detection;

[0027] 5) The output results of the fishing rod detection and the true label are calculated through the focal loss function, and the focal loss function is supervised and solved using Hungarian matching to update the parameters of the fishing rod detection model. When the number of training times of the fishing rod detection model reaches the preset threshold, the trained fishing rod detection model is obtained.

[0028] Furthermore, the process of updating the sample fitness evaluation model based on reinforcement learning by maximum entropy reinforcement learning in step S5 is specifically as follows:

[0029] 1) Evaluate the current action a through two target critic networks to obtain two target action values. Take the minimum value as the current target action value, and use the gradient descent method to update the critic network parameters θ1 and θ2;

[0030] 2) At the same time, the current action a is evaluated by two critic networks to obtain two action values. The minimum value is taken as the current action value, and the parameter φ of the Actor network is updated based on the maximum entropy.

[0031] 3) Update the parameters of the two target critic networks through soft update The calculation method is shown in formula (1):

[0032]

[0033] Here, τ represents the smoothing factor.

[0034] Furthermore, the method for identifying dangerous behaviors in step S5 is specifically as follows:

[0035] 1) Define dangerous behavior actions, including: non-dangerous behavior, potentially dangerous behavior, and dangerous behavior; define a cache queue QUEUE, and use it to store posture information of consecutive t frames;

[0036] 2) Send a frame of image from the dangerous behavior video to be detected into the trained fishing rod detection model to perform fishing rod detection and obtain the fishing rod target in the current frame image;

[0037] 3) At the same time, the current frame image is used to locate the fisherman through YOLOv3 to obtain the fisherman target in the current frame image;

[0038] 4) Track the fisherman target through Deep Sort tracking, obtain the fisherman target ID, and lock the target based on the detection information of consecutive frames;

[0039] 5) The locked target is estimated through human posture estimation based on HRNet to obtain the key point information of the fisherman's body;

[0040] 6) Extract the key point information on the arm of the fisherman from the key point information of the human body, and use formula (2) to calculate the arm state cosθ of the target person in the current frame image, and save it in the cache queue QUEUE as posture information;

[0041]

[0042] Where θ represents the angle between the forearm and the upper arm, d1 and d2 are the lengths of the forearm and upper arm, and d3 represents the length of the opposite side.

[0043] 7) By obtaining the posture information of t consecutive frames in the cache queue QUEUE and using the arc cosine function to calculate the range of the angle θ of the dangerous actor's arm, the action A of the dangerous actor's arm is calculated using formula (3) to determine the type of dangerous behavior;

[0044]

[0045] Among them, when A = 0, it means that the arm has no obvious movement, when A = 1, it means that the arm has obvious movement, and the change of θ from large to small indicates the hand-raising movement, otherwise it indicates the hand-shaking movement;

[0046] 8) Calculating the Euclidean distance between the fishing rod target in the current frame image and the angler target in the current frame image through the center point to obtain a matching result of the fishing rod target and the angler;

[0047] 9) If a fishing rod target is detected and accompanied by a hand-raising or hand-swinging action, that is, A=1, it is determined to be a rod-raising or rod-swinging action, and a high-voltage danger alarm is issued; otherwise, that is, when A=0 and a fishing rod target is detected, it is determined to be a potentially dangerous behavior; if no fishing rod target is detected, it is determined to be no dangerous behavior.

[0048] The advantages of the present invention are:

[0049] The present invention performs fitness evaluation on each training sample through a sample fitness evaluation model based on reinforcement learning, selects samples with high fitness and sends them to a deformable Transformer fishing rod detection model for training, then extracts multi-scale spatial features of slender objects through the deformable Transformer fishing rod detection model to perform fishing rod detection, and finally updates the sample fitness evaluation model based on reinforcement learning through a maximum entropy reinforcement learning algorithm, thereby improving the detection accuracy of the model; the method of the present invention can effectively extract the slender structural features of the fishing rod, improves the model's ability to recognize slender features, thereby improving the recognition effect of the fishing rod detection model, and can realize the detection of dangerous behaviors in high-voltage environments, ensuring the safety of people in high-voltage environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] Figure 1This is a flowchart of the operation of the method for controlling dangerous fishing behaviors based on reinforcement learning and deformable Transformer according to an embodiment of the present invention;

[0051] Figure 2 This is a flowchart of the operation of a sample fitness evaluation model based on reinforcement learning and maximum entropy in a method for controlling dangerous fishing behaviors based on reinforcement learning and a deformable Transformer according to an embodiment of the present invention;

[0052] Figure 3 This is a flowchart of dangerous behavior identification of a dangerous fishing behavior control method based on reinforcement learning and deformable Transformer in an embodiment of the present invention;

[0053] Figure 4 This is a diagram showing the training results of a fishing rod detection model for a method for controlling dangerous fishing behaviors based on reinforcement learning and deformable Transformer according to an embodiment of the present invention. DETAILED DESCRIPTION

[0054] To make the objectives, technical solutions, and advantages of the embodiments of the present invention more clear, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0055] The technical solution of the present invention is further described below with reference to the accompanying drawings and specific embodiments:

[0056] Example 1

[0057] like Figure 1 As shown in the figure, a method for controlling dangerous fishing behaviors based on reinforcement learning and deformable Transformer includes the following steps:

[0058] Step 1: Obtain a dangerous behavior control dataset, and after labeling, divide it into a dangerous behavior control training set, a dangerous behavior control validation set, and a dangerous behavior control test set according to a certain ratio;

[0059] Step 2: Construct a fishing rod detection model for detecting fishing rods and a sample fitness evaluation model based on reinforcement learning for evaluating the fitness of training samples;

[0060] like Figure 1As shown, the fishing rod detection model consists of: a multi-scale spatial depth convolution module, a deformable Transformer encoder, a deformable Transformer decoder, and a feedforward network; the multi-scale spatial depth convolution module includes: ResNet-50, SCNN network, and a controllable direction filter;

[0061] The operating mechanism of the fishing rod detection model is as follows: multi-scale features are extracted from the sample ResNet-50 trained by the fishing rod detection model output by the reinforcement learning-based sample fitness evaluation model to obtain a multi-scale feature map. After the multi-scale feature map is spatially deep convolved through the SCNN network, multi-directional spatial features are extracted through a controllable directional filter to obtain a multi-scale spatial feature map; it is encoded through a deformable Transformer encoder to obtain an encoded feature vector, which is then sent to a deformable Transformer decoder for decoding. The decoded feature vector is obtained and classified and regressed to the coordinate frame through two feedforward networks. At the same time, the parameters of the fishing rod detection model are updated based on the loss calculated based on the predicted coordinate frame and the actual coordinate frame, and the reward is calculated based on the performance evaluation results on the validation set as the parameter update of the reinforcement learning network based on maximum entropy.

[0062] The Deformable Transformer extends the Deformable Attention module with multi-scale feature maps, enabling faster model convergence and better detection of low-resolution objects, thus enabling better discrimination between detected objects and background. However, due to inherent structural flaws, the input data loses information about the relative positions of objects, hindering the extraction of the long and narrow features of fishing rods. The Multi-Scale Spatial Deep Convolution module transfers spatial information through a SCNN network, enabling more effective learning of spatial relationships and making it ideal for feature extraction of long and narrow structures such as fishing rods. However, the SCNN is relatively limited in extracting spatial information in the four directions of downward, upward, right, and left, which does not reflect the multi-directional characteristics of fishing rods in fishing scenes. Therefore, a steerable directional filter is implemented. Through a linear combination of a set of basis filters, filtering in any direction is performed, thereby extracting rod boundary information in different directions in fishing behavior images. The Multi-Scale Spatial Deep Convolution module combines the SCNN network and steerable directional filters based on the multi-scale feature maps to achieve multi-scale spatial feature extraction for long and narrow objects such as fishing rods.

[0063] Targeting targets with long and narrow structures such as fishing rods, a multi-scale spatial deep convolution module is proposed. Based on the multi-scale feature map extracted by ResNet-50, SCNN and controllable directional filters are combined to extract spatial information in different directions to obtain multi-scale spatial feature maps, thereby improving the model's feature extraction ability for long and narrow structures.

[0064] like Figure 2 Figure 2 shows the structure of a reinforcement learning-based sample fitness evaluation model, consisting of an actor network, two critic networks, two target critic networks, an experience pool, and a line detector. The model operates as follows: First, the line detector extracts a feature map s containing candidate line features from the training sample. Second, the actor network performs sample fitness evaluation on each training sample's feature map s, outputting a probabilistic action a to sample the training sample corresponding to s, thereby obtaining training samples rich in line feature information. The selected training samples are then trained using a deformable Transformer network-based fishing rod detection model. Performance evaluation is then performed on a dangerous behavior control validation set. The performance evaluation results are compared with the moving average of the previous performance evaluation results, and the change in performance evaluation results is calculated as a reward, r. This reward is then used to update the maximum entropy-based actor network until the actor network correctly selects samples suitable for training the fishing rod detection model, achieving optimal detection results.

[0065] Step 3: Initialize the Actor network parameter φ, two Critic network parameters θ1, θ2, and two target Critic network parameters Initialize experience pool

[0066] Step 4: Extract straight line features from the dangerous behavior control training set through the line detector as the current state s; send the current state s to the Actor network for sample fitness evaluation, obtain the sample fitness evaluation probability, and sample it through a sampler based on multinomial distribution to obtain the sampled training sample set, which is recorded as the current action a;

[0067] Step 5: Send the sampled training sample set to the fishing rod detection model for training, and perform performance evaluation using the dangerous behavior control validation set to obtain a trained fishing rod detection model;

[0068] Step 5.1: The dangerous behavior control dataset is fed into the multi-scale spatial deep convolution deformable self-attention module to extract multi-scale spatial features and obtain the multi-scale spatial features of the fishing rod target.

[0069] Step 5.2: Input the multi-scale spatial features into the encoder for encoding to obtain the encoded fishing rod features;

[0070] Step 5.3, input the encoded fishing rod features into the decoder for decoding to obtain output fishing rod features;

[0071] Step 5.4: The output fishing rod features are classified and regressed through a feedforward network to obtain the fishing rod category and fishing rod bounding box, which are used as the output results of fishing rod detection.

[0072] Step 5.5: Calculate the loss of the output result of the fishing rod detection and the true label through the focal loss function, and use Hungarian matching to supervise the focal loss function to update the parameters of the fishing rod detection model. When the number of times the fishing rod detection model is trained reaches a preset threshold, obtain the trained fishing rod detection model and continue to step 6; otherwise, return to step 5.1 and continue;

[0073] Step 6: Perform a performance evaluation on the trained fishing rod detection model on the dangerous behavior control validation set to obtain a performance evaluation result and compare it with the moving average of the previous performance evaluation results to obtain a performance change result as a reward r;

[0074] Step 7: Store (s, a, r, s′) in the experience pool D and update the Actor network through maximum entropy-based reinforcement learning.

[0075] Step 7.1: Evaluate the current action a through two target critic networks to obtain two target action values. Take the minimum value as the current target action value and use the gradient descent method to update the critic network parameters θ1 and θ2.

[0076] Step 7.2: Evaluate the current action a through two critic networks at the same time, obtain two action values, take the minimum value as the current action value, and update the parameter φ of the actor network based on the maximum entropy;

[0077] Step 7.3: Update the parameters of the two target critic networks by soft updating The calculation method is shown in formula (1):

[0078]

[0079] Where τ represents the smoothing factor;

[0080] The present invention adopts maximum entropy reinforcement learning to evaluate the fitness of training samples and select the optimal samples that meet the needs of model training, thereby improving the detection accuracy of the model.

[0081] Step 8. When the number of iterations of the Actor network update reaches the threshold, save the fishing rod detection model parameters; otherwise, return to step 3 to continue training;

[0082] Step 9: Identify dangerous behaviors, such as Figure 3 The specific steps are as follows:

[0083] Step 9.1, define dangerous behavior actions, including: non-dangerous behavior, potentially dangerous behavior, and dangerous behavior; define a cache queue QUEUE, and use it to store posture information of consecutive t frames;

[0084] No dangerous behavior means that a fisherman is detected, but the fishing rod in his hand is not detected, indicating that no fishing is taking place and there is no danger.

[0085] Potentially dangerous behavior means that a fisherman has been detected, and the fishing rod in his hand has been detected, indicating that he is fishing, but no lifting or casting of the rod has been detected, indicating a potential danger.

[0086] Dangerous behavior means that a fisherman is detected, and the fishing rod in the person's hand is detected, indicating that fishing is in progress. At the same time, the action of lifting or throwing the rod is detected, indicating dangerous behavior.

[0087] Step 9.2: Send a frame of image from the dangerous behavior video to be detected into the trained fishing rod detection model to perform fishing rod detection and obtain the fishing rod target in the current frame image;

[0088] Step 9.3: At the same time, the current frame image is used to locate the fisherman through YOLOv3 to obtain the fisherman target in the current frame image;

[0089] Step 9.4: Track the fisherman target through Deep Sort tracking to obtain the fisherman target ID, and lock the target based on the detection information of consecutive frames;

[0090] Step 9.5: Use HRNet-based human pose estimation to estimate the pose of the locked target and obtain the key point information of the fisherman's body;

[0091] Step 9.6: Extract the key point information on the arm of the fisherman from the key point information of the human body, and use formula (2) to calculate the arm state cosθ of the target person in the current frame image, and save it as posture information in the cache queue QUEUE;

[0092]

[0093] Among them, θ represents the angle between the forearm and the upper arm, d1 and d2 are the lengths of the forearm and upper arm, and d3 represents the length of the opposite side.

[0094] Step 9.7: Obtain the posture information of t consecutive frames in the cache queue QUEUE and use the arccosine function to find the range of the angle θ of the dangerous actor's arm, thereby using formula (3) to calculate the action A of the dangerous actor's arm to determine the type of dangerous behavior;

[0095]

[0096] Among them, when A=0, it means that the arm has no obvious movement, when A=1, it means that the arm has obvious movement, and the change of θ from large to small indicates the hand-raising movement, otherwise it indicates the hand-shaking movement.

[0097] Step 9.8, calculating the Euclidean distance between the fishing rod target in the current frame image and the fisherman target in the current frame image through the center point to obtain a matching result of the fishing rod target and the fisherman;

[0098] Step 9.9: If a fishing rod target is detected and accompanied by a hand-raising or hand-swinging action, that is, A=1, it is determined to be a rod-raising or rod-swinging action, and a high-voltage danger alarm is issued; otherwise, that is, when A=0 and a fishing rod target is detected, it is determined to be a potentially dangerous behavior; if no fishing rod target is detected, it is determined to be no dangerous behavior.

[0099] like Figure 4 As shown in FIG, the training result diagram of the fishing rod detection model of the present invention is that the fishing rod detection model of the present invention converges after about 30 iterations. Compared with the current target detection network YOLOX with better performance, the present invention is superior to YOLOX in convergence speed and accuracy. This shows the advantages of the present invention in the detection of targets with narrow and long structures such as fishing rods.

[0100] This paper uses a spatial deep convolution method to construct a module with multi-scale spatial deep convolution attention, extracting the multi-scale spatial features of slender objects. It then proposes a multi-scale deformable Transformer network model with spatial deep convolution for fishing rod detection. To select samples that meet the requirements for model training, an Actor network performs fitness evaluation on each training sample, selects samples with high fitness for training, and finally uses maximum entropy-based reinforcement learning to update the Actor decision network. By combining reinforcement learning and a deformable Transformer fishing rod detection model with spatial deep convolution with logical judgment of dangerous behaviors, a more reliable dangerous behavior management and control system is achieved.

[0101] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit the same. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the various embodiments of the present invention.

Claims

1. A method for controlling dangerous fishing behaviors based on reinforcement learning and deformable Transformer, characterized by: The following steps are involved: S1. Obtain a dangerous behavior control dataset, manually annotate it, and divide it into a dangerous behavior control training set, a dangerous behavior control validation set, and a dangerous behavior control test set according to a certain ratio; S2. Build a deformable Transformer network fishing rod detection model for detecting fishing rods and a reinforcement learning-based sample fitness evaluation model for evaluating the fitness of training samples; The deformable Transformer network fishing rod detection model includes: a multi-scale spatial depth convolution module, a deformable Transformer encoder, a deformable Transformer decoder, and a feedforward network; the multi-scale spatial depth convolution module includes: ResNet-50, an SCNN network, and a controllable directional filter; the operating mechanism of the deformable Transformer network fishing rod detection model is as follows: multi-scale features are extracted from the sample ResNet-50 trained with the fishing rod detection model output by the sample fitness evaluation model based on reinforcement learning to obtain a multi-scale feature map, and the multi-scale feature map is subjected to spatial depth convolution through the SCNN network, and then multi-directional spatial features are extracted through a controllable directional filter to obtain a multi-scale spatial feature map; encoding is performed through the deformable Transformer encoder to obtain an encoded feature vector, which is then sent to the deformable Transformer decoder for decoding to obtain a decoded feature vector and is classified and regressed in the coordinate frame respectively through two feedforward networks; at the same time, the parameters of the fishing rod detection model are updated according to the loss calculated based on the predicted coordinate frame and the actual coordinate frame, and the reward is calculated based on the performance evaluation results on the validation set as the reinforcement learning network update parameter based on maximum entropy; The sample fitness evaluation model based on reinforcement learning includes: Actor network, Critic network, target Critic network, and experience pool; S3. Initialize the sample fitness evaluation model based on reinforcement learning, and use the sample fitness evaluation model based on reinforcement learning to evaluate the sample fitness of the dangerous behavior control training set; S4. Send the sampled training sample set to the deformable Transformer network fishing rod detection model for training, and perform performance evaluation on the dangerous behavior control validation set, thereby obtaining the trained deformable Transformer network fishing rod detection model, and perform performance evaluation on the trained deformable Transformer network fishing rod detection model on the dangerous behavior control validation set; S5. Update the sample fitness evaluation model based on reinforcement learning through the maximum entropy reinforcement learning algorithm. When the number of update iterations reaches the set threshold, save the parameters of the deformable Transformer network fishing rod detection model for dangerous behavior identification.

2. The method for controlling dangerous fishing behaviors based on reinforcement learning and deformable transformer according to claim 1 is characterized in that: The operating mechanism of the sample fitness evaluation model based on reinforcement learning is as follows: 1) Extract feature maps s containing candidate straight line features from training samples based on line detector; 2) The Actor network evaluates the sample fitness of the feature graph s of each training sample and outputs the selected probabilistic action a to sample the training sample corresponding to s to obtain a training sample with linear feature information. 3) The selected training samples are trained using the deformable Transformer network fishing rod detection model, and performance evaluation is performed on the dangerous behavior control validation set. The performance evaluation results are compared with the moving average of the previous performance evaluation results, and the change in the performance evaluation results is obtained as a reward r. The maximum entropy-based Actor network is updated with the reward r until the Actor network can correctly select samples trained by the fishing rod detection model to obtain the optimal detection results.

3. The method for controlling dangerous fishing behaviors based on reinforcement learning and deformable transformer according to claim 2 is characterized in that: The method of initializing the sample fitness evaluation model based on reinforcement learning in step S3 and performing sample fitness evaluation on the dangerous behavior control training set through the sample fitness evaluation model based on reinforcement learning is as follows: 1) Initialize Actor network parameters , two critic network parameters , , two target critic network parameters , , initialize the experience pool ; 2) The dangerous behavior control training set is evaluated for sample fitness through the Actor network, and the sample fitness evaluation probability is obtained and sampled through a sampler based on multinomial distribution to obtain the sampled training sample set, which is recorded as the current action a .

4. The method for controlling dangerous fishing behaviors based on reinforcement learning and deformable transformer according to claim 3 is characterized in that: The method described in step S4 is as follows: sending the sampled training sample set to the deformable Transformer network fishing rod detection model for training, and performing performance evaluation using the dangerous behavior control validation set to obtain the trained deformable Transformer network fishing rod detection model: 1) The dangerous behavior control dataset is fed into a multi-scale spatial deep convolutional deformable self-attention module to extract multi-scale spatial features and obtain the multi-scale spatial features of the fishing rod target; 2) Input the multi-scale spatial features into the encoder for encoding to obtain the encoded fishing rod features; 3) Input the encoded fishing rod features into the decoder for decoding to obtain the output fishing rod features; 4) The output fishing rod features are classified and regressed through a feedforward network to obtain the fishing rod category and fishing rod bounding box, which are used as the output results of fishing rod detection; 5) The output results of the fishing rod detection and the true label are calculated through the focal loss function, and the focal loss function is supervised and solved using Hungarian matching to update the parameters of the fishing rod detection model. When the number of training times of the fishing rod detection model reaches the preset threshold, the trained fishing rod detection model is obtained.

5. The method for controlling dangerous fishing behaviors based on reinforcement learning and deformable transformer according to claim 4 is characterized in that: The process of updating the sample fitness evaluation model based on reinforcement learning by maximum entropy reinforcement learning described in step S5 is as follows: 1) Change the current action a Through the two target critic networks, two target action values ​​are obtained, the minimum value is taken as the current target action value, and the critic network parameters are updated using the gradient descent method. 、 ; 2) At the same time, the current action a Through the evaluation of two Critic networks, two action values ​​are obtained, the minimum value is taken as the current action value, and the parameters of the Actor network are updated based on the maximum entropy. ; 3) Update the parameters of the two target critic networks through soft update 、 , the calculation method is shown in formula (1): （1） in, represents the smoothing factor.

6. The method for controlling dangerous fishing behaviors based on reinforcement learning and deformable transformer according to claim 5 is characterized in that: The method for identifying dangerous behaviors in step S5 is as follows: 1) Define dangerous behavior actions, including: non-dangerous behavior, potentially dangerous behavior, and dangerous behavior; define a cache queue QUEUE and use it to store continuous t The pose information of the frame; 2) Send a frame of image from the dangerous behavior video to be detected into the trained fishing rod detection model to detect the fishing rod and obtain the fishing rod target in the current frame of image; 3) At the same time, the current frame image is used to locate the fisherman through YOLOv3 to obtain the fisherman target in the current frame image; 4) Track the fisherman target through Deep Sort tracking, obtain the fisherman target ID, and lock the target based on the detection information of consecutive frames; 5) The locked target is estimated through human pose estimation based on HRNet to obtain the key point information of the fisherman's body; 6) Extract the key point information on the arm of the fisherman from the key point information of the human body, and use formula (2) to calculate the arm state of the target person in the current frame image , and saved as posture information in the cache queue QUEUE; （2） in, Indicates the angle between the forearm and the upper arm. 、 The length of the forearm and upper arm, Indicates the length of the opposite side; 7) By obtaining the cache queue QUEUE continuously t The posture information of the frame is obtained, and the arm angle of the dangerous actor is obtained by the arc cosine function The range of change of , thus using formula (3) to calculate the dangerous behavior person's arm movement A, to determine the type of dangerous behavior; （3） Among them, when A =0 means there is no obvious movement of the arm. A =1 indicates that the arm has obvious movement, and The change from large to small indicates the hand-raising action, otherwise it indicates the hand-shaking action; 8) Calculate the Euclidean distance between the fishing rod target in the current frame image and the fisherman target in the current frame image through the center point to obtain the matching result of the fishing rod target and the fisherman; 9) If the fishing rod target is detected and accompanied by a hand-raising or hand-shaking action, A =1, it is determined to be a stick-lifting action or a stick-swinging action, and a high-voltage danger alarm is issued; otherwise, A =0 and a fishing rod target is detected, it is judged as a potentially dangerous behavior; if no fishing rod target is detected, it is judged as a non-dangerous behavior.

Images