Method for constructing a potential danger warning model for unmanned boats, and warning method, equipment, and storage medium

Abstract

The present invention provides a method for constructing a potential danger warning model for an unmanned boat, and a warning method, device, and storage medium, which belong to the field of computer vision technology. By constructing a sequence position data set and an object danger value data set, the designed unmanned boat potential danger warning model is trained. The model includes an object trajectory extraction network and a danger value prediction network, and a dynamic learning rate adjustment and adaptive reweighting model training strategy are designed. The trained model is finally deployed to the computing terminal of the unmanned boat. First, the collected video image is processed to obtain the position sequence data of the surrounding objects, and then sent to the danger value prediction network to obtain the real-time danger value of each object around the unmanned boat. The present invention improves the automation level of potential danger warning and enhances the accuracy of danger warning in complex scenes. The newly designed unmanned boat potential danger warning model and training strategy improve the generalization ability and robustness of danger warning.

Description

Technical Field

[0001] The present invention belongs to the field of computer vision technology, and in particular relates to a method for constructing a potential danger warning model for an unmanned boat, and a warning method, device, and storage medium. Background Art

[0002] Unmanned boats play an increasingly important role in modern marine activities, mainly due to their outstanding advantages in safety, cost-effectiveness and efficiency. These unmanned boats can perform a variety of tasks in complex and dangerous marine environments, including environmental monitoring, marine resource exploration, border patrols and scientific research, significantly reducing dependence on personnel, potential risks and labor costs. In addition, unmanned boats usually have the ability to sail autonomously for a long time and can complete tasks in adverse weather conditions, improving the efficiency of marine resource utilization.

[0003] The necessity of early warning of dangers around unmanned boats cannot be ignored. An effective early warning system can identify and assess potential threats in real time, such as other ships and marine obstacles, thereby improving the safety of unmanned boats and their operating environment. This early warning mechanism can not only reduce the risk of accidents and ensure that unmanned boats are not disturbed when performing their missions, but also support real-time decision-making and emergency response, and flexibly deal with complex situations. By improving monitoring and law enforcement capabilities, these systems can also more effectively combat illegal activities and maintain marine order.

[0004] In general, the effective operation of unmanned boats and the surrounding danger warning system complement each other, jointly promoting the sustainable use of marine resources and ecological protection, and ensuring the safety and efficiency of future marine activities.

[0005] Some existing warning methods for potential dangers of unmanned boats are as follows:

[0006] Danger warning method based on automatic identification system: The potential danger warning method of unmanned boats based on the automatic identification system (AIS) builds a comprehensive view of maritime traffic by receiving information from surrounding ships in real time, including data such as location, speed and heading. This system can dynamically monitor the movement status of other ships, help unmanned boats to timely predict possible collision risks, and make corresponding decision adjustments. By effectively integrating AIS data, unmanned boats can not only improve their own navigation safety, but also optimize route planning to ensure safe operation in complex marine environments. However, this system cannot predict and operate marine objects such as floating objects and marine garbage that cannot send information, and its scope of application of danger warning is limited.

[0007] Danger warning method based on radar system: The potential danger warning method for unmanned boats based on radar system monitors objects in the surrounding environment, including other ships, obstacles and ocean features, in real time by emitting electromagnetic waves and receiving reflected signals. The system can provide high-precision distance and speed information to help unmanned boats identify potential threats and calculate relative motion, thereby achieving timely collision warning. Combined with data processing algorithms, the radar system can effectively filter interference signals, improve the accuracy of target detection, and ensure the safe navigation of unmanned boats in complex and dynamic marine environments. However, the cost of radar systems is high, and there is a problem of insufficient resolution when monitoring over long distances, which affects the detection ability of small objects or distant targets and reduces the reliability of the overall warning. Summary of the invention

[0008] In view of the above problems, the first aspect of the present invention provides a method for constructing a potential danger warning model of an unmanned boat, comprising the following steps:

[0009] S1, collect video image data containing the actual movement of objects at sea, obtain the sequence position data of each object in the image video, and build a sequence position data set At the same time, the hazard value of the moving objects at sea included in the video image data is calculated by combining prior knowledge and the object hazard value calculation formula to obtain the object hazard value dataset. ;

[0010] S2, constructing a potential danger warning model for unmanned boats, including object trajectory extraction network and danger value prediction network;

[0011] The object trajectory extraction network includes a target detection module and a target matching module. The target detection module performs target detection on each video frame of the input image data based on self-attention and a multi-layer convolutional network to obtain the position data of the object in the current video frame. The target matching module matches the same object in different video frames and combines the position data of the same object in different video frames to obtain the sequence position data of the object.

[0012] The danger value prediction network includes a feature vector encoding module, a multi-head attention mechanism module and a prediction output module, which is used to take the sequence position data of each object as input, and after being processed by the danger value prediction network, predict the danger value of the object to help realize the potential danger warning of the unmanned boat;

[0013] S3, using the sequence position dataset Train the object trajectory extraction network using the object danger value dataset The hazard value prediction network is trained. Based on the designed training strategy, a larger loss function value is assigned to the training data with poor training results by calculating the adaptive weights to help the model pay more attention to the sample data with poor training results. Finally, a trained unmanned boat potential hazard warning model is obtained.

[0014] Preferably, the sequence position data set is constructed The specific process is:

[0015] For the initial image data obtained The various objects contained in the video are calibrated frame by frame. In each frame of the video, the labeling software is used to label the corresponding objects. To obtain the image annotation dataset ;

[0016] Image annotation dataset The objects marked on each frame are defined as , , combine the pixel coordinates of the rightmost, topmost, bottommost, and leftmost pixels of the object to get the object In the Pixel position data on the frame image , :

[0017]

[0018] in, The four pixels of the object, namely the leftmost, topmost, bottommost, and rightmost, are represented in sequence. Coordinates and coordinate, Indicates the total number of frames of the image;

[0019] Image annotation dataset Medium Object The pixel position data on each frame of the image is combined to obtain the object Complete sequence position data :

[0020]

[0021] in, Representing objects In the Pixel position data on the frame image;

[0022] Calculate the image annotation dataset in sequence Sequential position data of each object in , and combine the sequence position data of each object to obtain the object sequence position data set .

[0023] Preferably, construct an object danger value dataset The specific process is: using subjective judgment and formula calculation to define the object The danger value of the object in the video The risk value of the subjective judgment is evaluated to obtain the subjective judgment risk value, and this subjective judgment risk value is defined as ; For objects that appear in the field of view of this remote-controlled unmanned boat, record the time when it first appears in its camera , and record the time when the unmanned boat makes the turning operation , and according to Calculate the operation response time :

[0024]

[0025] The steering amplitude of the unmanned boat’s steering operation is recorded as , use the formula to calculate the object Operational risk value ;

[0026]

[0027] Based on the obtained operational risk value and subjective risk value Perform weighted summation to obtain the object The final corresponding risk value ;

[0028] Compute image annotation dataset The danger value of all objects in , and combine the calculated danger values ​​to obtain the object danger value data set .

[0029] Preferably, the target detection module includes a trapezoidal feature extraction unit, a self-attention mechanism unit and a detection unit;

[0030] The trapezoidal feature extraction unit uses the feature extraction block as the basic module of the feature extraction operation. The input of this feature extraction block first passes through two sequentially connected convolutional layers for feature extraction operation, and then passes through The function is processed nonlinearly, followed by a convolutional layer and a Function, input image data First, the BBS block of a feature extraction block is processed by two convolutional layers. The BBS blocks are connected from top to bottom and repeated three times to form the main path of the trapezoidal feature extraction unit; the feature extraction block at the top layer is connected to the right with a feature extraction block to form a branch path to extract global features. The feature extraction block in the middle is connected to the right with two feature extraction blocks to form branch path 2 to extract local features The feature block at the bottom layer is connected to the right with three feature extraction blocks to form a branch path to extract detail features , the specific calculation formula is as follows;

[0031]

[0032]

[0033]

[0034] in Represents the feature extraction block operation function, The operation function of the whole block;

[0035] The self-attention mechanism unit extracts the global features , local features and detail features Perform adaptive weight fusion to obtain fusion features ;

[0036]

[0037] in, Represents the self-attention mechanism module operation function;

[0038] The detection unit first uses two average pooling layers to fusion features The number of parameters of the entire module is processed, and then the intermediate features are obtained through two BN normalization layers. :

[0039]

[0040] in, represents the average pooling layer operation function, Represents the BN normalization layer operation function;

[0041] The intermediate feature Feaz then passes through two fully connected layers and The final target detection result is obtained after activation function processing :

[0042] After performing target detection on this video frame, record the leftmost, rightmost, topmost, and bottommost pixels of each object in the detection result. Coordinates, and combine these coordinates to represent the predicted position data of the object ;

[0043]

[0044] in, Indicates the predicted The first The position data of an object, Indicates The first The leftmost, rightmost, topmost, and bottommost pixels of an object coordinate.

[0045] Preferably, the input of the target matching module is the target detection result matrix of the same target in the wth video frame: , the output is the detection result matrix of this target in other video frames ; The target matching module consists of two fully connected layers plus a The activation function consists of:

[0046]

[0047] in, represents the target matching network operation function, represent Operation function, Represents the fully connected layer operation function;

[0048] The position data of the same object in different video frames are combined to represent the predicted Sequential position data of objects :

[0049]

[0050]

[0051] in, Indicates that the obtained The first The location data of the object, Represents the predicted sequence position data set of all objects.

[0052] Preferably, the risk value prediction network includes a feature vector encoding module, a multi-head attention mechanism module and a prediction output module;

[0053] The feature vector encoding module is based on the obtained sequence position data , for each sequence data Use two interconnected fully connected layers to vector encode this sequence data and obtain the encoded feature data ;

[0054] The multi-head attention mechanism module performs attention on the feature data Perform multi-level feature extraction operations, including the following processes:

[0055] Feature matrix calculation: The feature Through matrix transformation, they are mapped to On three different feature matrices:

[0056]

[0057] in, Respectively represent matrices The matrix mapping function, Indicates Group The mapping parameters used when mapping the feature matrix; Respectively represent characteristics The mapped Group Feature matrix;

[0058] Attention coefficient calculation: Based on the obtained Two feature matrices, calculate their attention coefficients :

[0059]

[0060] in, express Activation function, represents the scaling factor, Indicates The attention coefficient corresponding to the group attention head;

[0061] Multi-head attention mechanism output feature vector calculation: The attention coefficient corresponding to the group attention head and Group Multiply the feature matrix to get Features output by the group attention head , the output features of the attention head are passed through a fully connected layer to obtain the output features after the multi-head attention mechanism ;

[0062] The prediction output module is based on the extracted features , this feature is fed into two fully connected layers and After further processing by the activation function, the corresponding danger value prediction result of this object can be obtained. .

[0063] Preferably, S3 is specifically:

[0064] Object trajectory extraction network loss function Designed for:

[0065]

[0066] in, represents the mean absolute error calculation function, represents the mean square error calculation function; Indicates The real sequence position data of objects, Indicates the predicted Sequential position data of objects;

[0067] Danger value prediction network loss function Designed for:

[0068]

[0069] in, Indicates the actual danger value of the object. Represents the object danger value predicted by the model, Represents the design of normalized operation function;

[0070] The training strategy is:

[0071] S31, using object trajectory to extract network loss function Calculate the loss value when training the object sequence position data extraction module , using the risk value to predict the network loss function Calculate the loss value when training the object danger value prediction model ;

[0072] S32, based on the calculated loss value and , calculate the training value weight of this sample training data :

[0073]

[0074] in, Indicates the calculation weight, based on the loss value and , the training value weight of each sample can be calculated ; When this sample is trained, if the calculated loss value is large, the model will have a poor learning effect on this sample;

[0075] S33, model parameter update: based on the calculated loss value and And the training value weight calculated by S4-3 Update the loss value of this sample after training to get the updated loss value and :

[0076]

[0077]

[0078] Using this updated loss value and use The optimizer extracts the model parameters of the object sequence position module And the model parameters in the object danger value prediction module After updating, samples with poor training effect will occupy a larger loss value in the overall loss using this updated loss value;

[0079] S34, learning rate update: based on the reinforcement loss value obtained after the first iteration and , using this enhanced loss value and Adjust the learning rate to achieve dynamic learning rate update:

[0080] in, represents the learning rate before the model is updated, represents the learning rate after the model is updated, express Normalization function;

[0081] S35, repeat steps S31 to S34, a total of After completing the number of iterations, the network model parameters of the two models are and The network model parameters finally used by the unmanned boat surrounding object danger value prediction model are saved to obtain the optimized unmanned boat surrounding object danger value prediction model.

[0082] The second aspect of the present invention provides a potential danger warning method for an unmanned boat, which deploys the potential danger warning model of the unmanned boat constructed by the construction method described in the first aspect to an unmanned boat terminal, and includes the following process:

[0083] Capture video images based on high-definition image acquisition equipment deployed on the unmanned boat;

[0084] The video image data is processed by frame division and input into the potential danger warning model of the unmanned boat to obtain the real-time danger value of each object around the unmanned boat;

[0085] The predicted danger values ​​of various objects around the unmanned boat are compared with the judgment threshold. When there are moving objects around the unmanned boat with a danger value greater than the judgment threshold, an alarm signal will be issued to help the unmanned boat achieve real-time potential danger warning.

[0086] The third aspect of the present invention provides an unmanned boat potential danger warning device, the device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of the unmanned boat potential danger warning model constructed by the construction method described in the first aspect; when the processor executes the computer execution program stored in the memory, the processor executes a method for warning of potential danger to an unmanned boat.

[0087] The fourth aspect of the present invention provides a computer-readable storage medium, which stores a computer program or instruction of an unmanned boat potential danger warning model constructed by the construction method described in the first aspect. When the program or instruction is executed by a processor, the processor executes a method for warning of potential danger of an unmanned boat.

[0088] Compared with the prior art, the present invention has the following beneficial effects:

[0089] 1. Improve the automation level of potential danger warning: This method first collects and annotates video data of the movement of objects at sea through high-precision cameras to generate a rich training data set, thereby improving the training efficiency and accuracy of the model. Secondly, the object trajectory extraction network uses self-attention and multi-layer convolutional networks to realize automatic detection and tracking of targets in video frames, greatly reducing manual intervention and improving the real-time and accuracy of detection. Constructing a hazard value prediction model based on a multi-head attention mechanism enables the system to automatically analyze the movement trajectories of different objects and evaluate their dangers, further enhancing the intelligence level of the warning. Dynamic learning rate adjustment and adaptive reweighting strategies ensure that the model can automatically adapt to different types of data, improve sensitivity to challenging samples, and optimize overall performance. Finally, by integrating image acquisition equipment with the warning model, real-time monitoring and hazard value assessment of the surrounding environment can be achieved, and alarms can be automatically triggered to ensure the safe driving of unmanned boats. This highly automated system significantly improves the responsiveness of unmanned boats in complex marine environments, reduces human errors and delays, and improves overall safety and efficiency;

[0090] 2. Enhance the intelligence level of danger warning: The design of the object trajectory extraction network effectively improves the intelligence level of danger warning by combining the self-attention mechanism and the multi-layer convolutional network. First, the target detection module can accurately identify objects in the input image and extract their position data in each video frame, thereby ensuring real-time monitoring of potential threats. Subsequently, the target matching module achieves continuous tracking by associating the same object in different video frames, thereby improving the understanding of the object's motion trajectory. And by combining the position information of different frames to generate the object's serial position data, the system can more comprehensively analyze the dynamic behavior of the object and identify abnormal situations in a timely manner. This intelligent trajectory extraction method provides unmanned boats with more accurate and timely danger warning capabilities, enhancing safety and decision-making efficiency in complex environments;

[0091] 3. Enhance the accuracy of danger warnings in complex scenarios: Combining the multi-head attention mechanism and the adaptive reweighted training strategy can significantly enhance the accuracy of danger warnings for unmanned boats in complex scenarios. The multi-head attention mechanism improves the model's perception of diverse threats by focusing on different feature subspaces, enabling it to effectively identify potentially dangerous objects in complex environments. The adaptive reweighting strategy dynamically adjusts the weights of training samples, especially strengthening the learning of difficult or important samples, making the model more sensitive when facing challenges. This combined approach not only improves adaptability to environmental changes, but also enhances the robustness of the early warning system, ensuring that unmanned boats can predict and respond to potential threats in complex waters in a timely and accurate manner;

[0092] 4. Improve the generalization ability and robustness of danger warning: When performing feature extraction, the present invention constructs a trapezoidal feature extraction unit to obtain image features at different levels. Not only the detailed features of the image are extracted, but also the global features of the image are extracted to enhance the robustness of feature extraction. And when defining the danger value, the subjective judgment danger value and the objective operation danger value calculated by the formula are integrated. This improves the generalization ability and robustness of the prediction when predicting the danger value of objects around the unmanned boat. Combined with the above operations, the generalization ability and robustness of the unmanned boat danger warning can be improved, and the unmanned boat can still maintain a good danger warning capability in unfamiliar marine scenes and harsh marine environments. BRIEF DESCRIPTION OF THE DRAWINGS

[0093] Figure 1 The figure is an overall logic block diagram of the potential danger warning method for unmanned boats of the present invention.

[0094] Figure 2 This is a network structure diagram for object trajectory extraction in the present invention.

[0095] Figure 3 This is a schematic diagram of the risk value prediction network structure of the present invention.

[0096] Figure 4 It is a schematic diagram of the framework of the overall implementation system of the present invention.

[0097] Figure 5 This is a simplified diagram of the structure of the potential danger warning equipment for unmanned boats. DETAILED DESCRIPTION

[0098] The invention will be further described below in conjunction with specific embodiments.

[0099] The overall technical route of the present invention is as follows Figure 1 shown.

[0100] Construction of the unmanned boat potential danger warning data set: A high-precision camera is deployed on the unmanned boat to collect video image data containing the actual movement of objects at sea, annotate the video image data, and save the sequence position data of each object in the image video. Then, the hazard value of the moving objects at sea included in the video image data is calculated by combining prior knowledge and the object hazard value calculation formula (this hazard value represents the degree of impact on the safe driving of the unmanned boat, and the larger the hazard value, the greater the impact on the safe driving of the unmanned boat).

[0101] Object trajectory extraction network design: In the present invention, a target detection module based on self-attention and multi-layer convolutional network is first constructed to perform target detection on each video frame of the input image data to obtain the position data of the object in the current video frame. Then, a target matching module is constructed to match the same object in different video frames to achieve the effect of target tracking. The position data of the same object in different video frames are combined to obtain the sequence position data of the object.

[0102] Danger value prediction network design: Based on the extracted sequence position data of each object, a danger value prediction network based on a multi-head attention mechanism is constructed, and the sequence position data of each object is used as the input of this danger value prediction model. After being processed by the model, the danger value of this object is predicted to help realize the potential danger warning of the unmanned boat;

[0103] Construction of training strategy based on dynamic learning rate adjustment and adaptive reweighting: During model training, for training data with poor training results, a larger loss function value is assigned to it by calculating adaptive weights. This helps the model pay more attention to sample data with poor training results, so that the model can keep its attention on challenging samples to enhance the overall performance of the model. The generalization ability of the model is ensured by adjusting the dynamic learning rate during model training;

[0104] Actual deployment and use of the model: The image acquisition device is deployed in the unmanned boat, and the surrounding environment image data collected by the device is sent to the unmanned boat potential danger warning model to obtain the danger value of each object around the unmanned boat. Combined with the alarm threshold set by the unmanned boat, the potential danger warning operation of the unmanned boat is finally realized.

[0105] 1. Construction of Unmanned Boat Potential Danger Warning Dataset

[0106] Image acquisition: Deploy multiple remotely controlled surface vessels to collect images in the target sea area where the potential danger prediction of unmanned boats is performed to obtain initial image data. To ensure the quality of the initial image collection, the size and appearance of this remote-controlled surface ship are designed to imitate the actual unmanned boats that carry out potential danger warnings, so as to improve the robustness of subsequent unmanned boat potential danger warnings;

[0107] Image calibration: For the initial image data The various objects contained in the video are calibrated frame by frame. On each frame of the video, the labeling software LableImg is used to label the corresponding objects. To obtain the image annotation dataset ;

[0108] Construction of object sequence position dataset: based on image annotation dataset Objects marked on each frame , . Because the object In a video frame, it is composed of multiple pixels to represent the object. The position of the object on the video frame is determined by combining the pixel coordinates of the rightmost, topmost, bottommost, and leftmost pixels of the object to obtain the pixel coordinates of the object. In the Pixel position data on the frame image , :

[0109]

[0110] in, The four pixels of the object, namely the leftmost, topmost, bottommost, and rightmost, are represented in sequence. Coordinates and coordinate.

[0111] Image annotation dataset Medium Object The pixel position data on each frame of the image is combined to obtain the object Complete sequence position data ;

[0112]

[0113] in, Representing objects In the Pixel position data on the frame image;

[0114] As described above, calculate the image annotation dataset in sequence Sequential position data of each object in , and combine the sequence position data of each object to obtain the object sequence position data set :

[0115] .

[0116] Construction of object hazard value dataset: In defining the hazard value of an object, subjective judgment and formula calculation are used to define the object. Organize experts to review image annotation datasets The video data in the video and the objects in the video The risk value of the subjective judgment is evaluated to obtain the subjective judgment risk value, and this subjective judgment risk value is defined as In addition, for objects that appear in the field of view of this remote-controlled unmanned boat, the time when they first appear in its camera is recorded. , and record the time when the unmanned boat makes the turning operation , and according to Calculate the operation response time :

[0117] .

[0118] The steering amplitude of the unmanned boat’s steering operation is recorded as The shorter the reaction time, the greater the turning range, indicating that the object The greater the threat to the unmanned boat, the greater the threat to the unmanned boat. Operational risk value :

[0119]

[0120] Based on the obtained operational risk value and subjective risk value Perform weighted summation to obtain the object The final corresponding risk value ;

[0121]

[0122] According to the above operations, calculate the image annotation dataset The danger value of all objects in , and combine the calculated danger values ​​to obtain the object danger value data set :

[0123] 。

[0124] 2. Construction of warning model for potential dangers of unmanned boats

[0125] 1. Object trajectory extraction network design

[0126] The object trajectory extraction network includes a target detection module and a target matching module. The target detection module includes a trapezoidal feature extraction unit, a self-attention mechanism unit, and a detection unit. The specific model structure is as follows: Figure 2 As shown, the object trajectory extraction module constructed by the present invention realizes the video image data Object target tracking in the image video. That is, the object sequence position data set can be obtained by successfully tracking the object in the image video. .

[0127] In the present invention, firstly, the video image data The object detection module is then used to detect different video frames to obtain the distribution of objects in different video frames. The matching algorithm is then used to match the same object in different video frames to achieve the distribution of objects in the entire video image data. Since the accuracy of target detection will directly affect the accuracy of target tracking, the present invention constructs a target detection network based on a multi-layer convolutional network and a channel cross-attention mechanism to ensure the accuracy of target detection.

[0128] Object detection module design:

[0129] (1) Construction of trapezoidal feature extraction unit: When performing feature extraction, fewer convolutional layer feature extraction operations tend to characterize global features, while more convolutional layer feature extraction operations tend to characterize detail features. In order to ensure the adequacy and comprehensiveness of feature extraction, a trapezoidal feature extraction unit is constructed in the present invention to extract global features, local features, and detail features of input data. Since the convolutional layer has good feature extraction capabilities, a feature extraction block is constructed in the trapezoidal feature extraction unit as the basic module for feature extraction operations. The input of this feature extraction block first passes through two sequentially connected convolutional layers for feature extraction operations, and then passes through The function is processed nonlinearly to further enhance the feature representation ability and alleviate the problem of gradient disappearance. Then it passes through a convolution layer and a Function can complete the construction of the feature extraction block:

[0130]

[0131] in, express Operation activation function, Represents the convolutional layer operation function, Represents the feature extraction block operation function.

[0132] In the trapezoidal feature extraction unit, input image data First, it is processed by the BBS block of two convolutional layers and one feature extraction block. The BBS blocks are connected from top to bottom and repeated three times to form the main path of the trapezoidal feature extraction unit:

[0133]

[0134] in, Represents the convolutional layer operation function, Represents the feature extraction block operation function, represent Block operation functions.

[0135] For the main path of the gradient feature extraction unit, the feature extraction block at the top layer is connected to a feature extraction block to the right to form a branch path to extract the global feature. The feature extraction block in the middle is connected to the right with two feature extraction blocks to form branch path 2 to extract local features The feature block at the bottom layer is connected to the right with three feature extraction blocks to form a branch path to extract detail features :

[0136]

[0137]

[0138]

[0139] (2) Self-attention mechanism unit construction: based on the global features extracted in the above process , local features and detail features In order to enhance the efficiency of using global features, local features and detail features, improve feature utilization and enhance the accuracy of target detection, the present invention uses the self-attention mechanism to perform adaptive weight fusion on the three features to obtain the fusion feature. :

[0140] .

[0141] (3) Detection unit construction: based on the fusion features obtained in the above process After further processing of this fusion feature, the corresponding target detection result can be obtained. In this detection unit, two average pooling layers are first used to process the fusion feature to reduce the number of parameters of the entire module, increase the calculation efficiency, and improve the robustness of target detection. Then, the intermediate feature is processed through two BN normalization layers. , in order to further improve the generalization ability of the model.

[0142]

[0143] in, represents the average pooling layer operation function, Represents the BN normalization layer operation function.

[0144] The intermediate feature Feaz then passes through two fully connected layers and The final target detection result is obtained after activation function processing :

[0145]

[0146] in, express Activation function, Represents the fully connected layer operation function.

[0147] And record the leftmost, rightmost, topmost, and bottommost pixels of each object in the detection result after the target detection operation is performed on this video frame. Coordinates, and combine these coordinates to represent the predicted position data of the object :

[0148]

[0149] in, Indicates the predicted The first The position data of an object, Indicates The first The leftmost, rightmost, topmost, and bottommost pixels of an object coordinate.

[0150] Target matching module design:

[0151] In the present invention, based on the target detection module, the video image data Perform frame-by-frame target detection to obtain the corresponding target detection result on each frame. Perform target matching on different video frames based on the target detection results, match the same target in different video frames together, and then track the target. In the present invention, a simple neural network for target matching is proposed to ensure the target matching effect. The input of this neural network is the target detection result matrix of the same target in the wth video frame. , the output is the detection result matrix of this target in other video frames . This target matching network Two fully connected layers plus one The activation function consists of:

[0152]

[0153] in, represents the target matching network operation function, represent Operation function, Represents the fully connected layer operation function.

[0154] Based on the constructed target matching module, the position data of the same object in different video frames are combined to represent the predicted Sequential position data of objects :

[0155]

[0156]

[0157] in, Indicates that the obtained The first The location data of the object, Represents the predicted sequence position data set of all objects.

[0158] 2. Danger value prediction network

[0159] The risk value prediction network includes a feature vector encoding module, a multi-head attention mechanism module and a risk value prediction module. The model structure is as follows: Figure 3 shown.

[0160] (1) Feature vector encoding: based on the obtained sequence position data , for each sequence data Use two interconnected fully connected layers to vector encode this sequence data and obtain the encoded feature data :

[0161]

[0162] in, Represents the fully connected layer operation function.

[0163] (2) Multi-head attention mechanism module: In order to further improve the accuracy of predicting the danger value of objects around the unmanned boat and enhance the unmanned boat's ability to perceive dangers in the surrounding environment, the present invention constructs a multi-head attention mechanism module to Perform multi-level feature extraction operations.

[0164] 1) Feature matrix calculation: transform the features Through matrix transformation, they are mapped to On three different feature matrices:

[0165]

[0166] in, Respectively represent matrices The matrix mapping function, Indicates Group The mapping parameters used in feature matrix mapping. In order to improve the global feature extraction capability, a total of 6 sets of feature matrices are designed in this invention. . Respectively represent characteristics The mapped Group Feature matrix.

[0167] 2) Attention coefficient calculation: Based on the obtained Two feature matrices, calculate their attention coefficients :

[0168]

[0169] in, express Activation function, represents the scaling factor, Indicates The attention coefficient corresponding to the group of attention heads.

[0170] 3) Calculation of output feature vector of multi-head attention mechanism: The attention coefficient corresponding to the group attention head and Group Multiplying the feature matrix can get Features output by the group attention head , because in the present invention, a total of 6 sets of feature matrices are designed, that is, there are six attention heads. In the present invention, the output features of the six attention heads are After a fully connected layer, the output features after the multi-head attention mechanism are obtained :

[0171]

[0172]

[0173] in, Represents the fully connected layer operation function. Based on the above description process, the entire operation process of the multi-head attention mechanism is as follows:

[0174]

[0175] in, Represents the multi-head attention mechanism operation function.

[0176] (3) Prediction output module design: based on the extracted features , this feature is fed into two fully connected layers and After further processing by the activation function, the corresponding danger value prediction result of this object can be obtained. .

[0177] .

[0178] 3. Model Training Strategy

[0179] Object trajectory extraction network loss function Designed for:

[0180]

[0181] in, represents the mean absolute error calculation function, represents the mean square error calculation function; Indicates The real sequence position data of objects, Indicates the predicted Sequential position data of objects;

[0182] Danger value prediction network loss function Designed for:

[0183]

[0184] in, Indicates the actual danger value of the object. Represents the object danger value predicted by the model, Represents the design of normalized operation function;

[0185] The training strategy is:

[0186] S1, using object trajectory to extract network loss function Calculate the loss value when training the object sequence position data extraction module , using the risk value to predict the network loss function Calculate the loss value when training the object danger value prediction model ;

[0187] S2, based on the calculated loss value and , calculate the training value weight of this sample training data :

[0188]

[0189] in, Indicates the calculation weight, based on the loss value and , the training value weight of each sample can be calculated ; When this sample is trained, if the calculated loss value is large, the model will have a poor learning effect on this sample;

[0190] S3, model parameter update: based on the calculated loss value and And the training value weight calculated by S4-3 Update the loss value of this sample after training to get the updated loss value and :

[0191]

[0192]

[0193] Using this updated loss value and use The optimizer extracts the model parameters of the object sequence position module And the model parameters in the object danger value prediction module After updating, samples with poor training effect will occupy a larger loss value in the overall loss using this updated loss value;

[0194] S4, learning rate update: based on the reinforcement loss value obtained after the first iteration and , using this enhanced loss value and Adjust the learning rate to achieve dynamic learning rate update:

[0195] in, represents the learning rate before the model is updated, represents the learning rate after the model is updated, express Normalization function;

[0196] S5, repeat steps S1 to S4, a total of iterations After completing the number of iterations, the network model parameters of the two models are and The network model parameters finally used by the unmanned boat surrounding object danger value prediction model are saved to obtain the optimized unmanned boat surrounding object danger value prediction model.

[0197] 4. Model Deployment

[0198] Based on the designed unmanned boat potential danger warning model and the described model training strategy, an optimized unmanned boat surrounding object danger value prediction model is finally obtained. This optimized unmanned boat surrounding object danger value prediction model is deployed in the unmanned boat to help the unmanned boat judge the danger value of surrounding objects to realize the unmanned boat's potential danger warning of the surrounding environment.

[0199] Hardware deployment: In the present invention, a high-definition image acquisition device is deployed on the unmanned boat to acquire video images.

[0200] Software deployment: The optimized unmanned boat potential danger warning model is deployed to the computing terminal of the unmanned boat, where the object trajectory extraction network first processes the collected video images to obtain the position sequence data of the surrounding objects, and then sends this position sequence data to the danger value prediction network to obtain the real-time danger value of each object around the unmanned boat;

[0201] Environmental information perception: Based on the obtained danger values ​​of objects around the unmanned boat, the unmanned boat will have a clearer perception of the degree of influence of surrounding objects on the unmanned boat's own safe driving; based on the predicted danger values ​​of various objects around the unmanned boat, a judgment threshold can be set on the unmanned boat terminal. When there are moving objects around the unmanned boat with a danger value greater than this judgment threshold, an alarm signal will be issued to help the unmanned boat achieve real-time potential danger warning.

[0202] like Figure 5 As shown, the present invention also provides an unmanned boat potential danger warning device, the device includes at least one processor and at least one memory, and also includes a communication interface and an internal bus; the memory stores a computer execution program; the memory stores a computer execution program of the unmanned boat potential danger warning model constructed by the construction method as described above; when the processor executes the computer execution program stored in the memory, the processor can execute a method for warning the potential danger of an unmanned boat. The internal bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, the bus in the drawings of the present application is not limited to only one bus or one type of bus. The memory may include a high-speed RAM memory, and may also include a non-volatile storage NVM, such as at least one disk memory, and may also be a U disk, a mobile hard disk, a read-only memory, a disk or an optical disk, etc.

[0203] The device may be provided as a terminal, a server or other forms of device. In an exemplary embodiment, the electronic device may be implemented by one or more application specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0204] The present invention also provides a computer-readable storage medium, which stores a computer execution program of the unmanned boat potential danger warning model constructed by the construction method as described above. When the computer execution program is executed by a processor, the processor can execute a method for warning of potential danger of an unmanned boat.

[0205] Specifically, a system, device or equipment equipped with a readable storage medium may be provided, on which a software program code for implementing the functions of any of the above-mentioned embodiments is stored, and a computer or processor of the system, device or equipment reads and executes the instructions stored in the readable storage medium. In this case, the program code read from the readable medium itself can implement the functions of any of the above-mentioned embodiments, so the machine-readable code and the readable storage medium storing the machine-readable code constitute a part of the present invention.

[0206] The above description is only the preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0207] Although the above describes the specific implementation methods of the present invention, it is not intended to limit the scope of protection of the present invention. Those skilled in the art should understand that various modifications or variations that can be made by those skilled in the art on the basis of the technical solution of the present invention without creative work are still within the scope of protection of the present invention.

Claims

1. A method for constructing a potential danger warning model for an unmanned boat, characterized in that: The following steps are involved: S1, collect video image data containing the actual movement of objects at sea, obtain the sequence position data of each object in the image video, and build a sequence position data set At the same time, the hazard value of the moving objects at sea included in the video image data is calculated by combining prior knowledge and the object hazard value calculation formula to obtain the object hazard value dataset. ; S2, constructing a potential danger warning model for unmanned boats, including an object trajectory extraction network and a danger value prediction network; The object trajectory extraction network includes a target detection module and a target matching module. The target detection module performs target detection on each video frame of the input image data based on self-attention and a multi-layer convolutional network to obtain the position data of the object in the current video frame. The target matching module matches the same object in different video frames and combines the position data of the same object in different video frames to obtain the sequence position data of the object. The danger value prediction network includes a feature vector encoding module, a multi-head attention mechanism module and a prediction output module, which is used to take the sequence position data of each object as input, and after being processed by the danger value prediction network, predict the danger value of the object to help realize the potential danger warning of the unmanned boat; S3, using the sequence position dataset Train the object trajectory extraction network using the object danger value dataset The hazard value prediction network is trained. Based on the designed training strategy, a larger loss function value is assigned to the training data with poor training results by calculating the adaptive weights to help the model pay more attention to the sample data with poor training results. Finally, a trained unmanned boat potential hazard warning model is obtained.

2. The method for constructing a potential danger warning model for an unmanned boat according to claim 1, characterized in that: The construction of sequence position data set The specific process is: For the initial image data obtained The various objects contained in the video are calibrated frame by frame. In each frame of the video, the labeling software is used to label the corresponding objects. To obtain the image annotation dataset ; Image annotation dataset The objects marked on each frame are defined as , , combine the pixel coordinates of the rightmost, topmost, bottommost, and leftmost pixels of the object to get the object In the Pixel position data on the frame image , : in, The four pixels of the object, namely the leftmost, topmost, bottommost, and rightmost, are represented in sequence. Coordinates and coordinate, Indicates the total number of frames of the image; Image annotation dataset Medium Object The pixel position data on each frame of the image is combined to obtain the object Complete sequence position data : in, Representing objects In the Pixel position data on the frame image; Calculate the image annotation dataset in sequence Sequential position data of each object in , and combine the sequence position data of each object to obtain the object sequence position data set .

3. The method for constructing a potential danger warning model for an unmanned boat according to claim 2, characterized in that: Constructing object danger value dataset The specific process is: using subjective judgment and formula calculation to define the object The danger value of the object in the video The risk value of the subjective judgment is evaluated to obtain the subjective judgment risk value, and this subjective judgment risk value is defined as ; For objects that appear in the field of view of this remote-controlled unmanned boat, record the time when it first appears in its camera , and record the time when the unmanned boat makes the turning operation , and according to Calculate the operation response time : The steering amplitude of the unmanned boat’s steering operation is recorded as , use the formula to calculate the object Operational risk value ; Based on the obtained operational risk value and subjective risk value Perform weighted summation to obtain the object The final corresponding risk value ; Compute image annotation dataset The danger value of all objects in , and combine the calculated danger values ​​to obtain the object danger value data set .

4. The method for constructing a potential danger warning model for an unmanned boat according to claim 1, characterized in that: The target detection module includes a trapezoidal feature extraction unit, a self-attention mechanism unit and a detection unit; The trapezoidal feature extraction unit uses the feature extraction block as the basic module of the feature extraction operation. The input of this feature extraction block first passes through two sequentially connected convolutional layers for feature extraction operation, and then passes through The function is processed nonlinearly, followed by a convolutional layer and a Function, input image data First, the BBS block of a feature extraction block is processed by two convolutional layers. The BBS blocks are connected from top to bottom and repeated three times to form the main path of the trapezoidal feature extraction unit; the feature extraction block at the top layer is connected to the right with a feature extraction block to form a branch path to extract global features. The feature extraction block in the middle is connected to the right with two feature extraction blocks to form branch path 2 to extract local features The feature block at the bottom layer is connected to the right with three feature extraction blocks to form a branch path to extract detail features ; The self-attention mechanism unit extracts the global features , local features and detail features Perform adaptive weight fusion to obtain fusion features ; The detection unit first uses two average pooling layers to fusion features The number of parameters of the entire module is processed, and then the intermediate features are obtained through two BN normalization layers. : in, represents the average pooling layer operation function, Represents the BN normalization layer operation function; The intermediate feature Feaz then passes through two fully connected layers and The final target detection result is obtained after activation function processing : After performing target detection on this video frame, record the leftmost, rightmost, topmost, and bottommost pixels of each object in the detection result. Coordinates, and combine these coordinates to represent the predicted position data of the object ; in, Indicates the predicted The first The position data of an object, Indicates The first The leftmost, rightmost, topmost, and bottommost pixels of an object coordinate.

5. The method for constructing a potential danger warning model for an unmanned boat according to claim 4, characterized in that: The input of the target matching module is the target detection result matrix of the same target in the wth video frame , the output is the detection result matrix of this target in other video frames ; The target matching module consists of two fully connected layers plus a The activation function consists of: in, represents the target matching network operation function, represent Operation function, Represents the fully connected layer operation function; The position data of the same object in different video frames are combined to represent the predicted Sequential position data of objects : in, Indicates that the obtained The first The location data of the object, Represents the predicted sequence position data set of all objects.

6. The method for constructing a potential danger warning model for an unmanned boat according to claim 1, characterized in that: The risk value prediction network includes a feature vector encoding module, a multi-head attention mechanism module and a prediction output module; The feature vector encoding module is based on the obtained sequence position data , for each sequence data Use two interconnected fully connected layers to vector encode this sequence data and obtain the encoded feature data ; The multi-head attention mechanism module performs attention on the feature data Perform multi-level feature extraction operations. The process includes: Feature matrix calculation: The feature Through matrix transformation, they are mapped to On three different feature matrices: in, Respectively represent matrices The matrix mapping function, Indicates Group The mapping parameters used when mapping the feature matrix; Respectively represent characteristics The mapped Group Feature matrix; Attention coefficient calculation: Based on the obtained Two feature matrices, calculate their attention coefficients : in, express Activation function, represents the scaling factor, Indicates The attention coefficient corresponding to the group attention head; Multi-head attention mechanism output feature vector calculation: The attention coefficient corresponding to the group attention head and Group Multiply the feature matrix to get Features output by the group attention head , the output features of the attention head are passed through a fully connected layer to obtain the output features after the multi-head attention mechanism ; The prediction output module is based on the extracted features , this feature is fed into two fully connected layers and After further processing by the activation function, the corresponding danger value prediction result of this object can be obtained. .

7. The method for constructing a potential danger warning model for an unmanned boat according to claim 1, characterized in that: The S3 is specifically: Object trajectory extraction network loss function Designed for: in, represents the mean absolute error calculation function, represents the mean square error calculation function; Indicates The real sequence position data of objects, The predicted Sequential position data of objects; Danger value prediction network loss function Designed for: in, Indicates the actual danger value of the object. Represents the object danger value predicted by the model, Represents the design of normalized operation function; The training strategy is: S31, using object trajectory to extract network loss function Calculate the loss value when training the object sequence position data extraction module , using the risk value to predict the network loss function Calculate the loss value when training the object danger value prediction model ; S32, based on the calculated loss value and , calculate the training value weight of this sample training data : in, Indicates the calculation weight, based on the loss value and , the training value weight of each sample can be calculated ; When this sample is trained, if the calculated loss value is large, the model will have a poor learning effect on this sample; S33, model parameter update: based on the calculated loss value and And the training value weight calculated by S4-3 Update the loss value of this sample after training to get the updated loss value and : Using this updated loss value and use The optimizer extracts the model parameters of the object sequence position module And the model parameters in the object danger value prediction module After updating, samples with poor training effect will occupy a larger loss value in the overall loss using this updated loss value; S34, learning rate update: based on the reinforcement loss value obtained after the first iteration and , using this enhanced loss value and Adjust the learning rate to achieve dynamic learning rate update: in, represents the learning rate before the model is updated, represents the learning rate after the model is updated, express Normalization function; S35, repeat steps S31 to S34, a total of After the number of iterations is completed, the network model parameters of the two models are and The network model parameters finally used by the unmanned boat surrounding object danger value prediction model are saved to obtain the optimized unmanned boat surrounding object danger value prediction model.

8. A method for early warning of potential danger of an unmanned boat, characterized in that: The unmanned boat potential danger warning model constructed by the construction method according to any one of claims 1 to 7 is deployed to the unmanned boat terminal, and includes the following process: Capture video images based on high-definition image acquisition equipment deployed on the unmanned boat; The video image data is processed by frame division and input into the potential danger warning model of the unmanned boat to obtain the real-time danger value of each object around the unmanned boat; The predicted danger values ​​of various objects around the unmanned boat are compared with the judgment threshold. When there are moving objects around the unmanned boat with a danger value greater than the judgment threshold, an alarm signal will be issued to help the unmanned boat achieve real-time potential danger warning.

9. A potential danger warning device for unmanned boats, characterized by: The device includes at least one processor and at least one memory, the processor and the memory are coupled; the memory stores a computer execution program of an unmanned boat potential danger warning model constructed by the construction method as described in any one of claims 1 to 7; when the processor executes the computer execution program stored in the memory, the processor executes a method for warning of potential danger to an unmanned boat.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program or instruction of an unmanned boat potential danger warning model constructed by the construction method as described in any one of claims 1 to 7. When the program or instruction is executed by the processor, the processor executes a method for warning of potential danger of an unmanned boat.

Images