A method for identifying and tracking weak targets on the sea surface using boat-borne infrared vision

Abstract

The present application provides a method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision, which solves the technical problem that weak targets on the sea surface are missing in the existing night vision environment and cannot be accurately identified and tracked. It includes: using a target recognition model to identify the infrared video frame sequence to obtain the coordinate information and confidence of the target, and dividing it into high-confidence targets, low-confidence targets, and invalid targets; making a judgment, if it is empty, generating a new trajectory, otherwise, generating a target trajectory prediction result; matching the high-confidence target with the target trajectory prediction result; matching the low-confidence target with the trajectory that failed to match for the first time; updating the trajectories of successful matching and failed matching, and generating a new trajectory for the high-confidence target that failed to match; writing the updated coordinate information and the new trajectory into the trajectory sequence of the target, iterating until the last frame image, and completing the tracking. The present application can be widely used in the field of target recognition and tracking technology.

Description

Technical Field

[0001] The present application belongs to the technical field of target recognition and tracking, and more specifically, to a method for recognizing and tracking weak targets on the sea surface using boat-borne infrared vision. Background Art

[0002] Finding weak targets on the sea surface under high sea conditions and low visibility has always been a difficult problem for sea surface monitoring and maritime rescue. At present, visible light imaging technology is the most widely used. This technology has the characteristics of high resolution, rich colors, and low cost, but its visual distance will be greatly shortened at night, making it difficult to identify targets at night. Existing radar detection technology can actively detect all-weather and has a long detection distance, but the echo signal is easily disturbed by waves, rain and snow, and the detection accuracy of weak targets on the sea surface is poor. The effect of detecting sea surface buoys, floating balls and people falling into the water is not good, so it cannot meet the needs of sea surface perception scenes. In contrast, the infrared thermal imaging technology carried by unmanned boats can work all-weather, penetrate smoke and has strong anti-interference ability, and is widely used in the field of maritime rescue.

[0003] However, due to the development of infrared sensors, the texture details of sea surface targets collected are scarce. When the unmanned boat is shaking and the contrast between the target and the sea surface is low, it is very difficult to identify and track the target. Therefore, studying a fast and accurate sea surface weak target recognition and tracking method is of great significance to solve the problem of missing weak targets in night sea surface monitoring and rescue scenarios. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide a method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision, so as to solve the technical problem in the prior art that weak targets on the sea surface are missing in a night vision environment and cannot be accurately identified and tracked.

[0005] To achieve the above-mentioned purpose, the embodiment of the present application provides a method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision, comprising the following steps:

[0006] The target recognition model is used to identify the infrared video frame sequence to obtain the target's coordinate information and confidence, and then the targets are divided into high-confidence targets, low-confidence targets and invalid targets. The invalid targets are removed as sea surface background.

[0007] Read the target trajectory sequence and make a judgment. If it is empty, directly generate a new trajectory and proceed to the next round of iteration. Otherwise, generate the target trajectory prediction result.

[0008] Match the high-confidence target with the target trajectory prediction result to obtain the successfully matched trajectory, the trajectory that failed to match for the first time, and the high-confidence target that failed to match;

[0009] Match the low-confidence target with the trajectory that failed to match for the first time, obtain the trajectory that successfully matched, the trajectory that failed to match for the second time, and the low-confidence target that failed to match, and remove the low-confidence target that failed to match as the sea surface background;

[0010] Update the trajectory that was successfully matched, update the trajectory that failed to match for the second time, and generate a new trajectory for the high-confidence target that failed to match;

[0011] Write the updated coordinate information and new trajectory into the target's trajectory sequence, iterate until the last frame image, and complete the tracking;

[0012] The object recognition model includes an edge-assisted enhancement module and a hierarchical weighted constrained attention module;

[0013] The edge-assisted enhancement module includes four-angle edge convolution layers, nonlinear layers, depthwise separable convolutions, and average pooling layers to extract edge contour features of the image;

[0014] The hierarchical weighted constrained attention module consists of depthwise separable convolutions, ordinary convolution layers with a kernel size of 3×3, and normalization layers to balance the impact of the attention mask on the input.

[0015] Preferably, the nonlinear layer includes a ReLU activation function layer and a batch normalization layer.

[0016] Preferably, the common convolution layer with a kernel size of 3×3 includes a 3×3 convolution with a stride of 1, a batch normalization layer, and a SiLU activation function layer.

[0017] Preferably, the partitioning formula is:

[0018] ;

[0019] In the formula, For the The confidence level of a target is 0, 1, and 2, which respectively indicate that the target is an invalid target, a low-confidence target, and a high-confidence target. The target recognition model is given by The confidence level of a target, is the high score threshold, is the low score threshold.

[0020] Preferably, the process of matching the low confidence target with the trajectory that failed to match for the first time includes: calculating the intersection-over-union ratio, filtering out the intersection-over-union ratios that are less than the intersection-over-union ratio threshold, and performing Hungarian matching on the remaining intersection-over-union ratios.

[0021] Preferably, the formula for the intersection-over-union ratio is obtained:

[0022] ;

[0023] In the formula, is the intersection-over-union ratio of the low confidence target and the “trajectory that failed to match for the first time”, Low confidence target area for identification Target area with trajectory prediction The area of ​​intersection, Low confidence target area for identification Target area with trajectory prediction The combined area, Low confidence target area for identification And the target area of ​​trajectory prediction The square of the Euclidean distance between the farthest points of two regions, for and The square of the Euclidean distance between the center points of two regions, for and The square of the Euclidean distance between the horizontal coordinates of the farthest points in the two regions, for and The square of the Euclidean distance between the horizontal coordinates of the two region center points, for and The square of the Euclidean distance between the vertical coordinates of the farthest points in the two regions, for and The square of the Euclidean distance between the vertical coordinates of the two region center points, , , is the weight factor of the last three items.

[0024] Preferably, the formula for generating the target trajectory prediction result is as follows:

[0025] ;

[0026] ;

[0027] In the formula, is the observable target state vector, is the predicted state vector of the trajectory in the current frame, is the observation matrix, is the central abscissa of the trajectory, is the center ordinate, For width, For high.

[0028] Preferably, the process of matching the high confidence target with the target trajectory prediction result includes: comparing the Mahalanobis distance with the Mahalanobis distance threshold, discarding the distance greater than the Mahalanobis distance threshold, and performing Hungarian matching after calculating the cost of the remaining distances.

[0029] Preferably, the formula for the Mahalanobis distance is obtained:

[0030] ;

[0031] ;

[0032] In the formula, is the covariance matrix between variables, is the observation matrix, is the state prediction error covariance matrix of the trajectory in the current frame, is the transpose of the observation matrix, Indicates high confidence target and The Mahalanobis distance of the Kalman filter prediction results, is the observable target state vector, To identify the state vector of the target, including the The center horizontal coordinate, center vertical coordinate, width and height information of each target.

[0033] Preferably, the cost is calculated as follows:

[0034] ;

[0035] In the formula, For the A high confidence target and The cost of a trajectory, To tune the hyperparameters, For the A high confidence target and The Mahalanobis distance of the trajectories is For the The confidence of a high confidence target, and are all conditional functions, is the trajectory state, For the The categories of high-confidence objects, For the The category of the trajectory.

[0036] The beneficial effect of the present application is that the present application provides a method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision, which utilizes an improved target recognition model for identification, and improves the model's ability to identify weak targets on the sea surface by enhancing the model's attention to the target itself and the target's edge contour features.

[0037] The target recognition model has been improved on the basis of the traditional YOLOv8, including the following three points:

[0038] First, the receptive field of the traditional YOLOv8 backbone network is adjusted to make the backbone network focus on extracting local detail features of potential targets;

[0039] Second, an edge-assisted enhancement module is introduced into the backbone network to enhance the edge contour features of potential weak targets on the sea surface and enhance the model's ability to extract features of weak targets.

[0040] Third, a hierarchical weighted constrained attention module is introduced into the neck network of the traditional YOLOv8 to add feature masks to the multi-level features output by the backbone network, which amplifies the influence of the target’s own features and the surrounding environment features on the recognition results, reduces the improved target recognition model’s attention to distant background features, and balances the impact of the attention mask on the input.

[0041] In addition, the application adjusts the matching process and the constraints on the target coordinate information. Such improvements can alleviate the problem of the position change of the target image caused by the oscillation of the infrared sensor on the unmanned boat to the greatest extent, improve the recognition and tracking performance of weak targets on the sea surface, and can more accurately identify and track weak targets on the sea surface. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0043] Figure 1 A flow chart of a method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision system according to an embodiment of the present application;

[0044] Figure 2 A target recognition model training flow chart provided for an embodiment of the present application;

[0045] Figure 3 A schematic diagram of the structure of a target recognition model provided in one embodiment of the present application;

[0046] Figure 4A flowchart of high confidence target matching provided in one embodiment of the present application;

[0047] Figure 5 A flowchart of low-confidence target matching provided in one embodiment of the present application. DETAILED DESCRIPTION

[0048] In order to make the technical problems, technical solutions and beneficial effects to be solved by the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0049] See also Figure 1 , a method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision is provided in an embodiment of the present application, comprising:

[0050] S1: Use the improved target recognition model to identify and obtain the target's coordinate information and confidence.

[0051] The infrared thermal imaging camera carried by the unmanned boat is used to shoot the sea scene to obtain the infrared video frame sequence in the sea scene. Then, the infrared video frame sequence is recognized using the improved target recognition model to obtain the central horizontal coordinate of the target. , center vertical coordinate ,Width ,high and confidence ,in, Indicates Identify the target.

[0052] See also Figure 2 , is an improved target recognition model, which is improved based on the traditional YOLOv8 algorithm.

[0053] Specifically, the improved target recognition model introduces an edge-assisted enhancement module and a hierarchical weighted constrained attention module based on the traditional YOLOv8 algorithm, so that it can better enhance the target recognition model's attention to important features such as target edge contour features.

[0054] The backbone network of the improved target recognition model contains the main branch, the edge auxiliary enhancement module and the hierarchical weighted constrained attention module connected to the neck network. The main branch uses 5 convolutional layers (CBS3×3, ch) with a kernel size of 3 and a stride of 2 to downsample the infrared video frame sequence and extract multi-level image features. The C2f feature enhancement module is inserted in the 2nd to 5th convolutional layers to enhance the image features. The number of Bottleneck blocks stacked in the C2f module is {1, 2, 2, 1} respectively.

[0055] Figure 2 Where ch is the number of channels, which is 32 in this embodiment; CBS is a module composed of a 3×3 convolution with a stride of 2, a batch normalization layer, and a SiLU activation function layer in series; C2f is the feature enhancement module in YOLOv8; DSBlock is a downsampling module, which is a 3×3 maximum pooling layer with a stride of 2 in this embodiment; YOLOv8 PAN and YOLOv8 head are the corresponding parts in YOLOv8.

[0056] The input of the edge-assisted enhancement module is the same as the input of the main branch. The output of the edge-assisted enhancement module is the target edge contour feature output connected to the output feature map of the first C2f module of the main branch by pixel-by-pixel summation. The edge-assisted enhancement module is composed of several edge calculation convolution stacks of different modes, among which EdgeConv0, EdgeConv45, EdgeConv90, and EdgeConv135 are edge convolution layers of four angles; NL is a nonlinear layer, including a ReLU activation function layer and a batch normalization layer; DWConv is a depth-separable convolution; AvgPool is an average pooling layer in the channel direction.

[0057] Among them, the kernel parameters of edge computing convolution As shown below:

[0058] ;

[0059] In the formula, is the kernel parameter of edge computing convolution, - Compute the learnable parameters of convolution kernels at the edge.

[0060] The difference between the edge computing convolution of the present application and the ordinary convolution is that after the parameters of the edge computing convolution are reversely updated, some of the parameters are artificially set to 0 to obtain a convolution mode that can extract the edge contour features of the image. The present invention uses edge computing convolutions at four angles of 0°, 45°, 90°, and 135°, which respectively , , , The parameters are set to 0 to obtain a parameter mode similar to the Sobel operator (a discrete differential operator used for edge detection).

[0061] Among them, the expression of the edge auxiliary enhancement module is:

[0062] ;

[0063] In the formula, is the input of the edge-assisted enhancement module, is the output of the edge-assisted enhancement module, , , , Calculate the convolution kernel parameters for the edges in four directions respectively. represents the nonlinear function composed of nonlinear layers in the edge-assisted enhancement module, Represents a convolution operation.

[0064] In the hierarchical weighted constrained attention module (HWCA), DWConv is a depthwise separable convolution, and the ordinary convolution layer with a kernel size of 3×3 is a module obtained by connecting a 3×3 convolution with a stride of 1, a batch normalization layer, and a SiLU activation function layer in series; Sigmoid is a normalization layer, which scales the value to (0,1).

[0065] Among them, the expression of the hierarchical weighted constrained attention module is:

[0066] ;

[0067] ;

[0068] ;

[0069] In the formula, For the The input of the hierarchical weighted constrained attention module, represents a grouped convolution where the number of channels is equal to the number of groups, represents point-wise convolution, represents the nonlinear function of the nonlinear layer in the hierarchical weighted constrained attention module, represents a normal convolutional layer with a kernel size of 3×3 and an output channel number of 1 for generating feature masks. represents the normalization layer, and is the intermediate feature map, Output feature map for the hierarchical weighted constrained attention module, It is an incentive balance factor, which is used to balance the impact of the attention mask generated by the module on the input. It is usually between 0.4 and 0.9. In this embodiment, it is 0.6.

[0070] Specifically, the training process of the improved target recognition model is as follows: Figure 3 As shown, the following steps are included:

[0071] First, construct a training data set and a test data set. Specifically, it includes collecting measured infrared sea surface data, selecting and cropping a number of infrared images containing weak sea surface targets as training data, and marking the weak sea surface targets therein to obtain the real frame of the target, and dividing the marked weak sea surface targets into training data sets and test data sets in a ratio of 8:2. In this embodiment, a three-spectrum spherical infrared turntable is used to collect measured infrared sea surface data. The frame rate of the three-spectrum spherical infrared turntable is 50FPS, wherein the frame rate will affect the values ​​of some subsequent threshold parameters. There is no requirement for setting the frame rate of the three-spectrum spherical infrared turntable here, and it can be set according to actual conditions.

[0072] Secondly, the edge-assisted enhancement module in the improved target recognition model is trained separately. The training data set is the divided training data set, and the label is the edge contour feature map detected by the Canny operator (a multi-level edge detection algorithm). The cross entropy loss function is used to guide the edge-assisted enhancement module training during the training process. The training rounds are set to 50, the batches are set to 8, the initial learning rate is set to 0.002, and the SGD optimizer (stochastic gradient descent, a commonly used optimization algorithm) and the cosine roll-off learning rate descent strategy are used. It is worth noting that there is no requirement for the setting of the above numbers here, and they can be set according to the actual situation.

[0073] Assume that the edge contour feature map output by the edge auxiliary enhancement module is , tagged as , and The width and high The same, then the cross entropy loss function As shown below:

[0074] ;

[0075] In the formula, is the edge contour feature map The cross entropy loss function is: and They are and The corresponding pixel point in , are the horizontal and vertical coordinates of the corresponding pixel points, The value of is 0 or 1. When the pixel belongs to the edge contour of an object, the value is 1, otherwise it is 0. The value of is (0,1), and the specific value is determined by the Sigmoid function on the feature map The pixel value is normalized to obtain; Is a position parameter with a value of 0 or 1. If the pixel is located in the real frame of the target If it is internal, the value is 1, otherwise the value is 0.

[0076] Again, the annotated weak sea targets are used to generate labels for supervising the training of the hierarchical weighted constrained attention module. Assume that an infrared image contains two targets, whose center horizontal and vertical coordinates and width and height are and , then generate an image with the same width and height as the original infrared image and all pixel values ​​are 0, and then and The square area formed by the two points and and The pixel values ​​within the square area formed by the two points are all set to 1.

[0077] Finally, the parameters in the edge-assisted enhancement module are frozen, and the other modules of the improved target recognition model are trained. The training dataset is the divided training dataset, and the labels include annotations and labels. The training rounds are set to 200, the batches are set to 32, the initial learning rate is set to 0.0025, the SGD optimizer and the cosine roll-off learning rate descent strategy are used, the cross entropy loss function is used to guide the hierarchical weighted constrained attention module training, and the YOLOv8 loss function is used to guide the improved target recognition model training.

[0078] S2: Divide into high-confidence targets, low-confidence targets and invalid targets, and remove the invalid targets as sea surface background.

[0079] The identified targets are divided into high-confidence targets, low-confidence targets and invalid targets according to their confidence levels, and the invalid targets are removed as sea surface background.

[0080] Specifically, the formula for dividing the identified targets according to the confidence level in this embodiment is as follows:

[0081] ;

[0082] In the formula, For the The confidence level of a target includes three levels: 0, 1, and 2, which respectively indicate that the target is an invalid target, a low-confidence target, and a high-confidence target. The improved target recognition model is given in The confidence level of a target, is the high score threshold, which is 0.6 in this implementation. is the low score threshold, which is 0.25 in this implementation. There is no limitation on the setting of the threshold here, and it can be set according to the actual situation.

[0083] The specific operation is as follows: First, the confidence score of the identified target is compared with the low score threshold Compare and compare the confidence level less than the low score threshold The target is set as an invalid target, which refers to the sea surface background, noise or irrelevant objects, etc., which do not participate in the subsequent matching process; the confidence is greater than or equal to the low score threshold The target is set as a low confidence target.

[0084] Then, compare the low confidence target with the high score threshold Compare and set the confidence level to be greater than or equal to the high score threshold The target is set as a high-confidence target, thereby obtaining a high-confidence target and a low-confidence target as the most reliable targets for subsequent target matching.

[0085] S3: Read the target trajectory sequence and make a judgment. If it is empty, directly generate a new trajectory and proceed to the next round of iteration. Otherwise, generate the target trajectory prediction result.

[0086] The target trajectory sequence is read from the identified target and judged. If the trajectory sequence is empty, a new trajectory is directly generated and written into the target trajectory sequence, and then the next round of iteration is continued. Otherwise, the Kalman filter is used to predict the trajectory sequences of all targets to generate the predicted position and predicted size of the target, that is, the target trajectory prediction result. Among them, the target trajectory sequence is a set of parameters including the target's central horizontal and vertical coordinates, width and height, trajectory mismatch time, trajectory status, etc.

[0087] In an optional embodiment, the Kalman filter is a linear uniform speed Kalman filter, which is used to predict the central horizontal and vertical coordinates, width and height of the trajectory sequence.

[0088] In an optional embodiment, the Kalman filter uses the CV model as the motion model. The state vector of the target in the CV model is , the state transfer matrix , observation matrix , initialize the optimal estimation error covariance matrix , the initial noise covariance matrix of the state transfer process , the noise covariance matrix of the observation process As shown below:

[0089] ;

[0090] In the formula, is the time interval between each prediction, which is 1 in this embodiment.

[0091] ; ;

[0092] ;

[0093] ;

[0094] ;

[0095] In the formula, and are the covariance of the noise, taking 0.005 and 0.000001 respectively. is the target scale impact factor, calculated according to the above formula, Indicates A Kalman filter, Indicates The width of the target that the Kalman filter matches when it is created, Indicates The height of the target that a Kalman filter matches when it is created, and Assigned when generating the Kalman filter.

[0096] Specifically, the process of generating a new trajectory in this embodiment is as follows: first, the trajectory life cycle is set to 1, the trajectory mismatch time is set to 0, and the category information of the target is recorded. Then the corresponding Kalman filter is generated, including initializing the target's state vector using the target's central horizontal and vertical coordinates and width and height information ,calculate And initialize the optimal estimation error covariance matrix And the noise covariance matrix and .

[0097] Specifically, the state prediction stage of the Kalman filter is as follows:

[0098] Assume that the current frame is the k+1th frame and the previous frame is the kth frame. The optimal estimated state vector of the known trajectory in the previous frame is ,according to Get the predicted state vector of the trajectory in the current frame ,in, is the state transfer matrix of the trajectory from the previous frame to the current frame.

[0099] The optimal estimation error covariance matrix of the known trajectory in the previous frame ,according to Get the state prediction error covariance matrix of the trajectory in the current frame ,in, is the noise covariance matrix of the state transfer process.

[0100] Among them, the Kalman filter is used to calculate the central horizontal coordinate of the trajectory sequence , center vertical coordinate ,Width and high The prediction process is as follows:

[0101] ;

[0102] ;

[0103] In the formula, is the observable target state vector, is the predicted state vector of the trajectory in the current frame, is the observation matrix.

[0104] S4: Match the high-confidence target with the target trajectory prediction result to obtain the successfully matched trajectory, the trajectory of the first failed matching, and the high-confidence target that failed to match; match the low-confidence target with the trajectory of the first failed matching to obtain the successfully matched trajectory, the trajectory of the second failed matching, and the low-confidence target that failed to match, and remove the low-confidence target that failed to match as the sea surface background.

[0105] For details, please refer to Figure 4 , the process of matching high confidence targets with target trajectory prediction results is as follows:

[0106] First, calculate the covariance matrix between the variables , as shown below:

[0107] ;

[0108] In the formula, is the covariance matrix between variables, is the observation matrix, is the state prediction error covariance matrix of the trajectory in the current frame, is the transpose of the observation matrix.

[0109] Calculate the Mahalanobis distance between the high confidence target and all target trajectory prediction results , the formula is as follows:

[0110] ;

[0111] In the formula, Indicates high confidence target and The Mahalanobis distance of the Kalman filter prediction results, is the observable target state vector, is the covariance matrix between variables, To identify the state vector of the target, including the The center horizontal coordinate, center vertical coordinate, width and height information of each target.

[0112] The Mahalanobis distance and the Mahalanobis distance threshold Make a comparison and filter out the distances greater than the Mahalanobis distance threshold. The Mahalanobis distance is calculated using the center horizontal coordinate, center vertical coordinate, width, and height of the target trajectory prediction result. , assuming that the four parameters are independent of each other, the value of 9.488 at 95% confidence level with a chi-square distribution of 4 degrees of freedom is selected as the Mahalanobis distance threshold It is worth noting that the Mahalanobis distance threshold There is no limitation on the settings, and you can set them according to the situation.

[0113] Then, the remaining Mahalanobis distance is concatenated with the target category information as the cost of matching the target with the trajectory. The cost calculation formula is as follows:

[0114] ;

[0115] In the formula, For the A high confidence target and The cost of a trajectory, For the A high confidence target and The Mahalanobis distance of the trajectories is For the The confidence of a high confidence target, and are all conditional functions, is the trajectory state, For the The categories of high-confidence objects, For the There are 3 types of trajectories. is the trajectory state judgment function. When the trajectory is a stable trajectory, it takes 1, and otherwise it takes 0. is the category judgment function, only in the A high confidence target and When the categories of the trajectories are consistent, it takes 1, and otherwise it takes 0. To reconcile the hyperparameters, in this embodiment, the value is taken as 0.4 to adjust the ratio of the two costs.

[0116] Finally, Hungarian matching is performed based on the above cost calculation formula, and three types of results are obtained: trajectories that have been successfully matched, trajectories that have failed to match for the first time, and high-confidence targets that have failed to match.

[0117] See also Figure 5 , the process of matching low confidence targets with trajectories that failed to match for the first time is as follows:

[0118] First, we calculate the intersection of low confidence targets and the trajectory that failed to match for the first time. , the calculation formula is as follows:

[0119] ;

[0120] In the formula, is the intersection-over-union ratio of the low confidence target and the “trajectory that failed to match for the first time”, Low confidence target area for identification Target area with trajectory prediction The area of ​​intersection, Low confidence target area for identification Target area with trajectory prediction The combined area, Low confidence target area for identification And the target area of ​​trajectory prediction The square of the Euclidean distance between the farthest points of two regions, for and The square of the Euclidean distance between the center points of two regions, for and The square of the Euclidean distance between the horizontal coordinates of the farthest points in the two regions, for and The square of the Euclidean distance between the horizontal coordinates of the two region center points, for and The square of the Euclidean distance between the vertical coordinates of the farthest points in the two regions, for and The square of the Euclidean distance between the vertical coordinates of the two region center points, , , is the weight factor of the last three items, usually 0.2, 0.4, and 0.4.

[0121] Then, the intersection ratio threshold The intersection-and-union ratio is filtered out, where the intersection-and-union ratio threshold Take -0.468.

[0122] Finally, the remaining intersection-union ratios are subjected to Hungarian matching to obtain three types of results: successfully matched trajectories, trajectories that failed the second matching, and low-confidence targets that failed to match. The low-confidence targets that failed to match are discarded as sea surface background.

[0123] S5: Update the successfully matched trajectory and write the updated coordinate information into the trajectory sequence of the target, update the trajectory that failed to match for the second time and write the updated coordinate information into the trajectory sequence of the target, generate a new trajectory for the high-confidence target that failed to match, and write the new trajectory into the trajectory sequence of the target.

[0124] The process of updating the strategy for matching successful trajectories is as follows:

[0125] First, the high-confidence target and the low-confidence target corresponding to the successfully matched trajectory are taken as the measured state vector of the trajectory in the current frame. , the Kalman filter corresponding to the trajectory is updated, and the trajectory mismatch time of the stable trajectory is set to 0.

[0126] Specifically, the process of updating the Kalman filter is as follows:

[0127] First, calculate the Kalman gain of the trajectory in the current frame :

[0128] ;

[0129] In the formula, is the Kalman gain of the trajectory in the current frame, is the observation matrix, is the transpose of the observation matrix, is the state prediction error covariance matrix of the trajectory in the current frame, is the noise covariance matrix of the observation process.

[0130] Then calculate the optimal estimated state vector of the trajectory in the current frame :

[0131] ;

[0132] In the formula, is the optimal estimated state vector of the trajectory in the current frame, is the Kalman gain of the trajectory in the current frame, is the measured state vector of the trajectory in the current frame, is the predicted state vector of the trajectory in the current frame.

[0133] Finally, the optimal estimation error covariance matrix of the trajectory in the current frame is calculated :

[0134] ;

[0135] In the formula, is the optimal estimation error covariance matrix of the trajectory in the current frame, is the state prediction error covariance matrix of the trajectory in the current frame, is the Kalman gain of the trajectory in the current frame, is the observation matrix, is the identity matrix.

[0136] Then, the state of the unstable trajectory is judged. If the lifetime of the unstable trajectory reaches the stability threshold and the trajectory is recently The first-ranked category in the category information of the frame accounts for more than frame, the trajectory is set to a stable state. This application calls the trajectory in this state a stable trajectory, which is updated in the Kalman filter, and it is determined that the identified target has appeared, and is used to compare with the trajectory that successfully matches the next cycle, so as to achieve continuous tracking; if the unstable trajectory life cycle and historical category information do not meet the requirements, the trajectory life cycle is increased by 1, and the category information of the target is recorded, which is updated in the Kalman filter for the next cycle to continuously track the target, wherein, The value of is related to the frame rate of the infrared video, usually ranging from 25 to 50. The value of is 50. The value is 25. There is no limit on the value and it can be set according to the actual situation.

[0137] The process of generating a new trajectory is the same as that in S3, including setting the trajectory lifecycle to 1, setting the trajectory mismatch time to 0, and recording the target category information. , generate corresponding Kalman filter and other operations.

[0138] The process of updating the trajectory that failed to match is:

[0139] Determine whether the trajectory mismatch time of the "second failed matching trajectory" is greater than the maximum allowable mismatch time If it is greater than the maximum allowable mismatch time , it is deleted as the ocean background; otherwise, the trajectory mismatch time is increased by 1, and the central horizontal and vertical coordinates of the Kalman filter prediction value and the width and height of the trajectory at the last successful match are used as observation values ​​to update the Kalman filter. The updating steps are consistent with the above-mentioned Kalman filter updating steps. The maximum allowable mismatch time in this embodiment The value is 200. Here is the maximum allowable mismatch time There is no limit on the value and it can be set according to the actual situation.

[0140] The present application provides a method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision, which utilizes an improved target recognition model for identification, and improves the model's ability to identify weak targets on the sea surface by enhancing the model's attention to the target itself and the target's edge contour features.

[0141] The target recognition model has been improved on the basis of the traditional YOLOv8, including the following three points:

[0142] First, the receptive field of the traditional YOLOv8 backbone network is adjusted to make the backbone network focus on extracting local detail features of potential targets;

[0143] Second, an edge-assisted enhancement module is introduced into the backbone network to enhance the edge contour features of potential weak targets on the sea surface and enhance the model's ability to extract features of weak targets.

[0144] Third, a hierarchical weighted constrained attention module is introduced into the neck network of the traditional YOLOv8 to add feature masks to the multi-level features output by the backbone network, which amplifies the influence of the target’s own features and the surrounding environment features on the recognition results, reduces the improved target recognition model’s attention to distant background features, and balances the impact of the attention mask on the input.

[0145] In addition, the application adjusts the matching process and the constraints on the target coordinate information. Such improvements can alleviate the problem of the position change of the target image caused by the oscillation of the infrared sensor on the unmanned boat to the greatest extent, improve the recognition and tracking performance of weak targets on the sea surface, and can more accurately identify and track weak targets on the sea surface.

[0146] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0147] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

Claims

1. A method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision, characterized in that: The following steps are involved: The target recognition model is used to identify the infrared video frame sequence to obtain the target's coordinate information and confidence, and then the targets are divided into high-confidence targets, low-confidence targets and invalid targets. The invalid targets are removed as sea surface background. Read the target trajectory sequence and make a judgment. If it is empty, directly generate a new trajectory and proceed to the next round of iteration. Otherwise, generate the target trajectory prediction result. Matching the high-confidence target with the target trajectory prediction result to obtain a successfully matched trajectory, a trajectory that failed to match for the first time, and a high-confidence target that failed to match; Matching the low-confidence target with the trajectory that failed to match for the first time, obtaining a successfully matched trajectory, a trajectory that failed to match for the second time, and a low-confidence target that failed to match, and removing the low-confidence target that failed to match as a sea surface background; Update the trajectory that was successfully matched, update the trajectory that failed to match for the second time, and generate a new trajectory for the high-confidence target that failed to match; Writing the updated coordinate information and the new trajectory into the trajectory sequence of the target, iterating until the last frame image, and completing the tracking; The target recognition model includes an edge-assisted enhancement module and a hierarchical weighted constraint attention module; The edge auxiliary enhancement module includes edge convolution layers at four angles, a nonlinear layer, a depthwise separable convolution and an average pooling layer to extract edge contour features of the image; The hierarchical weighted constrained attention module includes depthwise separable convolution, ordinary convolution layers with a kernel size of 3×3, and normalization layers to balance the impact of the attention mask on the input.

2. The method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision according to claim 1 is characterized in that: The nonlinear layer includes a ReLU activation function layer and a batch normalization layer.

3. The method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision according to claim 1 is characterized in that: The common convolution layer with a kernel size of 3×3 includes a 3×3 convolution with a stride of 1, a batch normalization layer, and a SiLU activation function layer.

4. The method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision according to claim 1 is characterized in that: The formula for the division is: ； In the formula, For the The confidence level of a target is 0, 1, and 2, which respectively indicate that the target is an invalid target, a low-confidence target, and a high-confidence target. The target recognition model is given by The confidence level of a target, is the high score threshold, is the low score threshold.

5. The method for identifying and tracking weak targets on the sea surface using infrared vision on a boat according to claim 1 is characterized by: The process of matching the low confidence target with the trajectory that failed to match for the first time includes: calculating the intersection-over-union ratio, filtering out the intersection-over-union ratios that are less than the intersection-over-union ratio threshold, and performing Hungarian matching on the remaining intersection-over-union ratios.

6. The method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision according to claim 5 is characterized by: The formula for the intersection-over-union ratio is obtained: ； In the formula, is the intersection-over-union ratio of the low confidence target and the "trajectory that failed to match for the first time", Low confidence target area for identification Target area with trajectory prediction The area of ​​intersection, Low confidence target area for identification Target area with trajectory prediction The combined area, Low confidence target area for identification and the target area for trajectory prediction The square of the Euclidean distance between the farthest points of two regions, for and The square of the Euclidean distance between the center points of two regions, for and The square of the Euclidean distance between the horizontal coordinates of the farthest points in the two regions, for and The square of the Euclidean distance between the horizontal coordinates of the two region center points, for and The square of the Euclidean distance between the vertical coordinates of the farthest points in the two regions, for and The square of the Euclidean distance between the vertical coordinates of the two region center points, , , is the weight factor of the last three items.

7. The method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision according to claim 1 is characterized in that: The formula for generating the target trajectory prediction result is as follows: ； ； In the formula, is the observable target state vector, is the predicted state vector of the trajectory in the current frame, is the observation matrix, is the central abscissa of the trajectory, is the center ordinate, For width, For high.

8. The method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision according to claim 1 is characterized by: The process of matching the high confidence target with the target trajectory prediction result includes: comparing the Mahalanobis distance with the Mahalanobis distance threshold, discarding the distance greater than the Mahalanobis distance threshold, and performing Hungarian matching after calculating the cost of the remaining distances.

9. The method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision according to claim 8 is characterized in that: The formula for the Mahalanobis distance is obtained: ； ； In the formula, is the covariance matrix between variables, is the observation matrix, is the state prediction error covariance matrix of the trajectory in the current frame, is the transpose of the observation matrix, Indicates high confidence target and The Mahalanobis distance of the Kalman filter prediction results, is the observable target state vector, To identify the state vector of the target, including the The center horizontal coordinate, center vertical coordinate, width and height information of each target.

10. The method for identifying and tracking weak targets on the sea surface using a boat-borne infrared vision according to claim 8, characterized in that: The calculation formula of the cost is as follows: ； In the formula, For the A high confidence target and The cost of a trajectory, To tune the hyperparameters, For the A high confidence target and The Mahalanobis distance of the trajectories is For the The confidence of a high confidence target, and are all conditional functions, is the trajectory state, For the The categories of high-confidence objects, For the The category of the trajectory.

Images