A sea surface multi-target tracking method based on twin network target detection, and a device and a readable storage medium thereof

Abstract

This invention belongs to the field of multi-target tracking technology, specifically disclosing a method, device, and readable storage medium for multi-target tracking on the sea surface based on Siamese network target detection. The method includes the following steps: S1, dividing the sea surface scene dataset into regular scene data and bright scene data, using regular scene data to train the N-YOLO detection sub-network and bright scene data to train the S-YOLO detection sub-network; constructing a Siamese network based on the N-YOLO and S-YOLO detection sub-networks; S2, inputting video data acquired by a camera into the detection module, building a Siamese network discrimination system based on an evaluation function, selectively outputting the detection results of the N-YOLO and S-YOLO detection sub-networks to obtain robust target detection information; S3, performing Kalman filtering prediction on the current position of the tracked trajectory object, and performing additional prediction on the next two frames of images, calculating the IOU value between the predicted boxes, and marking unoccluded or potentially occluded image frames according to a threshold.

Description

Technical Field

[0001] This invention belongs to the field of multi-target tracking technology, specifically relating to a method, device, and readable storage medium for multi-target tracking on the sea surface based on twin network target detection. Background Technology

[0002] With the increasing marine activities and extensive development of marine resources, multi-target tracking technology for sea surface scenes has become an important requirement in the fields of marine monitoring and resource management. Existing multi-target tracking methods for sea surfaces are mostly based on multimodal fusion using equipment such as cameras and radar. This type of method suffers from high equipment costs and maintenance costs, as well as complex procedures such as data synchronization. In contrast, multi-target tracking methods for sea surfaces using pure vision algorithms can obtain considerable tracking information for sea surface scenes at extremely low cost. However, existing methods are limited by physical factors such as weather, location, and noise, requiring manual adjustment of some parameters to adapt to the current sea surface environment. Furthermore, they struggle to address the problem of objects occluding each other, resulting in significant tracking errors and poor flexibility when facing complex environments and occlusion phenomena, hindering large-scale deployment.

[0003] A patent application with publication number CN117079097A discloses a method for recognizing sea surface targets based on visual saliency. This method addresses the complex characteristics of sea surface lighting, noise, and background by expanding the dataset using two data augmentation techniques: photometric distortion and geometric distortion. For photometric distortion, the brightness, noise, contrast, and saturation of the image are enhanced to adapt to the influence of sea surface lighting and noise. When processing geometric distortion, random scaling, cropping, translation, shearing, rotation, and Mosaic data augmentation are added to improve the detection capability for differences in target size. This sea surface target recognition method improves the accuracy of sea surface scene detection by enhancing the image, which significantly increases the computational cost of sea surface target recognition, hindering its large-scale deployment and application.

[0004] A patent application with publication number CN117152530A discloses a sea surface target tracking system based on multimodal information fusion. This system can fuse various types of information to determine the type of a sea target and then decide whether to track it based on that type. The method collects a frontal view image of the sea surface captured by a visual camera, reflection point data detected by radar, a top-view image of the sea surface captured by a satellite, and target signals obtained from a shipborne AIS. A decision processing module then fuses the target type with the reflection point data, the top-view image, and the target signal to obtain the target's tracking trajectory. While this method improves the accuracy of target identification in sea surface scenes and provides real-time corrections such as trajectory prediction, the overall hardware requirements for this sea surface target tracking system are high, resulting in a high cost and hindering large-scale deployment.

[0005] It is evident that existing multi-target tracking methods for the sea surface suffer from problems such as low versatility and high resource consumption. Furthermore, in actual sea surface target tracking processes, phenomena such as occlusion between objects in the sea scene can reduce the computational accuracy of existing multi-target tracking methods, thus affecting their tracking performance. Summary of the Invention

[0006] To address the problem that existing multi-target tracking methods for sea surface scenes struggle to accurately detect, identify, and robustly track target objects under varying weather conditions and complex occlusion environments, this invention provides a multi-target tracking method for sea surface scenes based on twin network target detection, along with its associated device and readable storage medium, applicable to both weather changes and occlusion conditions.

[0007] To achieve the above objectives, the present invention is implemented through the following technical solution:

[0008] The first aspect of this invention provides a method for multi-target tracking on the sea surface based on twin network target detection, comprising the following steps:

[0009] S1. Based on the characteristic that weather changes can easily cause water surface reflection, the sea surface scene dataset is divided into normal scene data and bright scene data. The normal scene data is used to train the N-YOLO detection sub-network, and the bright scene data is used to train the S-YOLO detection sub-network. A twin network based on the N-YOLO detection sub-network and the S-YOLO detection sub-network is constructed.

[0010] S2. Input the video data collected by the camera into the detection module based on the Siamese network. Build a Siamese network discrimination system based on the evaluation function. Selectively output the detection results of the N-YOLO detection subnetwork and the S-YOLO detection subnetwork to obtain robust target detection information.

[0011] S3. Perform Kalman filtering prediction on the current position of the tracked object, and make additional predictions for the next two frames of images. Calculate the IOU value between the predicted boxes, and mark the unoccluded or potentially occluded image frames according to the threshold.

[0012] S4. Model motion features for unoccluded image frames and appearance features for potentially occluded image frames to construct a loss function matrix, and perform correlation matching between trajectories and detected objects based on Hungarian matching.

[0013] S5. Update the Kalman filter parameters of the trajectory object based on the associated results, and summarize the target ID, category, and trajectory information in the sea surface scene.

[0014] In step S1, the N-YOLO detection subnetwork and the S-YOLO detection subnetwork are completely consistent with the network framework of YOLOv5. The weights of the N-YOLO detection subnetwork are trained using a regular scene image dataset, and the weights of the S-YOLO detection subnetwork are trained using a highlight scene image dataset.

[0015] According to the above-described multi-target tracking method for sea surface based on twin network target detection, preferably, step S2 consists of the following steps:

[0016] S21. Use a twin network to perform target detection on each frame of video data, and obtain the output results of the N-YOLO detection subnetwork and the S-YOLO detection subnetwork, namely the target detection box coordinates and confidence scores.

[0017] S22. Calculate the optical channel value pixel by pixel for each frame of image data to obtain the optical channel mean (AVLC) and obtain the brightness information representing the image.

[0018] S23. Based on the confidence level and the mean of the optical channel, establish an evaluation function, build a twin network discrimination system, and select the target detection box of the corresponding detection sub-network as the output according to the evaluation score.

[0019] According to the above-mentioned multi-target tracking method for sea surface based on twin network target detection, preferably, in step S21, the N-YOLO detection sub-network and the S-YOLO detection sub-network use non-shared network weights. After inputting the image data of the same sea surface scene into the N-YOLO detection sub-network and the S-YOLO detection sub-network respectively, different image detection results are output, and corresponding object detection boxes and confidence scores are obtained.

[0020] In step S22, to evaluate the brightness of the input image, optical channel characteristics are introduced as a scoring criterion; the optical channel values ​​of the input image data are expressed as:

[0021]

[0022] Where D represents the optical channel value of the input image data; I represents the input image data; y∈Ω(x) represents the window centered at pixel x; I C (y) represents the RGB channels of the input image data; This represents the maximum value of the corresponding pixel position in the RGB channel of the input image data.

[0023] After processing, the image data changed from RGB three-channel to single-channel. It is a smoothing filter that performs a smoothing filter operation on a single-channel image after the maximum value extraction operation, with a window centered on pixel x as the size, and uses the maximum value in the window to cover the value of the center pixel x.

[0024] The optical channel mean AVLC is expressed as:

[0025]

[0026] Among them, D i,j represents the optical channel value of the pixel at the i-th row and j-th column of the image; w represents the width of the input object; h represents the height of the input object.

[0027] In step S23, an evaluation function is established based on the confidence level and the mean value of the optical channel output by the detection module. This evaluation function includes two branches.

[0028] The first branch of the evaluation function is the score, which is based on the confidence level output by the detection module. d It is represented as:

[0029]

[0030] Where confidence represents the confidence level of all detected objects after the original image is output by the detection subnetwork; α∈(0,1] is a hyperparameter representing the weight; i represents the i-th detected object; n represents the number of detected objects output by the regular detection subnetwork or the highlight detection subnetwork; in this branch, the evaluation methods of the N-YOLO detection subnetwork and the S-YOLO detection subnetwork are the same.

[0031] The second branch of the evaluation function is based on the average optical channel value of the image frame. For the image itself, high-brightness scenes, i.e., images with a high average optical channel value (AVLC), rely more on the detection results of the S-YOLO detection sub-network, while scenes with normal brightness, i.e., images with a low average optical channel value, rely more on the detection results of the N-YOLO detection sub-network. Based on different objective functions, different evaluation methods are set. The evaluation branches of the N-YOLO and S-YOLO detection sub-networks based on the average optical channel value of the image frame are expressed as follows:

[0032]

[0033] Where I represents the input image; AVLC represents the mean optical channel value of the i-th or j-th detected object; n represents the number of objects detected by the S-YOLO detection subnetwork; and m represents the number of objects detected by the N-YOLO detection subnetwork.

[0034] Based on the two branches of the evaluation function, the evaluation function is expressed as follows:

[0035]

[0036] Wherein, S (superscript and subscript) represents the S-YOLO detection subnetwork; N (superscript and subscript) represents the N-YOLO detection subnetwork; α represents the scoring weight of the detection subnetwork; confidence represents the confidence level of the object detected by the detection subnetwork; and AVLC is the optical channel mean.

[0037] A Siamese network discrimination system is built based on the above evaluation function. The target detection box of the corresponding detection sub-network is selected as the output according to the evaluation score and provided to the tracking module based on the predicted potential occlusion image frame for trajectory association matching.

[0038] According to the above-described multi-target tracking method for sea surface based on twin network target detection, preferably, step S3 includes the following specific steps:

[0039] S31. Perform Kalman filtering prediction on the state variables of the trajectory instance to obtain the current frame prediction box coordinates of the trajectory.

[0040] S32. Based on the predicted state variables, perform two more Kalman filter predictions without changing any parameters to obtain the predicted bounding box coordinates of the next frame and the frame after that of the trajectory.

[0041] S33. Calculate the IOU value between each pair of prediction boxes in the next frame. Perform the same calculation operation on all prediction boxes in the frame after next. Determine whether there are trajectory instances with IOU values ​​exceeding the threshold. If there are, mark the current frame as a potential occluded image frame. If not, mark the current frame as an unoccluded image frame.

[0042] According to the above-described multi-target tracking method for sea surface based on twin network target detection, preferably, in step S31, the state variable of the trajectory instance is represented as:

[0043]

[0044] Where (x,y) represents the center position of the detection box, w represents the width of the detection box, and h represents the height of the detection box. Based on this, Kalman filtering is performed to predict the trajectory position in the current frame.

[0045] In step S32, the current values ​​of the Kalman filter parameters of the trajectory instances are kept, and two additional predictions are performed to obtain the predicted positions of all trajectory instances in the next frame and the frame after that.

[0046] In step S33, the possibility of potential occlusion between two trajectory instances is measured by calculating the Intersection over Union (IOU) between each predicted bounding box and other predicted bounding boxes. When the IOU value between predicted bounding boxes of trajectory instances exceeds the threshold of 0.7, the current frame is marked as a potentially occluded image frame; otherwise, the current frame is marked as an unoccluded image frame.

[0047] According to the above-described multi-target tracking method for sea surface based on twin network target detection, preferably, step S4 includes the following specific steps:

[0048] S41. For an unoccluded image frame, perform an IOU calculation between each pair of the detection bounding boxes output by the current frame detection module and the predicted bounding boxes of all trajectory instances to obtain the distance metric between the trajectory instances and the detected object.

[0049] S42. For potentially occluded image frames, appearance modeling is performed using the appearance descriptors of trajectory instance objects and objects within the detection box. Cosine similarity is calculated between the appearance attributes of the trajectory and the appearance attributes of the detected objects in the current image frame to obtain the appearance metric between the trajectory instance and the detected objects.

[0050] S43. Construct a loss function matrix based on distance and appearance metrics, and use this matrix to perform Hungarian matching to achieve maximum bipartite matching between trajectory instances and detected objects.

[0051] According to the above-described method for multi-target tracking on the sea surface based on Siamese network target detection, preferably, in step S41, in an unobstructed image frame, the distance between the detection box output by the current frame detection module and the predicted box of the trajectory instance in the current frame is measured, which is expressed as:

[0052]

[0053] Among them, s d Represents a distance metric; w di and h di Let w represent the width and height of the i-th detection box, respectively; pj and h pj Let w represent the width and height of the j-th prediction box, respectively; o and h o These represent the width and height of the overlapping portion of the detection box and the prediction box, respectively.

[0054] In step S42, in the image frame marked as potential occlusion, a trajectory instance and an appearance metric of the detected object are introduced; for the detected object d i Through a 128-dimensional appearance descriptor vector r di This describes the appearance features of an object in the current frame. The appearance descriptor vector is calculated by a pre-trained CNN network and satisfies ||r di || = 1; for trajectory instance t j The appearance descriptor update method is represented as:

[0055]

[0056] in, Represents the modulus normalization function; r tj The appearance descriptor representing a trajectory instance; r dk This represents the new appearance descriptor associated with the detected object; β1 and β2 represent the fusion update weights, with β1 and β2 taking values ​​of 0.2 and 0.8 respectively.

[0057] The appearance similarity between the detected object and the trajectory instance is represented by cosine distance, and the appearance metric between the trajectory instance and the detected object is expressed as follows:

[0058]

[0059] Among them, s a Indicates appearance measurement; r di Represents the appearance descriptor of the i-th detected object; r tj This represents the appearance descriptor of the j-th instance object.

[0060] In step S43, the loss function matrix constructed using distance and appearance metrics as features is expressed as follows:

[0061] c i,j =λ1×s d (i,j)+λ2×s a (i,j);

[0062] Among them, c i,j λ1 represents the loss between the i-th detected object and the j-th trajectory instance. For unoccluded image frames, λ1 takes the value 1 while λ2 takes the value 0, depending on the motion features of the object. For potentially occluded image frames, λ1 takes the value 0 while λ2 takes the value 1, depending on the appearance features of the object. Hungarian matching is performed through the loss function matrix to achieve the matching association between the detected object and the trajectory instance.

[0063] According to the above-described multi-target tracking method for sea surface based on twin network target detection, preferably, step S5 includes the following specific steps:

[0064] For trajectory instances that successfully match the detected object, the Kalman filter parameters are updated based on the observed values ​​of the detected object; for trajectory instances that fail to match the detected object, it is determined whether the cumulative number of consecutive failed matches has reached a threshold. Trajectory instances exceeding the threshold are deleted, while trajectory instances that have not exceeded the threshold are not updated.

[0065] In step S5, the tracking information data of the sea surface scene is stored in real time and stored in offline video or text.

[0066] A second aspect of the present invention provides an electronic device, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of the sea surface multi-target tracking method based on twin network target detection as described in the first aspect.

[0067] A third aspect of the present invention provides a computer-readable storage medium, characterized in that the computer-readable storage medium stores a computer program, which, when executed by a computer processor, implements any step of the sea surface multi-target tracking method based on twin network target detection as described in the first aspect.

[0068] Compared with the prior art, the beneficial effects of the present invention are as follows:

[0069] 1. This invention targets video data of sea surface scenes. Through a detection module based on twin networks and a tracking module that predicts potential occlusion image frames, the video data is processed frame by frame to extract tracking information data of the sea surface scene. Without the need to optimize hardware, the accuracy and robustness of multi-target tracking are optimized through algorithms, thereby saving a lot of costs.

[0070] 2. This invention proposes a conventional N-YOLO detection subnetwork and a high-brightness S-YOLO detection subnetwork with consistent network structure, which can be directly applied to sea surface scenes with changing weather or severe water surface reflection, thus avoiding the poor detection results caused by the sea surface environment to a certain extent. This invention improves the problem of inter-object occlusion by predicting the position of trajectory instances in the next two frames and applying different tracking strategies to unoccluded image frames and potentially occluded image frames, thereby improving the overall tracking performance of the system.

[0071] 3. Facing the challenges of complex environments and occlusion phenomena, this invention has high robustness and flexibility, and can respond quickly during operation. It can quickly extract high-precision sea surface scene tracking information in complex sea environments and occlusion phenomena, thereby achieving effective tracking of multiple targets on the sea surface under complex sea weather changes and occlusion conditions.

[0072] 4. This invention addresses the problem of existing multi-target tracking and perception methods in marine scenes struggling to robustly track target objects under weather changes and occlusion conditions. This invention trains detection modules based on a Siamese network to specifically identify normal-brightness and high-brightness marine scenes. A Siamese network discrimination system is built based on an evaluation function, selectively outputting the sub-network detection results to obtain robust target detection information. Kalman filtering prediction is performed based on the current position of the tracking root trajectory object, with additional position prediction for the next two frames. The Intersection over Union (IOU) value between predicted frames is calculated. Unoccluded and potentially occluded image frames are marked with a threshold, and motion feature modeling or appearance feature modeling is applied based on the markings. Hungarian matching is performed based on the loss function matrix obtained from the modeling to extract various tracking information of objects in the marine scene. This invention achieves highly robust multi-target tracking in marine scenes under weather changes and occlusion conditions, and can be applied to perception of marine scenes in complex environments. Attached Figure Description

[0073] Figure 1 This is a schematic diagram of the process of the present invention;

[0074] Figure 2 A schematic diagram of the twin network process for the detection module;

[0075] Figure 3 A flowchart illustrating the process of predicting whether a trajectory instance has potential occlusion. Detailed Implementation

[0076] The present invention will be further described in detail below through specific embodiments, but this does not limit the scope of the present invention.

[0077] Example 1

[0078] A method for multi-target tracking on the sea surface based on Siamese network target detection, such as Figure 1-3 As shown, it includes the following steps:

[0079] S1. Based on the characteristic that weather changes can easily cause water surface reflection, the sea surface scene dataset is divided into normal scene data and bright scene data. The normal scene data is used to train the N-YOLO detection sub-network, and the bright scene data is used to train the S-YOLO detection sub-network. A twin network based on the N-YOLO detection sub-network and the S-YOLO detection sub-network is constructed.

[0080] In step S1, the N-YOLO detection subnetwork and the S-YOLO detection subnetwork are completely consistent with the network framework of YOLOv5. The weights of the N-YOLO detection subnetwork are trained using a regular scene image dataset, and the weights of the S-YOLO detection subnetwork are trained using a highlight scene image dataset.

[0081] S2. Input the video data collected by the camera into the detection module based on the Siamese network. Build a Siamese network discrimination system based on the evaluation function. Selectively output the detection results of the N-YOLO detection subnetwork and the S-YOLO detection subnetwork to obtain robust target detection information.

[0082] The specific steps of step S2 are as follows:

[0083] S21. Use a twin network to perform target detection on each frame of video data, and obtain the output results of the N-YOLO detection subnetwork and the S-YOLO detection subnetwork, namely the target detection box coordinates and confidence scores.

[0084] In step S21, the N-YOLO detection subnetwork and the S-YOLO detection subnetwork use non-shared network weights. After inputting the same image data of the sea scene into the N-YOLO detection subnetwork and the S-YOLO detection subnetwork respectively, different image detection results are output, and corresponding object detection boxes and confidence scores are obtained.

[0085] S22. Calculate the optical channel value pixel by pixel for each frame of image data to obtain the optical channel mean (AVLC) and obtain the brightness information representing the image.

[0086] In step S22, to evaluate the brightness of the input image, optical channel characteristics are introduced as a scoring criterion; the optical channel values ​​of the input image data are expressed as:

[0087]

[0088] Where D represents the optical channel value of the input image data; I represents the input image data; y∈Ω(x) represents the window centered at pixel x; I C (y) represents the RGB channels of the input image data; This represents the maximum value of the corresponding pixel position in the RGB channel of the input image data.

[0089] After processing, the image data changed from RGB three-channel to single-channel. It is a smoothing filter that performs a smoothing filter operation on a single-channel image after the maximum value extraction operation, with a window centered on pixel x as the size, and uses the maximum value in the window to cover the value of the center pixel x.

[0090] The optical channel mean AVLC is expressed as:

[0091]

[0092] Among them, D i,jrepresents the optical channel value of the pixel at the i-th row and j-th column of the image; w represents the width of the input object; h represents the height of the input object.

[0093] S23. Based on the confidence level and the mean of the optical channel, establish an evaluation function, build a twin network discrimination system, and select the target detection box of the corresponding detection sub-network as the output according to the evaluation score.

[0094] In step S23, an evaluation function is established based on the confidence level and the mean value of the optical channel output by the detection module. This evaluation function includes two branches.

[0095] The first branch of the evaluation function is the score, which is based on the confidence level output by the detection module. d It is represented as:

[0096]

[0097] Where confidence represents the confidence level of all detected objects after the original image is output by the detection subnetwork; α∈(0,1] is a hyperparameter representing the weight; i represents the i-th detected object; n represents the number of detected objects output by the regular detection subnetwork or the highlight detection subnetwork; in this branch, the evaluation methods of the N-YOLO detection subnetwork and the S-YOLO detection subnetwork are the same.

[0098] The second branch of the evaluation function is based on the average optical channel value of the image frame. For the image itself, high-brightness scenes, i.e., images with a high average optical channel value (AVLC), rely more on the detection results of the S-YOLO detection sub-network, while scenes with normal brightness, i.e., images with a low average optical channel value, rely more on the detection results of the N-YOLO detection sub-network. Based on different objective functions, different evaluation methods are set. The evaluation branches of the N-YOLO and S-YOLO detection sub-networks based on the average optical channel value of the image frame are expressed as follows:

[0099]

[0100] Where I represents the input image; AVLC represents the mean optical channel value of the i-th or j-th detected object; n represents the number of objects detected by the S-YOLO detection subnetwork; and m represents the number of objects detected by the N-YOLO detection subnetwork.

[0101] Based on the two branches of the evaluation function, the evaluation function is expressed as follows:

[0102]

[0103] Wherein, S (superscript and subscript) represents the S-YOLO detection subnetwork; N (superscript and subscript) represents the N-YOLO detection subnetwork; α represents the scoring weight of the detection subnetwork; confidence represents the confidence level of the object detected by the detection subnetwork; and AVLC is the optical channel mean.

[0104] Specifically, the same image frame of the sea scene is used as input to both the S-YOLO and N-YOLO detection sub-networks. From the perspective of detection results, the corresponding object localization and confidence scores are obtained respectively. After passing through the evaluation branch based on the confidence scores output by the detection modules, the evaluation branch score is obtained. d The brightness of the detected object is calculated from the perspective of the overall composition of the image frame. After passing through an evaluation branch based on the average light channel value of the image frame, the evaluation branch score is obtained. S and score N Based on the above two branches, the corresponding complete evaluation functions of the S-YOLO detection subnetwork and N-YOLO detection subnetwork are calculated. The higher the score, the more reliable the detection result of the detection subnetwork is, and the lower the score, the less reliable the detection result is. A Siamese network discrimination system is built according to the above evaluation functions. The target detection box of the corresponding detection subnetwork is selected as the output according to the evaluation score and provided to the tracking module of the predicted potential occlusion image frame for trajectory association matching.

[0105] S3. Perform Kalman filtering prediction on the current position of the tracked object, and make additional predictions for the next two frames of images. Calculate the IOU value between the predicted boxes, and mark the unoccluded or potentially occluded image frames according to the threshold.

[0106] The specific steps of step S3 are as follows:

[0107] S31. Perform Kalman filtering prediction on the state variables of the trajectory instance to obtain the current frame prediction box coordinates of the trajectory.

[0108] In step S31, the state variables of the trajectory instance are represented as follows:

[0109]

[0110] Where (x,y) represents the center position of the detection box, w represents the width of the detection box, and h represents the height of the detection box. Based on this, Kalman filtering is performed to predict the trajectory position in the current frame.

[0111] S32. Based on the predicted state variables, perform two more Kalman filter predictions without changing any parameters to obtain the predicted bounding box coordinates of the next frame and the frame after that of the trajectory.

[0112] In step S32, the current values ​​of the Kalman filter parameters of the trajectory instances are kept, and two additional predictions are performed to obtain the predicted positions of all trajectory instances in the next frame and the frame after that.

[0113] S33. Calculate the IOU value between each pair of prediction boxes in the next frame. Perform the same calculation operation on all prediction boxes in the frame after next. Determine whether there are trajectory instances with IOU values ​​exceeding the threshold. If there are, mark the current frame as a potential occluded image frame. If not, mark the current frame as an unoccluded image frame.

[0114] In step S33, the possibility of potential occlusion between two trajectory instances is measured by calculating the Intersection over Union (IOU) between each predicted bounding box and other predicted bounding boxes. When the IOU value between predicted bounding boxes of trajectory instances exceeds the threshold of 0.7, the current frame is marked as a potentially occluded image frame; otherwise, the current frame is marked as an unoccluded image frame.

[0115] S4. Model motion features for unoccluded image frames and appearance features for potentially occluded image frames to construct a loss function matrix, and perform correlation matching between trajectories and detected objects based on Hungarian matching.

[0116] The specific steps of step S4 are as follows:

[0117] S41. For an unoccluded image frame, perform an IOU calculation between each pair of the detection bounding boxes output by the current frame detection module and the predicted bounding boxes of all trajectory instances to obtain the distance metric between the trajectory instances and the detected object.

[0118] In step S41, in an unoccluded image frame, a distance metric is performed between the detection bounding box output by the current frame detection module and the predicted bounding box of the current frame trajectory instance, which is expressed as:

[0119]

[0120] Among them, s d Represents a distance metric; w di and h di Let w represent the width and height of the i-th detection box, respectively; pj and h pj Let w represent the width and height of the j-th prediction box, respectively; o and h o These represent the width and height of the overlapping portion of the detection box and the prediction box, respectively.

[0121] S42. For potentially occluded image frames, appearance modeling is performed using the appearance descriptors of trajectory instance objects and objects within the detection box. Cosine similarity is calculated between the appearance attributes of the trajectory and the appearance attributes of the detected objects in the current image frame to obtain the appearance metric between the trajectory instance and the detected objects.

[0122] In step S42, in the image frame marked as potential occlusion, a trajectory instance and an appearance metric of the detected object are introduced; for the detected object d i Through a 128-dimensional appearance descriptor vector r di This describes the appearance features of an object in the current frame. The appearance descriptor vector is calculated by a pre-trained CNN network and satisfies ||r di || = 1; for trajectory instance t j The appearance descriptor update method is represented as:

[0123]

[0124] in, Represents the modulus normalization function; r tj The appearance descriptor representing a trajectory instance; r dk This represents the new appearance descriptor associated with the detected object; β1 and β2 represent the fusion update weights, with β1 and β2 taking values ​​of 0.2 and 0.8 respectively.

[0125] The appearance similarity between the detected object and the trajectory instance is represented by cosine distance, and the appearance metric between the trajectory instance and the detected object is expressed as follows:

[0126]

[0127] Among them, s a Indicates appearance measurement; r di Represents the appearance descriptor of the i-th detected object; r tj This represents the appearance descriptor of the j-th instance object.

[0128] S43. Construct a loss function matrix based on distance and appearance metrics, and use this matrix to perform Hungarian matching to achieve maximum bipartite matching between trajectory instances and detected objects.

[0129] In step S43, the loss function matrix constructed using distance and appearance metrics as features is expressed as follows:

[0130] c i,j =λ1×s d (i,j)+λ2×s a (i,j);

[0131] Among them, c i,j λ1 represents the loss between the i-th detected object and the j-th trajectory instance. For unoccluded image frames, λ1 takes the value 1 while λ2 takes the value 0, depending on the motion features of the object. For potentially occluded image frames, λ1 takes the value 0 while λ2 takes the value 1, depending on the appearance features of the object. Hungarian matching is performed through the loss function matrix to achieve the matching association between the detected object and the trajectory instance.

[0132] Specifically, in this example, under the premise of good camera stability, the trajectory motion of the detected objects in the sea scene is more linear than that of pedestrians. Therefore, for unoccluded image frames, the loss function matrix is ​​constructed by relying on motion feature modeling. In addition, the appearance differences between the detected objects in the sea scene are quite obvious. Therefore, for potentially occluded image frames, relying on appearance feature modeling can robustly associate the occluded object with the old trajectory when it reappears.

[0133] S5. Update the Kalman filter parameters of the trajectory object based on the associated results, and summarize the target ID, category, and trajectory information in the sea surface scene.

[0134] The specific steps of step S5 are as follows:

[0135] For trajectory instances that successfully match the detected object, the Kalman filter parameters are updated based on the observed values ​​of the detected object; for trajectory instances that fail to match the detected object, it is determined whether the cumulative number of consecutive failed matches has reached a threshold. Trajectory instances exceeding the threshold are deleted, while trajectory instances that have not exceeded the threshold are not updated.

[0136] In step S5, the tracking information data of the sea surface scene is stored in real time and stored in offline video or text.

[0137] Specifically: The Siamese network, constructed from a conventional N-YOLO detection subnetwork and a high-brightness S-YOLO detection subnetwork, can directly locate and detect targets in the sea scene from the original image frames. It can perform high-speed computation without the need for expensive preprocessing operations such as image enhancement, and is suitable for image recognition under different lighting conditions. The multi-target tracking framework based on occlusion prediction of the next two image frames can adopt different tracking strategies for unoccluded image frames and potentially occluded image frames, effectively improving the target tracking effect under occlusion conditions.

[0138] This embodiment processes the video of the sea scene frame by frame. The image frames are input into the detection module based on the Siamese network, and the branch result corresponding to the high score is selected as the target detection information according to the evaluation function. The target detection information is input into the tracking module based on the predicted potential occlusion image frames, and the image frames are marked as unoccluded or potentially occluded, and the corresponding tracking strategy is implemented. Finally, for trajectory instances that successfully match the detected object, the Kalman filter parameters are updated according to the observed value of the detected object. For trajectory instances that fail to match the detected object, it is determined whether the cumulative number of consecutive matching failures has reached the threshold. Trajectory instances that exceed the threshold are deleted, and trajectory instances that do not exceed the threshold are not updated.

[0139] Example 2

[0140] An electronic device includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement any step of the sea surface multi-target tracking method based on twin network target detection as described in Embodiment 1.

[0141] The hardware of the electronic device in this embodiment also includes a GPU, a display buffer memory, a RAMD / A converter, and a heat sink that work in conjunction with the processor. The GPU is responsible for processing the graphics display of the electronic device, providing image rendering and acceleration functions, and using its parallel computing advantages to accelerate the processing of large-scale data-intensive tasks such as deep learning training and scientific computing.

[0142] Furthermore, the process of the sea surface multi-target tracking method based on twin network target detection described in Embodiment 1 can be implemented as a computer software program. For example, this embodiment includes a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the method. In such an embodiment, the computer program can be downloaded and installed from a network, and / or installed from a removable medium. When the computer program is executed by a processor, it performs the functions defined in the method of this application.

[0143] Example 3

[0144] A computer-readable storage medium storing a computer program that, when executed by a processor, implements any step of a method for tracking multiple targets on the sea surface based on twin network target detection as described in Embodiment 1.

[0145] The computer-readable medium described in this application may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In this application, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In this application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. Computer-readable signal media can also be any computer-readable medium other than computer-readable storage media, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wireless, wire, optical fiber, RF, etc., or any suitable combination thereof.

[0146] Computer program code for performing the operations of this application can be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages ​​such as Python and C++, as well as conventional procedural programming languages ​​or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0147] In this embodiment, the computer-readable storage medium can be accelerated using hardware such as a GPU. The parallel computing advantage of the GPU is used to accelerate any step in the implementation of a multi-target tracking method for sea surface based on twin network target detection as described in Embodiment 1.

[0148] In summary, this invention effectively overcomes the shortcomings of the prior art and has high industrial applicability. The above embodiments are intended to illustrate the substantive content of this invention, but are not intended to limit the scope of protection of this invention. Those skilled in the art should understand that modifications or equivalent substitutions can be made to the technical solutions of this invention without departing from the essence and scope of protection of this invention.

[0149] The above embodiments are specific implementations of the present invention, but the implementation of the present invention is not limited to the above embodiments. Any other combination, change, modification, substitution, or simplification that does not exceed the design concept of the present invention shall fall within the protection scope of the present invention.

Claims

1. A method for sea surface multi-target tracking based on twin network object detection, characterized in that, Includes the following steps: S1. Divide the sea surface scene dataset into regular scene data and highlighted scene data. Regular scene data is used to train the N-YOLO detection sub-network, and highlighted scene data is used to train the S-YOLO detection sub-network. Construct a Siamese network based on the N-YOLO detection sub-network and the S-YOLO detection sub-network. S2. Input the video data collected by the camera into the detection module, build a twin network discrimination system based on the evaluation function, and selectively output the detection results of the N-YOLO detection subnetwork and the S-YOLO detection subnetwork to obtain robust target detection information; S3. Perform Kalman filtering prediction on the current position of the tracked object, and perform additional prediction on the next two frames of images. Calculate the IOU value between the predicted boxes, and mark the unoccluded or potentially occluded image frames according to the threshold. S4. Model motion features for unoccluded image frames and appearance features for potentially occluded image frames to construct a loss function matrix, and perform correlation matching between trajectories and detected objects based on Hungarian matching. S5. Update the Kalman filter parameters of the trajectory object based on the correlation results, and summarize the target ID, category, and trajectory information in the sea surface scene; The specific steps of step S2 are as follows: S21. Use a twin network to perform target detection on each frame of video data, and obtain the output results of the N-YOLO detection subnetwork and the S-YOLO detection subnetwork, namely the target detection box coordinates and confidence scores. S22. Calculate the optical channel value pixel by pixel for each frame of image data to obtain the optical channel mean (AVLC) and obtain the brightness information representing the image. S23. Based on the confidence level and the mean of the optical channel, establish an evaluation function, build a twin network discrimination system, and select the target detection box of the corresponding detection sub-network as the output according to the size of the evaluation score; The specific steps of step S3 are as follows: S31. Perform Kalman filter prediction on the state variables of the trajectory instance to obtain the current frame prediction box coordinates of the trajectory; S32. Based on the predicted state variables, perform two more Kalman filter predictions without changing any parameters to obtain the predicted bounding box coordinates of the next frame and the frame after that of the trajectory. S33. Calculate the IOU value between each pair of prediction boxes in the next frame. Perform the same calculation operation on all prediction boxes in the frame after next. Determine whether there are trajectory instances with IOU values ​​exceeding the threshold. If there are, mark the current frame as a potential occluded image frame. If not, mark the current frame as an unoccluded image frame.

2. The sea surface multi-target tracking method based on twin network object detection according to claim 1, characterized in that, In step S21, the N-YOLO detection subnetwork and the S-YOLO detection subnetwork use non-shared network weights. After inputting the same image data of the sea scene into the N-YOLO detection subnetwork and the S-YOLO detection subnetwork respectively, different image detection results are output, and corresponding object detection boxes and confidence scores are obtained. In step S22, the optical channel values ​​of the input image data are represented as follows: ； wherein D represents the light channel value of the input image data; I represents the input image data; represents a window centered at pixel point x; represents the RGB channel of the input image data; represents the maximum value of the corresponding pixel position in the RGB channel of the input image data; After processing, the image data changed from RGB three-channel to single-channel. It is a smoothing filter that performs a smoothing filter operation on a single-channel image after the maximum value extraction operation, with a window centered on pixel x as the size, and uses the maximum value in the window to cover the value of the center pixel x; The optical channel mean AVLC is expressed as: ； in, represents the optical channel value of the pixel at the i-th row and j-th column of the image; w represents the width of the input object; h represents the height of the input object; In step S23, an evaluation function is established based on the confidence level and the mean value of the optical channel output by the detection module. This evaluation function includes two branches. The first branch of the evaluation function is the evaluation branch based on the confidence score output by the detection module. It is represented as: ； Where confidence represents the confidence level of all detected objects after the original image is output by the detection sub-network; is a hyperparameter representing weights; i represents the i-th detection object; n represents the number of detection objects output by the regular detection subnetwork or the highlight detection subnetwork; in this branch, the evaluation methods for the N-YOLO detection subnetwork and the S-YOLO detection subnetwork are the same; The second branch of the evaluation function is an evaluation branch based on the optical channel mean of the image frame. The evaluation branches of the N-YOLO detection subnetwork and the S-YOLO detection subnetwork based on the optical channel mean of the image frame are expressed as follows: ； ； Where I represents the input image; AVLC represents the mean optical channel value of the i-th or j-th detected object; n represents the number of objects detected by the S-YOLO detection subnetwork; and m represents the number of objects detected by the N-YOLO detection subnetwork. Based on the two branches of the evaluation function, the evaluation function is expressed as follows: ； ； Wherein, the superscript and subscript S represent the S-YOLO detection subnetwork; the superscript and subscript N represent the N-YOLO detection subnetwork; This indicates the scoring weight of the detection subnetwork; confidence represents the confidence level of the object detected by the detection subnetwork; AVLC is the optical channel mean. Based on the evaluation function described above, a Siamese network discrimination system is built. The target detection box of the corresponding detection sub-network is selected as the output according to the evaluation score and provided to the tracking module for trajectory association and matching.

3. The multi-target tracking method for sea surface based on Siamese network target detection according to claim 2, characterized in that, In step S31, the state variables of the trajectory instance are represented as follows: ； in, The center position of the detection box is represented by w, the width of the detection box is represented by h, and the height of the detection box is represented by h. Based on this, Kalman filtering is performed to predict the trajectory position in the current frame. In step S32, the current values ​​of the Kalman filter parameters of the trajectory instances are kept, and two additional predictions are performed to obtain the predicted positions of all trajectory instances in the next frame and the frame after that. In step S33, the possibility of potential occlusion between two trajectory instances is measured by calculating the Intersection over Union (IOU) between each predicted bounding box and other predicted bounding boxes. When the IOU value between predicted bounding boxes of trajectory instances exceeds the threshold, the current frame is marked as a potentially occluded image frame; otherwise, the current frame is marked as an unoccluded image frame.

4. The multi-target tracking method for sea surface based on twin network target detection according to claim 1, characterized in that, The specific steps of step S4 are as follows: S41. For an unoccluded image frame, perform an IOU calculation between each pair of the detection box output by the current frame detection module and the predicted boxes of all trajectory instances to obtain the distance metric between the trajectory instance and the detected object. S42. For potentially occluded image frames, appearance modeling is performed by using the appearance descriptors of the trajectory instance objects and the objects in the detection box within the frame. Cosine similarity is calculated between the appearance attributes of the trajectory and the appearance attributes of the detected objects in the current image frame to obtain the appearance metric between the trajectory instance and the detected objects. S43. Construct a loss function matrix based on distance and appearance metrics, and use this matrix to perform Hungarian matching to achieve maximum bipartite matching between trajectory instances and detected objects.

5. The multi-target tracking method for sea surface based on twin network target detection according to claim 4, characterized in that, In step S41, in an unoccluded image frame, a distance metric is performed between the detection bounding box output by the current frame detection module and the predicted bounding box of the current frame trajectory instance, which is expressed as: ； in, Represents a distance metric; and Let represent the width and height of the i-th detection box, respectively; and Let represent the width and height of the j-th prediction box, respectively; and These represent the width and height of the overlapping portion of the detection box and the prediction box, respectively; In step S42, in the image frame marked as potential occlusion, a trajectory instance and an appearance metric of the detected object are introduced; for the detected object Through appearance descriptor vector To describe the appearance features of an object in the current frame; for trajectory instances The appearance descriptor update method is represented as: ； in, Represents the modulus normalization function; A descriptor representing the appearance of a trajectory instance; This indicates a new appearance descriptor associated with the detected object; and These represent the fusion update weights; The appearance similarity between the detected object and the trajectory instance is represented by cosine distance, and the appearance metric between the trajectory instance and the detected object is expressed as follows: ； in, Indicates appearance measurement; This represents the appearance descriptor of the i-th detected object; This represents the appearance descriptor of the j-th instance object; In step S43, the loss function matrix constructed using distance and appearance metrics as features is expressed as follows: ； in, This represents the loss between the i-th detected object and the j-th trajectory instance. For an unoccluded image frame, Takes the value 1 A value of 0 indicates modeling based on the motion features of the object; for potentially occluded image frames, The value is 0 and The value is 1, which depends on the appearance features of the object to be modeled; Hungarian matching is performed through the loss function matrix to achieve the matching association between the detected object and the trajectory instance.

6. The method for multi-target tracking on the sea surface based on twin network target detection according to claim 1, characterized in that, The specific steps of step S5 are as follows: For trajectory instances that successfully match the detected object, update the Kalman filter parameters based on the observed values ​​of the detected object; for trajectory instances that fail to match the detected object, determine whether the cumulative number of consecutive failed matches has reached the threshold, delete trajectory instances that exceed the threshold, and do not perform update operations on trajectory instances that do not exceed the threshold. In step S5, the tracking information data of the sea surface scene is stored in real time and stored in offline video or text.

7. An electronic device, comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements any step in the sea surface multi-target tracking method based on twin network target detection as described in any one of claims 1-6.

8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program, which, when executed by a computer processor, implements any step in the sea surface multi-target tracking method based on twin network target detection as described in any one of claims 1-6.

Images