A target tracking method based on target detection applied on unmanned ship

By combining YOLO and Darknet53 on the unmanned ship for target detection and feature extraction, and using the MMS structure to transform the feature matrix, the problem of inaccurate target tracking and high computational cost on the unmanned ship is solved, and efficient and accurate target tracking is achieved.

CN113657215BInactive Publication Date: 2024-11-15DALIAN UNIV OF TECH
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202110878972.0
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2021-08-02
Publication Date
2024-11-15
Estimated Expiration
Not applicable · inactive patent

AI Technical Summary

Technical Problem

The existing multi-objective tracking method based on target detection has problems such as high computational cost and inaccurate target tracking in unmanned ship applications, especially in the case of ship swaying, the traditional IOU matching method is not applicable.

Method used

The YOLO and Darknet53 structures are used for object detection and feature extraction. The MMS structure is used to convert the feature matrix with non-fixed size into a fixed size matrix, and the similarity comparison is used for target tracking. The fusion detection and tracking process is integrated to reduce the number of feature calculations. .

Benefits of technology

Efficient and real-time target tracking on unmanned ships is achieved, which saves calculation costs and improves tracking accuracy in ship swaying, avoids wasting computing power by repeatedly calculating image feature maps.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN113657215B_ABST
    Figure CN113657215B_ABST
Patent Text Reader

Abstract

The present invention relates to a target tracking method, a target tracking method based on target detection and applied to an unmanned ship, comprising the following steps: (1) using YOLO and the Darknet53 structure therein to perform target detection and feature value extraction on a target, (2) mapping and extracting the feature value of a detection target window, (3) converting a feature matrix of uncertain size, length and width into a feature matrix of the same size through an MMS structure, (4) repeating steps 2 and 3 to save the feature matrix #imgabs0# obtained for each target window, (5) performing steps 1, 2, 3 and 4 operations on the second frame original image taken by a camera to obtain the feature matrix #imgabs1# of each target window, and (6) performing similarity comparison on the target windows in the first and second frames original images to determine whether they are the same target. The present invention integrates target detection and tracking, and the feature value that only needs to be calculated once can be used for target detection and tracking at the same time, saving calculation costs.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention relates to a target tracking method, and more particularly to a target tracking method based on target detection and applied to an unmanned ship. Background Art

[0002] At present, the multi-target tracking methods based on target detection all detect the current frame target by the detector (YOLO, etc.), obtain the confidence and bounding-box, and use the deep neural network to extract the features of each detected target. Then, based on the tracking frame of the previous frame tracker, the target position of the current frame is predicted and matched with the detector. This multi-target tracking method mainly has the following two technical problems: First, this tracking algorithm separates the detection and tracking problems, and needs to calculate the feature values ​​of the input image twice, which increases the computing cost. Second, it is not applicable to the target tracking method used for unmanned ships. The reason is that during the voyage of the ship, there will be swaying, and the offset of the target in the two frames captured by the sensor is too large, so it is not appropriate to use IOU to judge whether the targets of the two frames match. Summary of the invention

[0003] In order to overcome the shortcomings of the prior art, the present invention aims to provide a target tracking method based on target detection for use on unmanned ships. The method mainly solves the problem of inaccurate target tracking when the ship is swaying and the problem of wasting computing power by performing feature calculations on images multiple times.

[0004] In order to achieve the above-mentioned invention object and solve the problems existing in the prior art, the technical solution adopted by the present invention is: a target tracking method based on target detection applied to an unmanned ship, comprising the following steps:

[0005] Step 1: Use YOLO and the Darknet53 structure to detect and extract features of the target. Define the image captured by the camera as the original image. When the target tracking starts, pass the first frame of the original image captured by the camera into the yolov3 network, output the target detection result of yolov3 and the 32×32×512 dimensional feature map obtained by the 19th residual structure in the Darknet53 feature extraction network. Save the types and location information of each detected target;

[0006] Step 2: Map and extract the feature values ​​of the detected target window, and pass the first frame of the original image into the yolov3 network to detect all the target windows w 1 ,w 2 ,L,w n , which is mapped to the dimensional feature map obtained by the Darknet53 feature extraction network The feature matrix of all target windows is obtained by first converting the first target window w 1 The obtained feature matrix Extract it and perform a 2-time upsampling operation. The specific mapping relationship is described by formula (1):

[0007] (x,y)=(S*x′,S*y′) (1)

[0008] The specific feature extraction is described by equations (2) and (3).

[0009] Window left, top:

[0010]

[0011] Right and bottom of the window:

[0012]

[0013] Where S is the product of all strides before and including the 19th residual structure in the Darknet53 feature extraction network, (x, y) is the coordinate of a point on the original image of the first frame, x is the horizontal coordinate value of the original image of the first frame, y is the vertical coordinate value of the original image of the first frame, and x′ is the dimensional feature map The corresponding horizontal axis value on the left is y′, which is the dimension feature map. The corresponding vertical coordinate value on the top;

[0014] Step 3: Use the MMS structure to transform the feature matrix with uncertain size, length and width into a feature matrix of the same size, taking into account the detection target window w 1 Mapping to dimensional feature map The feature matrix obtained on The length and width are not necessarily equal. If the feature matrix If we perform a truncation operation to obtain a feature matrix with equal length and width, some information will be lost, making the similarity comparison inaccurate. The designed MMS (Mesh Maximum Selection) structure is passed in, so that the input feature matrix Fm of unfixed size is converted into 2 Transform to get feature matrix of the same size To achieve the subsequent similarity comparison operation, the feature matrix Fm before MMS is passed in 2 The dimension is a*b*512, which includes the following sub-steps:

[0015] (a) The first layer of MMS, for the feature matrix obtained above Find the maximum value in each depth dimension and get a 1×512 dimension matrix L 1 ;

[0016] (b) The second layer of MMS, the feature matrix Divide into four blocks of 2×2, where the specific division is described by formula (4):

[0017]

[0018] Where n is the current MMS layer number, a is the feature matrix Fm before entering MMS 2 The width in the dimension, b is the feature matrix Fm before passing into MMS 2 Height in dimension, is the width of the block in row i and column j in n*n blocks. is the height value of the block in the i-th row and j-th column of the n*n blocks. For each divided block, find its maximum value at each depth, and finally get a 4×512 dimension matrix L 2 ;

[0019] (c) The third layer MMS, the feature matrix Fm 2 Divide into nine blocks of 3×3, where the specific division is described by formula (4). For each divided block, find the maximum value at each depth, and finally get a 9×512 dimension matrix L 3 ;

[0020] (d) The dimension matrix L obtained by the 1st, 2nd and 3rd layer MMS 1 , L 2 , L 3 Splicing is performed to achieve the a*b*512 feature matrix that is never fixed Transformed to a 1×14×512 feature matrix

[0021] Step 4: Repeat steps 2 and 3 to obtain the 1×14×512 feature matrix for each target window Save it;

[0022] Step 5: Perform steps 1, 2, 3, and 4 on the second original image captured by the camera to obtain a 1×14×512 feature matrix for each target window.

[0023] Step 6: Compare the similarity of the target windows in the first and second frames of the original image to determine whether they are the same target and implement target tracking, which specifically includes the following sub-steps:

[0024] (a) The 1×14×512 feature matrix obtained by matching the first window of the first frame original image with the first target of the same category detected in the second frame original image Expanded into a 7168×1 dimension feature vector

[0025] (b) The two-dimensional feature vector Subtracting and then summing the absolute values ​​is equivalent to obtaining the two-dimensional feature vector Interpolated L 1 norm, which is equivalent to obtaining the two-dimensional feature vector The distance D w , described by formula (5),

[0026]

[0027] Where D w Represents a two-dimensional feature vector distance;

[0028] (c) D w Perform three full connections, and the last full connection is connected to a neuron. Take the sigmoid of the result of this neuron to get the similarity s, so that its value is between 0 and 1, representing the similarity between the two input images. Select the threshold t according to the actual project situation. According to the similarity, it can be judged whether it is the same object. The specific judgment of whether it is the same object is described by formula (6):

[0029]

[0030] (d) If they are the same object, the same ID is given to the bounding boxes obtained by target detection corresponding to the windows participating in the operation in the first and second frames of the original image to achieve target tracking. If they are not the same object, the first window in the first frame of the original image and the next target window of the same category detected in the next frame are subjected to the operations (a), (b), and (c) in the above step 6 to determine whether they are the same object, until all windows of the same category as the first window in the first frame of the original image are traversed;

[0031] (e) After the traversal is completed, steps 2, 3, 4, 5, and 6 are repeated for the next window in the first frame of the original image;

[0032] (f) After the comparison between the first frame original image and the second frame original image is completed, the next frame original image is compared with the second frame original image. The comparison method is the same as steps 1 to 6 above. And so on, the comparison of subsequent frames is performed to achieve target tracking.

[0033] The invention has the following beneficial effects: a target tracking method based on target detection applied to an unmanned ship, comprising the following steps: (1) using YOLO and the Darknet53 structure therein to detect the target and extract the feature value of the target, (2) mapping and extracting the feature value of the detected target window, (3) converting the feature matrix of uncertain size, length and width into a feature matrix of the same size through the MMS structure, (4) repeating steps 2 and 3 to obtain the 1×14×512 feature matrix of each target window Save it, (5) perform steps 1, 2, 3, and 4 on the second frame of the original image captured by the camera to obtain a 1×14×512 feature matrix for each target window (6) Compare the similarity of the target windows in the first and second frames of the original image to determine whether they are the same target, and implement target tracking. The present invention has the following advantages: First, target detection and target tracking are integrated, and the feature value that only needs to be calculated once can be used for target detection and target tracking at the same time, saving computing costs. Second, real-time tracking is performed, and the target position information is realized by the target detection network. The IOU factor is abandoned and only the feature value of the target detected in the image is used to determine whether it is the same target, which is more in line with target tracking in the case of ship swaying. Third, since the length and width of the window obtained by detection are not necessarily equal, the MMS structure involved can convert feature maps of non-fixed size and unequal length and width into fixed-size feature maps, so as to achieve subsequent similarity comparison and avoid repeated calculation of image feature maps to waste computing power. BRIEF DESCRIPTION OF THE DRAWINGS

[0034] Figure 1 It is a flow chart of the steps of the method of the present invention.

[0035] Figure 2 It is a structural block diagram of the MMS of the present invention. DETAILED DESCRIPTION

[0036] The present invention will be further described below in conjunction with the accompanying drawings.

[0037] like Figure 1 As shown, a target tracking method based on target detection applied to an unmanned ship includes the following steps:

[0038] Step 1: Use YOLO and the Darknet53 structure to detect and extract features of the target. Define the image captured by the camera as the original image. When the target tracking starts, pass the first frame of the original image captured by the camera into the yolov3 network, output the target detection result of yolov3 and the 32×32×512 dimensional feature map Fm obtained by the 19th residual structure in the Darknet53 feature extraction network. 1 , save the types and location information of each detected target;

[0039] Step 2: Map and extract the feature values ​​of the detected target window, and pass the first frame of the original image into the yolov3 network to detect all the target windows w 1 ,w 2 ,L,w n , which is mapped to the dimensional feature map obtained by the Darknet53 feature extraction network The feature matrix of all target windows is obtained by first converting the first target window w 1 The obtained feature matrix Extract it and perform a 2-time upsampling operation. The specific mapping relationship is described by formula (1):

[0040] (x,y)=(S*x′,S*y′) (1)

[0041] The specific feature extraction is described by equations (2) and (3).

[0042] Window left, top:

[0043]

[0044] Right and bottom of the window:

[0045]

[0046] Where S is the product of all strides before and including the 19th residual structure in the Darknet53 feature extraction network, (x, y) is the coordinate of a point on the original image of the first frame, x is the horizontal coordinate value of the original image of the first frame, y is the vertical coordinate value of the original image of the first frame, and x′ is the dimensional feature map The corresponding horizontal axis value on the left is y′, which is the dimension feature map. The corresponding vertical coordinate value on the top;

[0047] Step 3: Use the MMS structure to transform the feature matrix with uncertain size, length and width into a feature matrix of the same size, taking into account the detection target window w 1 Mapping to dimensional feature map The feature matrix obtained on The length and width are not necessarily equal. If the feature matrix If we perform a truncation operation to obtain a feature matrix with equal length and width, some information will be lost, making the similarity comparison inaccurate. The designed MMS (Mesh Maximum Selection) structure is passed in, so that the input feature matrix Fm of unfixed size is converted into 2 Transform to get feature matrix of the same size To achieve the subsequent similarity comparison operation, the feature matrix Fm before MMS is passed in 2 The dimension is a*b*512, which includes the following sub-steps:

[0048] (a) The first layer of MMS, for the feature matrix obtained above Find the maximum value in each depth dimension and get a 1×512 dimension matrix L 1 ;

[0049] (b) The second layer of MMS, the feature matrix Divide into four blocks of 2×2, where the specific division is described by formula (4):

[0050]

[0051] Where n is the current MMS layer number, a is the feature matrix Fm before entering MMS 2 The width in the dimension, b is the feature matrix Fm before passing into MMS 2 Height in dimension, is the width of the block in row i and column j in n*n blocks. is the height value of the block in the i-th row and j-th column of the n*n blocks. For each divided block, find its maximum value at each depth, and finally get a 4×512 dimension matrix L 2 ;

[0052] (c) The third layer MMS, the feature matrix Fm 2 Divide into nine blocks of 3×3, where the specific division is described by formula (4). For each divided block, find the maximum value at each depth, and finally get a 9×512 dimension matrix L 3 ;

[0053] (d) The dimension matrix L obtained by the 1st, 2nd and 3rd layer MMS 1 , L 2 , L 3 Splicing is performed to achieve the a*b*512 feature matrix that is never fixed Transformed to a 1×14×512 feature matrix

[0054] Step 4: Repeat steps 2 and 3 to obtain the 1×14×512 feature matrix for each target window Save it;

[0055] Step 5: Perform steps 1, 2, 3, and 4 on the second original image captured by the camera to obtain a 1×14×512 feature matrix for each target window.

[0056] Step 6: Compare the similarity of the target windows in the first and second frames of the original image to determine whether they are the same target and implement target tracking, which specifically includes the following sub-steps:

[0057] (c) The 1×14×512 feature matrix obtained by matching the first window of the first frame original image with the first target of the same category detected in the second frame original image Expanded into a 7168×1 dimension feature vector

[0058] (d) The two-dimensional feature vector Subtracting and then summing the absolute values ​​is equivalent to obtaining the two-dimensional feature vector The L1 norm of the interpolation is equivalent to obtaining the two-dimensional feature vector The distance D w , described by formula (5),

[0059]

[0060] Where D w Represents a two-dimensional feature vector distance;

[0061] (c) D w Perform three full connections, and the last full connection is connected to a neuron. Take the sigmoid of the result of this neuron to get the similarity s, so that its value is between 0 and 1, representing the similarity between the two input images. Select the threshold t according to the actual project situation. According to the similarity, it can be judged whether it is the same object. The specific judgment of whether it is the same object is described by formula (6):

[0062]

[0063] (d) If they are the same object, the same ID is given to the bounding boxes obtained by target detection corresponding to the windows participating in the operation in the first and second frames of the original image to achieve target tracking. If they are not the same object, the first window in the first frame of the original image and the next target window of the same category detected in the next frame are subjected to the operations (a), (b), and (c) in the above step 6 to determine whether they are the same object, until all windows of the same category as the first window in the first frame of the original image are traversed;

[0064] (e) After the traversal is completed, steps 2, 3, 4, 5, and 6 are repeated for the next window in the first frame of the original image;

[0065] (f) After the comparison between the first frame original image and the second frame original image is completed, the next frame original image is compared with the second frame original image. The comparison method is the same as steps 1 to 6 above. And so on, the comparison of subsequent frames is performed to achieve target tracking.

Claims

1. A target tracking method based on target detection applied to an unmanned ship, characterized in that The following steps are involved: Step 1: Use YOLO and the Darknet53 structure in it to detect the target and extract the feature value of the target. Define the image taken by the camera as the original image. At the beginning of target tracking, pass the first frame of the original image taken by the camera into the yolov3 network, output the target detection result of yolov3 and the 32×32×512 degree feature map Fm1 obtained by the 19th residual structure in the Darknet53 feature extraction network, and save the types and location information of each detected target. Step 2: Map and extract the feature values ​​of the target window. After passing the first frame of the original image into the yolov3 network, all the target windows w1, w2, L, w are detected. n , which is mapped to the dimensional feature map obtained by the Darknet53 feature extraction network The feature matrix of all target windows is obtained by first taking the feature matrix of the first target window w1 Extract it and perform a 2-time upsampling operation. The specific mapping relationship is described by formula (1): (x,y)=(S*x′,S*y′) (1) The specific feature extraction is described by equations (2) and (3). Window left, top: Right and bottom of the window: Where S is the product of all strides before and including the 19th residual structure in the Darknet53 feature extraction network, (x, y) is the coordinate of a point on the original image of the first frame, x is the horizontal coordinate value of the original image of the first frame, y is the vertical coordinate value of the original image of the first frame, and x′ is the dimensional feature map The corresponding horizontal axis value on the left is y′, which is the dimension feature map. The corresponding vertical coordinate value on the top; Step 3: Use the MMS structure to transform the feature matrix of uncertain size, length and width into a feature matrix of the same size, taking into account the mapping of the detection target window w1 to the dimensional feature map The feature matrix obtained on The length and width are not necessarily equal. If the feature matrix If we perform a truncation operation to obtain a feature matrix with equal length and width, some information will be lost, making the similarity comparison inaccurate. The designed MMS (Mesh Maximum Selection) structure is passed in to transform the input feature matrix Fm2 of unfixed size into a feature matrix of the same size. To implement the subsequent similarity comparison operation, the dimension of the feature matrix Fm2 before being input into MMS is a*b*512, which specifically includes the following sub-steps: (a) The first layer of MMS, for the feature matrix obtained above Find the maximum value in each depth dimension to obtain a 1×512 dimension matrix L1; (b) The second layer of MMS, the feature matrix Divide into four blocks of 2×2, where the specific division is described by formula (4): Where n is the current MMS layer number, a is the width of the feature matrix Fm2 dimension before entering the MMS, and b is the height of the feature matrix Fm2 dimension before entering the MMS. is the width of the block in row i and column j in n*n blocks. The height value of the block in the i-th row and j-th column of the n*n blocks is obtained. The maximum value of each divided block at each depth is obtained, and finally a 4×512 dimension matrix L2 is obtained; (c) The third layer of MMS, the feature matrix Fm2 is divided into nine blocks of 3×3, where the specific division is described by equation (4). The maximum value of each divided block at each depth is calculated, and finally a 9×512 dimensional matrix L3 is obtained; (d) The dimension matrices L1, L2, and L3 obtained by the 1st, 2nd, and 3rd layers of MMS are concatenated to achieve the a*b*512 feature matrix that is never fixed. Transformed to a 1×14×512 feature matrix Step 4: Repeat steps 2 and 3 to obtain the 1×14×512 feature matrix for each target window Save it; Step 5: Perform steps 1, 2, 3, and 4 on the second original image captured by the camera to obtain a 1×14×512 feature matrix for each target window. Step 6: Compare the similarity of the target windows in the first and second frames of the original image to determine whether they are the same target and implement target tracking, which specifically includes the following sub-steps: (a) The 1×14×512 feature matrix obtained by matching the first window of the first frame original image with the first target of the same category detected in the second frame original image Expanded into a 7168×1 dimension feature vector (b) The two-dimensional feature vector Subtracting and then summing the absolute values ​​is equivalent to obtaining the two-dimensional feature vector The L1 norm of the interpolation is equivalent to obtaining the two-dimensional feature vector The distance D w , described by formula (5), Where D w Represents a two-dimensional feature vector distance; (c) D w Perform three full connections, and the last full connection is connected to a neuron. Take the sigmoid of the result of this neuron to get the similarity s, so that its value is between 0 and 1, representing the similarity between the two input images. Select the threshold t according to the actual project situation. According to the similarity, it can be judged whether it is the same object. The specific judgment of whether it is the same object is described by formula (6): (d) If they are the same object, the same ID is given to the bounding boxes obtained by target detection corresponding to the windows participating in the operation in the first and second frames of the original image to achieve target tracking. If they are not the same object, the first window in the first frame of the original image and the next target window of the same category detected in the next frame are subjected to the operations (a), (b), and (c) in the above step 6 to determine whether they are the same object, until all windows of the same category as the first window in the first frame of the original image are traversed; (e) After the traversal is completed, steps 2, 3, 4, 5, and 6 are repeated for the next window in the first frame of the original image; (f) After the comparison between the first frame original image and the second frame original image is completed, the next frame original image is compared with the second frame original image. The comparison method is the same as steps 1 to 6 above. And so on, the comparison of subsequent frames is performed to achieve target tracking.