A quadruped robot target tracking method based on a twin network

Abstract

The application aims at the single target tracking field and discloses a quadruped robot target tracking method based on a twin network. First, data preprocessing is performed on a video, then an image alignment module is constructed, preliminary positioning is performed before feature extraction of the image, so that the target is always in the search area, and the number of times that the tracker loses the target is reduced. Then, a feature extraction subnetwork is used for feature extraction, and the extracted multiple convolution features are hierarchically aggregated, then a scale adaptive subnetwork is used to adapt to the scale change of the target in the tracking process, and a regression subnetwork updated online is used to adapt to the change of light and target shape. Finally, the position of the target in the current frame is obtained.

Description

Technical Field

[0001] This invention relates to the field of single-target tracking in computer vision, and particularly to a target tracking method for quadruped robots based on Siamese networks. Background Technology

[0002] Object tracking is a crucial research area in computer vision. It has wide applications in many fields, including intelligent video surveillance, automated monitoring, human-computer interaction, and robot vision navigation. Given the position and size of a target in an initial frame, the goal of object tracking is to estimate the position and size of the target in subsequent frames. Despite significant progress in recent years, designing a highly robust tracker remains extremely challenging due to numerous influencing factors during object tracking, such as changes in lighting, occlusion, background clutter, and rapid camera movement.

[0003] In recent years, quadruped robot technology has developed rapidly. Compared with wheeled robots, quadruped robots have the ability to move flexibly in complex terrain environments and have a broader development prospect. Therefore, in-depth research on quadruped robots and improving their intelligence level is of great significance. In many scenarios, robots that can travel on complex terrains that ordinary wheeled robots cannot traverse are needed. Because quadruped robots need to move in various complex environments, such as stairs and slopes, camera shake occurs when the robot moves on bumpy and rugged surfaces, which brings unique challenges and difficulties to target tracking for quadruped robots. Compared to target tracking in ordinary videos, the camera's range of motion is very large. When walking on gravel or ice, the camera's motion becomes more intense and accompanied by shaking, so the position of the object in adjacent frames will change significantly. At the same time, quadruped robots not only walk indoors but also outdoors, which causes drastic changes in lighting during target tracking, requiring the tracking algorithm to have high robustness.

[0004] Existing correlation filtering-based tracking methods, while boasting high tracking speeds, fail to fully and effectively utilize target features, leading to frequent tracking failures. Meanwhile, existing Siamese network-based methods undergo offline training on large datasets, maintaining a fixed model throughout the tracking process by inputting the search region into the network. However, they lack online update modules, making them highly susceptible to the impact of target shape and appearance changes on the tracker. Summary of the Invention

[0005] Purpose of the invention: This invention addresses the problems faced by target tracking from the perspective of quadruped robots by proposing a quadruped robot target tracking method based on Siamese networks. This method solves the problems existing in the target tracking of quadruped robots from the perspective of existing technologies, improves tracking accuracy, and reduces tracking failures caused by drastic changes in the camera.

[0006] Technical Solution: The quadruped robot target tracking method based on Siamese networks proposed in this invention employs a multi-path architecture to improve tracking accuracy and robustness. To address the camera shake issue caused by quadruped robot movement, an image alignment module is proposed to perform preliminary localization before feature extraction, ensuring the target remains within the search area and reducing the number of times the tracker loses the target. To address the issue of scale changes during target tracking, a scale adaptation module is proposed to adapt to changes in target scale during tracking, while an online-updated regression subnetwork adapts to changes in lighting and target shape.

[0007] A target tracking method for quadruped robots based on Siamese networks, the specific steps of which are as follows:

[0008] Step 1: Data Preprocessing. First, the tracked video needs to be processed into consecutive video frames, and the data types of these frames need to be converted. This improves the efficiency of subsequent processes. Video frame processing includes: reading the images and converting them to RGB format; converting the data type to Tensor.

[0009] Step 2: Construct the image alignment module. The specific process is as follows: Given a preprocessed initial frame and the current image frame to be tracked, as well as the location of the target in the initial frame. Before feature extraction in the image input network, the preprocessed initial frame and the current frame are aligned and registered. First, the SIFT method is used to find the feature points of the two images. Then, the ORB feature matching algorithm is used to register the feature points of the two images. Based on the registration points, the transformation matrix H of the two images is calculated. The calculation method adopts the random sample consistency estimation method. The transformation matrix H is shown below:

[0010]

[0011] Where H is the transformation matrix, h 11 …h 33 represents the value of each element in the transformation matrix.

[0012] The target positions in the two images are aligned using a calculated transformation matrix. This means that the approximate position of the target in the current frame is estimated based on its position in the previous frame. The target's position in the previous frame is represented as follows:

[0013]

[0014] Among them, P l x represents the position coordinates of the target in the previous frame. l y l These are coordinate values.

[0015] The estimated position of the target in the current frame is represented as:

[0016]

[0017] Among them, P n x represents the estimated position coordinates of the target in the current frame. n y n These are coordinate values.

[0018] The estimated position coordinates of the target in the current frame can be calculated using the target position and transformation matrix from the previous frame, as follows:

[0019]

[0020] Where, x n y n x represents the estimated coordinates of the target in the current frame. l y l Here are the coordinates of the target in the previous frame. H is the transformation matrix, h 11 …h 33 represents the value of each element in the transformation matrix.

[0021] Step 3: Crop the initial frame using the target's position in the first frame to obtain a template image. The estimated target position P calculated in Step 2 is then used. n Crop the current image to obtain the search image.

[0022] First, calculate the side lengths of the template graph and the search graph that need to be cut. Define the side length of the template graph as sz and the side length of the search graph as sx. The formula for calculating the side length of the template graph is as follows:

[0023]

[0024] Where w and h are the width and length of the target, respectively, α is the preset scaling factor, and sz is the side length of the square of the clipping template image in the initial frame.

[0025] The process for calculating the side length of the search graph is as follows:

[0026] pad = (ST) / (sz*β)

[0027] sx = sz + 2 * pad

[0028] Where S is the side length of the preset search image, T is the side length of the preset template image, sz is the side length of the square from which the template image is cut from the image, pad is the size of the fill area of ​​the search image relative to the template image at the image edges, and β is the preset template scaling factor. Then, the side lengths of the search image, sx and sz can be calculated.

[0029] After obtaining the side length sz of the square cut from the template image and the side length sx of the square cut from the search image, the estimated target position P obtained in step 1 is used in the current frame. n The search image is obtained by cropping around the target center, and a template image is also obtained by cropping around the target center in the first frame. If the square area exceeds the boundary of the original image, it is filled with the average pixel value. Then, the cropped template image is scaled to 127×127, and the search image is scaled to 255×255.

[0030] Step 4: Construct as follows Figure 1 The deep Siamese network model shown is as follows. The template image and search image obtained in step 3 are fed into the trained feature extraction subnetwork. The feature extraction subnetwork extracts features from the template image and search image obtained in step 3 and then concatenates and aggregates them. Then, a scale adaptation subnetwork generates multiple candidate tracking boxes. Finally, a regression subnetwork generates a Gaussian response map based on the features from the feature extraction subnetwork, and the final tracking result is selected based on the Gaussian response map.

[0031] This paper uses ResNet-50 as the feature extraction sub-network, and concatenates and aggregates the features of the template image and search image through the conv3_3, conv4_6, and conv5_3 convolutional layers of the feature extraction network. The feature extraction sub-network is trained offline. For ease of understanding, the convolutional layer parameters described in this paper are expressed as follows: number of input channels_number of output channels_kernel size × kernel size_stride.

[0032] The output features of the three convolutional layers are scaled and concatenated. A (512_256_1×1_1) convolutional layer, a (1024_256_1×1_1) convolutional layer, and a (2048_256_1×1_1) convolutional layer are used to scale the channels of the output features of the conv3_3, conv4_6, and conv5_3 convolutional layers respectively, scaling the number of channels of the output features of the three convolutional layers to 256.

[0033] The scaled features are then concatenated and aggregated as follows:

[0034]

[0035]

[0036] Where Z is the concatenated template feature, X is the concatenated search region feature, and α l ,β l The weights for concatenation and aggregation are given by Z, where l is the index of the convolutional layer and Z is the weight of the convolutional layer. lThe template features output from the three convolutional layers have been scaled to a size of 256×7×7, X l The search region features are output by the three convolutional layers, with a size of 256×31×31, where l∈{3,4,5}.

[0037] Step 5: Construct a scale-adaptive subnetwork. Feed the concatenated features into the scale-adaptive subnetwork for prediction, such as... Figure 2 As shown, the scale-adaptive subnetwork includes a feature adjustment layer, a deep correlation layer, and a supervision layer. The deep correlation layer is used to obtain a similarity response map between template features and search region features. First, the template features are transformed into a 256×1×5×5 feature map, and then correlation operations are performed between it and the search region features to obtain the similarity response map. The supervision layer has two branches: a category regression layer for foreground and background classification and a regression layer for the length, width, and position of the target tracking box. The scale-adaptive subnetwork is trained offline.

[0038] Setting k prediction anchor points, the network needs to output k channels to predict the probability of the tracking box and 4*k channels to regress the size of the target box. First, the scale-adaptive subnetwork uses four non-shared feature adjustment layers to increase the number of channels for the template feature Z and the search region X, encoding the difference between the template feature and the search region feature to make the features more suitable for the target tracking task. The encoded features become two branches, and after deep correlation operations, the resulting response maps are used for classification and regression, respectively, to obtain the template feature [Z] after increasing the number of channels in the classification branch. a ] cls and search region features [X a ] cls And the template features [Z] after adding channels to the regression branch. a ] reg and search region features [X a ] reg The feature adjustment layer is a "convolution-normalization-activation function" block, and the parameters of the convolution kernel are (256_256_3×3_1).

[0039] Then, deep correlation operations are performed on both the classification and regression branches simultaneously. The deep correlation operation is shown below:

[0040]

[0041]

[0042] Where w and h are the width and height of the target, respectively. [Z a ] cls and [X] a ] reg To add channels to the template features, [Xa ] cls and [X] a ] reg To add features to the search region after adding channels, * indicates a related operation. This represents the probability of the predicted bounding box at each anchor point. This represents the regression result output by the region generation network, containing k sets of predicted values. Each set contains four channel vectors: dx, dy, dw, and dz. Here, dx is the difference between the anchor point's x-coordinate and the target's actual x-coordinate, dy is the difference between the anchor point's y-coordinate and the target's actual y-coordinate, dw is the difference between the predicted bounding box's width and the target's actual width, and dh is the difference between the predicted bounding box's height and the target's actual height.

[0043] The size of the candidate boxes predicted by the network can be calculated using the following formula:

[0044] x r =x m +w m *dx

[0045] y r =y m +h m *dy

[0046] w r =w m *e aw

[0047] h r =h m *e ah

[0048] Where, x r y r w r h r These are the coordinates and dimensions of the final candidate bounding box, respectively. m y m w m h m dx, dy, dw, and dz are the coordinates, length, and width of each anchor point, respectively, and dx, dy, dw, and dz are the offset values ​​between the anchor points predicted by the network and the actual positions of the targets.

[0049] Step 6: Construct the regression subnetwork. After the feature extraction subnetwork in the deep Siamese network, the regression subnetwork utilizes the initial frames... Figure 1The output features of the conv5_3 convolutional layer in the feature extraction subnetwork are used for training, outputting a two-dimensional Gaussian response map centered on the target location. The regression subnetwork module includes a convolutional layer with kernel parameters of (256_1_1×1_1) and a convolutional layer with kernel size of (1_1_w×h_1). The first convolutional layer is used to reduce the dimensionality of the depth features and reduce the consumption of computational resources. The second convolutional layer is used to generate the Gaussian response map to locate the possible location of the target. The size of the convolutional kernel is the same as the size of the tracked target, where w and h are the width and height of the target, respectively.

[0050] After obtaining the Gaussian response map, the top 10 predicted boxes with the highest probabilities from the candidate tracking boxes obtained in step 5 are selected. The overlap ratio between the predicted boxes and the region in the Gaussian response map is calculated, and the box with the highest overlap ratio is the final predicted value. The formula for calculating the overlap ratio is as follows:

[0051] R = mR0 + nR p

[0052] Where R0 is the overlap ratio between the predicted bounding box output by the scale-adaptive subnetwork and the Gaussian response map, R p This is the probability score of the predicted bounding box obtained by the scale-adaptive subnetwork, where m and n are preset coefficients. Finally, the predicted bounding box with the largest R value is selected as the final tracking result.

[0053] Compared with the prior art, the advantages and beneficial effects of the present invention are as follows: The present invention combines an image alignment module and a deep twin network module, which effectively solves the problem that the tracking algorithm of a quadruped robot is prone to losing the target due to camera shaking caused by motion during target tracking. Attached Figure Description

[0054] Figure 1 This is a flowchart of the tracking method.

[0055] Figure 2 This is a flowchart of the alignment module in this tracking method.

[0056] Figure 3 This is a flowchart of the scale-adaptive subnetwork in this tracking method. Detailed Implementation

[0057] This invention first registers and aligns the current frame with the first frame, then crops the image around the aligned target. The cropped image is then fed into a feature extraction network for feature extraction. The extracted template features and search region features are then fed into a scale-adaptive sub-network, and finally, a regression sub-network selects the results to obtain the final prediction.

[0058] The technical solution of the present invention will be described in detail below with reference to the accompanying drawings and examples.

[0059] Step 1: Data Preprocessing. First, the tracked video needs to be processed into consecutive video frames, and the data types of these frames need to be converted. This improves the efficiency of subsequent processes. Video frame processing includes: reading the images and converting them to RGB format; converting the data type to Tensor.

[0060] Step 2: Constructing an image alignment module. This invention aims to address the problem of camera shake during quadruped robot movement, which leads to drastic changes in target position. The specific process is as follows: Given a preprocessed initial frame and the current image frame to be tracked, along with the target's position in the initial frame. Before feature extraction in the image input network, the preprocessed initial frame and the current frame are aligned and registered. First, the SIFT method is used to find feature points in the two images. Then, the ORB feature matching algorithm is used to register the feature points. Based on the registration points, the transformation matrix H between the two images is calculated using the random sample consistency estimation method. The transformation matrix H is shown below:

[0061]

[0062] Where H is the transformation matrix, h 11 …h 33 represents the value of each element in the transformation matrix.

[0063] The target positions in the two images are aligned using a calculated transformation matrix. This means that the approximate position of the target in the current frame is estimated based on its position in the previous frame. The target's position in the previous frame is represented as follows:

[0064]

[0065] Among them, P l x represents the position coordinates of the target in the previous frame. l y l These are coordinate values.

[0066] The estimated position of the target in the current frame is represented as:

[0067]

[0068] Among them, P n x represents the estimated position coordinates of the target in the current frame. n y n These are coordinate values.

[0069] The estimated position coordinates of the target in the current frame can be calculated using the target position and transformation matrix from the previous frame, as follows:

[0070]

[0071] Where, x n y n x represents the estimated coordinates of the target in the current frame. l y l Here are the coordinates of the target in the previous frame. H is the transformation matrix, h 11 …h 33 represents the value of each element in the transformation matrix.

[0072] Step 3: Crop the initial frame using the target's position in the first frame to obtain a template image. The estimated target position P calculated in Step 2 is then used. n Crop the current image to obtain the search image.

[0073] First, calculate the side lengths of the template graph and the search graph to be cut. Define the side length of the template graph as sz and the side length of the search graph as sx. The formula for calculating the side length of the template graph is as follows:

[0074]

[0075] Where w and h are the width and length of the target, respectively, α is the preset scaling factor, and sz is the side length of the square from which the template image is cut from the image.

[0076] The process for calculating the side length of the search graph is as follows:

[0077] pad = (ST) / (sz*β)

[0078] sx = sz + 2 * pad

[0079] Where S is the side length of the preset search image, set to 300 in this example, and T is the side length of the preset template image, set to 150 in this example. sz is the side length of the square from which the template image is cut from the image, pad is the size of the fill area of ​​the search image relative to the template image at the image edges, and β is the preset template scaling factor, set to 0.8 in this example. Then, the side lengths of the search image and template images can be calculated: sx is the side length of the square from which the search image is cut from the image, and sz is the side length of the square from which the template image is cut from the image.

[0080] After obtaining the side length sz of the square cut from the template image and the side length sx of the square cut from the search image, the estimated target position P obtained in step 1 is used in the current frame. n The search image is obtained by cropping around the target center, and a template image is also obtained by cropping around the target center in the first frame. If the square area exceeds the boundary of the original image, it is filled with the average pixel value. Then, the cropped template image is scaled to 127×127, and the search image is scaled to 255×255.

[0081] Step 4: Construct as follows Figure 1 The deep Siamese network model shown is as follows. A feature extraction sub-network extracts features from the template image and search image obtained in step 3, and then concatenates and aggregates them. Next, a scale adaptation sub-network generates multiple candidate tracking boxes. Finally, a regression sub-network generates a Gaussian response map based on the features from the feature extraction sub-network, and the final tracking result is selected based on the Gaussian response map.

[0082] This paper uses ResNet-50 as the feature extraction sub-network, and concatenates and aggregates the features of the template image and search image through the conv3_3, conv4_6, and conv5_3 convolutional layers of the feature extraction network. The feature extraction sub-network is trained offline. For ease of understanding, the convolutional layer parameters described in this paper are expressed as follows: number of input channels_number of output channels_kernel size × kernel size_stride.

[0083] The output features of the three convolutional layers are scaled and concatenated. A (512_256_1×1_1) convolutional layer, a (1024_256_1×1_1) convolutional layer, and a (2048_256_1×1_1) convolutional layer are used to scale the channels of the output features of the conv3_3, conv4_6, and conv5_3 convolutional layers respectively, scaling the number of channels of the output features of the three convolutional layers to 256.

[0084] The scaled features are then concatenated and aggregated as follows:

[0085]

[0086]

[0087] Where Z is the concatenated template feature, X is the concatenated search region feature, and α l ,β l The weights for concatenation and aggregation are given by Z, where l is the index of the convolutional layer and Z is the weight of the convolutional layer. l The template features output from the three convolutional layers have been scaled to a size of 256×7×7, X l The search region features are output by the three convolutional layers, with a size of 256×31×31, where l∈{3,4,5}.

[0088] Step 5: Construct a scale-adaptive subnetwork. Feed the concatenated features into the scale-adaptive subnetwork for prediction, such as... Figure 2As shown, the scale-adaptive subnetwork includes a feature adjustment layer, a deep correlation layer, and a supervision layer. The deep correlation layer obtains a similarity response map between template features and search region features. First, the template features are transformed into a 256×1×5×5 feature map, and then correlation operations are performed between it and the search region features to obtain the similarity response map. The supervision layer has two branches: a category regression layer for foreground and background classification and a regression layer for the length, width, and position of the target tracking bounding box. The scale-adaptive subnetwork is trained offline.

[0089] In this example, the number of anchor points k is set to 400. The network needs to output 400 channels to predict the probability of the tracking box and 4*400 channels to regress the size of the target box. First, the scale-adaptive subnetwork uses four feature adjustment layers with non-shared parameters to increase the number of channels for the template feature Z and the search region x, encoding the difference between the template feature and the search region feature to make the features more suitable for the target tracking task. The encoded features become two branches. After deep correlation operation, the resulting response maps are used for classification and regression, respectively, to obtain the template feature [Z] after increasing the number of channels in the classification branch. a ] cls and search region features [X a ] cls And the template features [Z] after adding channels to the regression branch. a ] reg and search region features [X a ] reg The feature adjustment layer is a "convolution-normalization-activation function" block, and the parameters of the convolution kernel are (256_256_3×3_1).

[0090] Then, deep correlation operations are performed on both the classification and regression branches simultaneously. The deep correlation operation is shown below:

[0091]

[0092]

[0093] Where w and h are the width and height of the target, respectively. [Z a ] cls and [X] a ] reg To add channels to the template features, [X a ] cls and [x a ] reg To add features to the search region after adding channels, * indicates a related operation. This represents the probability of the predicted bounding box at each anchor point. This represents the regression result output by the region generation network, containing k sets of predicted values. Each set contains four channel vectors: dx, dy, dw, and dz. Here, dx is the difference between the anchor point's x-coordinate and the target's actual x-coordinate, dy is the difference between the anchor point's y-coordinate and the target's actual y-coordinate, dw is the difference between the predicted bounding box's width and the target's actual width, and dh is the difference between the predicted bounding box's height and the target's actual height.

[0094] The size of the candidate boxes predicted by the network can be calculated using the following formula:

[0095] x r =x m +w m *dx

[0096] y r =y m +h m *dy

[0097] w r =w m *e dw

[0098] h r =h m *e dh

[0099] Where, x r y r w r h r These are the coordinates and dimensions of the final candidate bounding box, respectively. m y m w m h m dx, dy, dw, and dz are the coordinates, length, and width of each anchor point, respectively, and dx, dy, dw, and dz are the offset values ​​between the anchor points predicted by the network and the actual positions of the targets.

[0100] Step 6: Construct the regression sub-network. After the feature extraction network in the deep Siamese network, the regression sub-network utilizes the initial frames... Figure 1 The output features of the conv5_3 convolutional layer in the feature extraction subnetwork are used for training, outputting a two-dimensional Gaussian response map centered on the target location. In this example, the regression network parameters are updated every 10 frames. The regression subnetwork module includes a convolutional layer with a kernel size of (256_1_1×1_1) and a convolutional layer with a kernel size of (1_1_w×h_1). The first convolutional layer is used to reduce the dimensionality of the depth features and reduce the consumption of computational resources. The second convolutional layer is used to generate the Gaussian response map to locate the possible location of the target. The size of the convolutional kernel is the same as the size of the tracked target, where w and h are the width and height of the target, respectively.

[0101] After obtaining the Gaussian response map, the top 10 predicted boxes with the highest probabilities from the candidate tracking boxes obtained in step 5 are selected. The overlap ratio between the predicted boxes and the region in the Gaussian response map is calculated, and the box with the highest overlap ratio is the final predicted value. The formula for calculating the overlap ratio is as follows:

[0102] R = mR0 + nR p

[0103] Where R0 is the overlap ratio between the predicted bounding box output by the scale-adaptive subnetwork and the Gaussian response map, R p This is the probability score of the predicted bounding box obtained by the scale-adaptive subnetwork, where m and n are preset coefficients. Finally, the predicted bounding box with the largest R value is selected as the final tracking result.

[0104] Repeat the above steps to complete the tracking of the target in consecutive video frames.

[0105] The specific embodiments described herein are merely illustrative of the spirit of the invention. Those skilled in the art to which this invention pertains may make various modifications or additions to the described specific embodiments or use similar methods to replace them, without departing from the spirit of the invention or exceeding the scope defined by the appended claims.

Claims

1. A target tracking method for a quadruped robot based on Siamese networks, characterized in that, Includes the following steps: Step 1, Data Preprocessing Step 2: Construct an image alignment module. Before inputting the image into the feature extraction sub-network, obtain the estimated position of the target in the current frame. First, align and register the two images, the preprocessed initial frame and the current frame. Then, using the calculated transformation matrix, map the position in the initial frame to the current frame to obtain the estimated position of the target in the current frame. Then, perform the subsequent feature extraction process. Step 3: Using the estimated position of the target in the current frame obtained in Step 2, crop the image. First, crop the initial frame using the position of the target in the first frame to obtain a template image with the target position as the center and a square shape. Then, crop the current frame using the estimated position of the target calculated in Step 2 to obtain a search image with the estimated position of the target as the center and a square shape. Step 4: Construct a feature extraction sub-network. The template image and search image obtained in Step 3 are fed into the trained feature extraction sub-network. The template image and search image are processed through the conv3_3, conv4_6 and conv5_3 convolutional layers of the feature extraction sub-network, and the output features are concatenated and aggregated. The aggregated features are used for scale prediction in the subsequent scale adaptation sub-network. ResNet-50 is used as the feature extraction sub-network, which is trained offline. Step 5: Construct a scale adaptation sub-network. The concatenated features are fed into the scale adaptation sub-network for prediction. The scale adaptation sub-network includes a feature adjustment layer, a depth correlation layer and a supervision layer. The depth correlation layer is used to obtain the similarity response map between the template features and the search region features. The supervision layer has two branches: one for foreground and background classification, and the other for regression of the length, width and position of the target tracking box. Finally, multiple candidate tracking boxes are output. Step 6: Construct a regression sub-network. After the feature extraction network in the deep Siamese network, the regression sub-network is trained using the output features of the conv5_3 convolutional layer in the feature extraction sub-network of the initial frame. It outputs a two-dimensional Gaussian response map centered on the target location. The output features of the current frame after passing through the conv5_3 convolutional layer in the feature extraction sub-network are input into the regression sub-network to obtain the Gaussian response map of the current search region. Then, the multiple candidate tracking boxes obtained in step 5 are selected to obtain the final tracking result.

2. The quadruped robot target tracking method based on Siamese networks as described in claim 1, characterized in that: The specific method for step 1 is as follows: First, the tracked video needs to be processed into continuous video frames. The data type of the video frames to be processed needs to be converted. The processing of video frames includes: reading the image and converting the image into RGB format; converting the data type into Tensor type.

3. The quadruped robot target tracking method based on Siamese networks as described in claim 1, characterized in that: Step 2 is explained in the following steps: Given a preprocessed initial frame and the current image frame to be tracked, as well as the target's position in the initial frame, the two images are aligned and registered before feature extraction by the image input network. First, the SIFT method is used to find feature points in the two images. Then, the ORB feature matching algorithm is used to register the feature points. Finally, the transformation matrix between the two images is calculated based on the registered points. The calculation method employs an estimation method for the consistency of random samples, and the transformation matrix... As shown below: in For the transformation matrix, … For each element value of the transformation matrix; The target positions in the two images are aligned using a calculated transformation matrix. This means that the approximate position of the target in the current frame is estimated based on its position in the previous frame. The target's position in the previous frame is represented as follows: in, These are the position coordinates of the target in the previous frame. , These are coordinate values; The estimated position of the target in the current frame is represented as: in, The estimated position coordinates of the target in the current frame. , These are coordinate values; The estimated position coordinates of the target in the current frame can be calculated using the target position and transformation matrix from the previous frame, as follows: in, , These are the estimated coordinates of the target in the current frame. , These are the coordinates of the target in the previous frame; For the transformation matrix, … represents the value of each element in the transformation matrix.

4. The quadruped robot target tracking method based on Siamese networks as described in claim 1, characterized in that: The specific method for step 3 is as follows: First, calculate the side lengths of the template image and the search image that need to be cut. Define the side length of the template image as... The side length of the search graph is The formula for calculating the side length of the template image is as follows: in, These are the width and length of the target, respectively. The preset scaling factor, The side length of the square in the clipping template image in the initial frame; The process for calculating the side length of the search graph is as follows: in, The side length of the preset search graph. The side length of the preset template image, To cut the side length of a square from a template image, The size of the fill around the edges of the image relative to the template image. The template scaling factor is preset, and then the side length of the search graph is calculated. To cut out the side length of the search square from the image, To cut the side length of a square from a template image; Get the side length of the square from the template image cut out from the image. and the side length of the square cut from the image to search the image. Then, in the current frame, a search image is obtained by cropping with the estimated target position obtained in step 2 as the center. At the same time, a template image is obtained by cropping with the target center in the first frame. If the square area exceeds the boundary of the original image, it is filled with the average pixel value. Then, the cropped template image and the search image are scaled.

5. The quadruped robot target tracking method based on Siamese networks as described in claim 1, characterized in that: Step 4 is explained in detail below: The output features of the three convolutional layers are scaled and concatenated. The output features of the conv3_3, conv4_6 and conv5_3 convolutional layers are scaled by three convolutional layers respectively, and the number of channels of the output features of the three convolutional layers is scaled to 256. The scaled features are then concatenated and aggregated as follows: in, It is a template feature that has been spliced ​​together. It is the feature of the search region after being stitched together. To concatenate the aggregated weights, The index of the convolutional layer. The template features output from the three convolutional layers have been scaled down to a size of 256×7×7. The search region features are output from three convolutional layers, with a size of 256×31×31. .

6. The quadruped robot target tracking method based on Siamese networks as described in claim 1, characterized in that: Step 5 is explained in detail below: set up For each predicted anchor point, the scale-adaptive subnetwork needs to output... Each channel is used to output the probability of the predicted tracking box. One channel is used to regress the size of the target box; first, the scale-adaptive subnetwork uses four non-shared feature adjustment layers to increase template features. and search area The number of channels is used to encode the differences between template features and search region features. The encoded features are divided into two branches. After deep correlation operation, the resulting response maps are used for classification and regression, respectively, to obtain the template features of the classification branch with added channels. and search region features And template features after adding channels to the regression branch. and search region features The feature adjustment layer is a convolution-normalization-activation function structure; Then, deep correlation operations are performed on both the classification and regression branches simultaneously. The deep correlation operation is shown below: in, h represents the width and height of the target, respectively. and To add channels to the template features, and To add features to the search area after adding channels, Indicates the relevant operation. This represents the probability of the predicted bounding box at each anchor point. This represents the regression result output by the region generation network, including... Group predicted values, each group contains Each of the following channel vectors is... ,in, anchor point Coordinates and actual target The difference in coordinates, anchor point Coordinates and actual target The difference in coordinates, The difference between the width of the predicted bounding box and the actual width of the target. This is the difference between the height of the predicted bounding box and the actual height of the target. The size of the candidate boxes predicted by the network can be calculated using the following formula: in, These are the coordinates and dimensions of the final candidate bounding box, respectively. These are the coordinates, length, and width of each anchor point.

7. The quadruped robot target tracking method based on Siamese networks as described in claim 1, characterized in that: Step 6 is explained in detail below: Following the feature extraction subnetwork, the regression subnetwork is trained using the output features of the initial frame after passing through the conv5_3 convolutional layer of the feature extraction network, outputting a two-dimensional Gaussian response map centered on the target location. The regression subnetwork module includes a convolutional layer with a kernel size of 256_1_1×1_1 and a convolutional layer with a kernel size of 1_1_w×h_1. The first convolutional layer is used to reduce the dimensionality of the depth features, reducing the consumption of computational resources, while the second convolutional layer is used to generate the Gaussian response map to locate the possible position of the target. The size of the convolutional kernel is the same as the size of the tracked target. , h are the width and height of the target, respectively; the convolutional layer parameters are expressed as: number of input channels_number of output channels_kernel size × kernel size_stride; After obtaining the Gaussian response map, select the top K predicted boxes with the highest probabilities from the candidate tracking boxes obtained in step 5. Calculate the overlap between the predicted boxes and the region in the Gaussian response map. The box with the highest overlap is the final predicted value. The formula for calculating the overlap is as follows: in, It is the overlap rate between the predicted bounding box output by the scale-adaptive subnetwork and the Gaussian response map. It is the probability score of the predicted bounding box obtained by the scale-adaptive subnetwork. Given preset coefficients, the one with the largest value is selected last. The predicted bounding box of the value is used as the final tracking result.

8. The quadruped robot target tracking method based on Siamese networks as described in claim 5, characterized in that: 512_256_1×1_1 convolutional layers, 1024_256_1×1_1 convolutional layers, and 2048_256_1×1_1 convolutional layers are used to scale the channels of the output features of the conv3_3, conv4_6, and conv5_3 convolutional layers, respectively; where the convolutional layer parameters are expressed as: number of input channels_number of output channels_kernel size×kernel size_stride.

9. The quadruped robot target tracking method based on Siamese networks as described in claim 4, characterized in that: The template image obtained by cutting is scaled to 127×127, and the search image is scaled to 255×255.

10. The quadruped robot target tracking method based on Siamese networks as described in claim 7, characterized in that: The value of K is 10.

Images