Target tracking processing method and device, equipment and medium

Abstract

The invention provides a target tracking processing method and device, equipment and a medium, and the processing method comprises the steps: carrying out the preprocessing of a first-frame image and a current image, and generating a corresponding first-frame initial image and a current initial image; performing feature extraction processing on the first-frame initial image and the current initial image through a weight adaptive fusion encoder to generate input feature data; performing deep fusion processing on the first-frame initial image and the current initial image; carrying out feature extraction processing through a weight adaptive fusion encoder to generate depth feature data; performing attention weight fusion processing on the input feature data and the depth feature data through a weight adaptive fusion encoder to generate target feature tensor data; and processing the target feature tensor data through a full convolutional neural network to generate a target image. Through the target tracking processing method and device, the equipment and the medium provided by the invention, the target tracking effect can be improved.

Description

Technical Field

[0001] The present invention relates to the field of robots, and in particular to a processing method, device, equipment and medium for target tracking. Background Art

[0002] With the continuous development of robotics technology, the demand for robots in various dynamic and complex environments is gradually increasing. Especially in the fields of autonomous driving, industrial automation, intelligent monitoring, etc., the robot's environmental perception ability is crucial. The robot's visual perception system obtains environmental information through image sensors and uses computer vision technology to detect, locate and track targets.

[0003] However, traditional visual perception systems face some key problems. In particular, traditional image processing methods often perform poorly in complex backgrounds, dynamic scale changes, low light, and occlusion, which affects the robot's positioning and decision-making capabilities. Complex backgrounds make it difficult for traditional image processing methods to effectively separate targets and backgrounds, resulting in false detections and missed detections. Therefore, there is room for improvement. Summary of the invention

[0004] The purpose of the present invention is to provide a target tracking processing method, device, equipment and medium, which can improve the target tracking effect.

[0005] In order to solve the above technical problems, the present invention is achieved through the following technical solutions:

[0006] The present invention provides a target tracking processing method, comprising:

[0007] Preprocessing the first frame image and the current image respectively to extract the target area in the image, and generating the corresponding first frame initial image and current initial image;

[0008] Performing feature extraction processing on the first frame initial image and the current initial image through a weight adaptive fusion encoder to generate input feature data;

[0009] Performing a deep fusion process on the first frame initial image and the current initial image; and performing a feature extraction process on the processed image through the weight adaptive fusion encoder to generate deep feature data;

[0010] Performing attention weight fusion processing on the input feature data and the deep feature data through the weight adaptive fusion encoder to generate target feature tensor data;

[0011] The target feature tensor data is processed by a fully convolutional neural network to generate a target image.

[0012] In one embodiment of the present invention, the step of performing deep fusion processing on the first frame initial image and the current initial image includes:

[0013] Performing depth estimation processing on the first frame initial image and the current initial image respectively to generate a corresponding first frame depth image and a current depth image;

[0014] The first-frame depth image and the current depth image are respectively normalized according to the central depth value of the previous frame image to generate a corresponding first-frame depth enhancement image and a current depth enhancement image.

[0015] In one embodiment of the present invention, the step of performing attention weight fusion processing on the input feature data and the deep feature data through the weight adaptive fusion encoder to generate target feature tensor data includes:

[0016] Performing layer normalization processing on the input feature data and the depth feature data respectively through the initial normalization layer of the weight adaptive fusion encoder to generate corresponding input normalized data and depth normalized data;

[0017] The input normalized data and the depth normalized data are processed respectively by the linear layer of the weight adaptive fusion encoder to calculate the corresponding input attention parameters and the depth attention parameters;

[0018] The input attention parameter and the deep attention parameter are processed respectively by the softmax layer of the weight adaptive fusion encoder to generate corresponding input attention weights and deep attention weights;

[0019] Calculate the inner product of the input attention weight and the median vector matrix of the input attention parameter to generate input summary data; calculate the inner product of the depth attention weight and the median vector matrix of the depth attention parameter to generate depth summary data;

[0020] The input attention weight and the deep attention weight are processed by the attention weight fusion layer of the weight adaptive fusion encoder to generate a corresponding hybrid attention weight value;

[0021] The mixed attention weight value, the input summary data, and the input feature data are processed by the feature-weight fusion layer of the weight adaptive fusion encoder to generate corresponding input merged feature data; the mixed attention weight value, the depth summary data, and the depth feature data are processed by the feature-weight fusion layer to generate corresponding depth merged feature data;

[0022] The input merged feature data and the deep merged feature data are processed respectively by the output normalization layer of the weight adaptive fusion encoder, and the processing results are respectively input into the fully connected layer of the weight adaptive fusion encoder for processing to generate corresponding input feature tensor data and deep feature tensor data;

[0023] The weight adaptive fusion encoder performs multiple attention weight fusion processes on the input feature tensor data and the deep feature tensor data to generate target feature tensor data.

[0024] In one embodiment of the present invention, the mixed attention weight value W AMF , expressed as: W AMF =A F +A D , where A F represents the input attention weight, A D represents the deep attention weight.

[0025] In one embodiment of the present invention, the input merged feature data B F , expressed as: B F =F+(A′ F ☉W AMF ), where F represents the input feature data, A' F represents the input summary data, ⊙ represents element-by-element multiplication; the deep merge feature data B D , expressed as: B D =D+(A′ D ☉W AMF) , where D represents the deep feature data, A' D Represented as deeply aggregated data.

[0026] In one embodiment of the present invention, the step of performing multiple attention weight fusion processes on the input feature tensor data and the deep feature tensor data through the weight adaptive fusion encoder to generate target feature tensor data includes:

[0027] The weight adaptive fusion encoder performs multiple attention weight fusion processes on the input feature tensor data and the deep feature tensor data, and obtains the target feature tensor data based on the final output input feature tensor data.

[0028] In one embodiment of the present invention, the step of processing the target feature tensor data by a fully convolutional neural network to generate a target image includes:

[0029] Processing the target feature tensor data through a fully convolutional neural network to generate a corresponding target classification score map, a normalized size parameter, and a local offset matrix;

[0030] Determining an initial position of a target by locating a peak in the target classification score map;

[0031] Correcting the initial position according to the local offset matrix to determine the final position of the target;

[0032] The set limit frame is scaled according to the normalized size parameter, and the final position is encapsulated by the scaled limit frame to generate a target image.

[0033] The present invention also provides a target tracking processing device, comprising:

[0034] A preprocessing module is used to preprocess the first frame image and the current image respectively to extract the target area in the image and generate the corresponding first frame initial image and current initial image;

[0035] A first extraction module is used to perform feature extraction processing on the first frame initial image and the current initial image to generate input feature data;

[0036] A second extraction module is used to perform a depth fusion process on the first frame initial image and the current initial image; and perform a feature extraction process on the processed image to generate depth feature data;

[0037] A deep fusion module, used to perform attention weight fusion processing on the input feature data and the deep feature data to generate target feature tensor data; and

[0038] The image output module is used to process the target feature tensor data to generate a target image.

[0039] The present invention also provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, the steps of the target tracking processing method are implemented.

[0040] The present invention also provides a computer-readable storage medium, which stores a computer program, and is characterized in that when the computer program is executed by a processor, the steps of the target tracking processing method are implemented.

[0041] As described above, the present invention provides a method, device, equipment and medium for processing target tracking, which significantly improves the target tracking accuracy and robustness of robots in complex scenarios through enhanced processing of depth information and adaptive fusion of multimodal information, combined with attention mechanism. This method can not only effectively deal with common problems such as dynamic scale changes and fuzzy targets, but also maintain high target recognition and tracking stability in complex environments. Its broad application prospects, especially in the fields of autonomous driving, industrial automation and intelligent monitoring, provide important support for future technological development.

[0042] Of course, any product implementing the present invention does not necessarily need to achieve all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings required for describing the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other accompanying drawings can be obtained based on these accompanying drawings without paying creative work.

[0044] Figure 1 A flowchart of a target tracking processing method in one embodiment of the present invention;

[0045] Figure 2 A schematic diagram of a weight adaptive fusion encoder according to an embodiment of the present invention;

[0046] Figure 3 A schematic diagram of a target tracking processing device according to an embodiment of the present invention;

[0047] Figure 4 FIG. 4 is a schematic diagram of an electronic device according to an embodiment of the present invention.

[0048] In the figure: 100, preprocessing module; 200, first extraction module; 300, second extraction module; 400, depth fusion module; 500, image output module; 1, electronic device; 12, memory; 13, processor. DETAILED DESCRIPTION

[0049] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0050] See also Figure 1The present invention provides a target tracking processing method, which can be applied to a robot to enable the robot to accurately track the target in a complex scene. The processing method may include the following steps:

[0051] Step S10, preprocessing the first frame image and the current image respectively to extract the target area in the image, and generating the corresponding first frame initial image and current initial image;

[0052] Step S20, performing feature extraction processing on the first frame initial image and the current initial image through a weight adaptive fusion encoder to generate input feature data;

[0053] Step S30, performing deep fusion processing on the first frame initial image and the current initial image; and performing feature extraction processing on the processed image through a weight adaptive fusion encoder to generate deep feature data;

[0054] Step S40, performing attention weight fusion processing on the input feature data and the deep feature data through a weight adaptive fusion encoder to generate target feature tensor data;

[0055] Step S50: Process the target feature tensor data through a fully convolutional neural network to generate a target image.

[0056] In one embodiment, when executing step S10, specifically, the preprocessing steps may include cropping, padding, and resizing. The first frame image refers to the first frame image in the video sequence, which may include the initial state and the real bounding box of the target. The current image refers to the frame image being processed in the video sequence. The previous frame image refers to the previous frame image relative to the current image, which is used to provide the target position.

[0057] In one embodiment, for the first frame, the target area can be cropped according to the groundtruth of the target. For the current image, the target area can be cropped according to the target position of the previous frame. This is because the target may move in the video sequence, and the search results of the previous frame can provide an approximate target position, so as to perform more accurate cropping. The sizes of the cropped images may not be the same, for example, they may be (192, 192, 3) and (384, 384, 3) respectively.

[0058] In one embodiment, in order to prevent the target from becoming too small during the cropping process and affecting the accuracy of subsequent processing, padding processing can be performed around the cropped image, such as adding a solid color area (such as selecting white, black or other background colors). This can ensure that the input image has enough information in the network. Specifically, if the size of the cropped image is smaller than the preset size, a solid color will be filled in the missing part of the image to make it meet the preset size requirement to maintain the consistency of the image and enable the network to process fixed-size input.

[0059] In one embodiment, since deep learning models usually require input images of a fixed size, resizing can be performed to ensure that the final size of the first frame initial image and the current initial image are preset sizes to ensure processing consistency and calculation effectiveness in the network.

[0060] In one embodiment, by processing the first frame image and the current image, the stability and robustness of target tracking can be ensured, and the impact of cumulative errors and short-term changes can be reduced. Although using the previous frame and the current frame for tracking also has certain advantages, for the sake of overall efficiency and adaptability to specific tasks, in this embodiment, the previous frame image and the current image are not used for processing.

[0061] Specifically, the purpose of target tracking is to continuously track a specific target in a video sequence. In order to ensure the stability of tracking, it is necessary to solve problems such as target appearance change and tracking drift. Among them, target appearance change means that in a video sequence, the target may undergo various changes, such as deformation, lighting change, occlusion, etc. These changes may cause the appearance of the target to differ greatly between different frames. Tracking drift means that if tracking is only based on the previous frame and the current frame, cumulative errors are likely to occur, causing the tracked target to gradually deviate from the actual position. This phenomenon is called tracking drift.

[0062] Furthermore, the first frame image contains the initial state of the target and the most accurate bounding box (groundtruth). In the first frame image, the appearance information of the target is the most reliable and clear. Through the first frame image, the benchmark features of the target can be obtained, which is very important for subsequent tracking. Using the first frame image as a reference point can avoid misjudgments caused by short-term changes. Even if the target has undergone large changes in the frames before the current frame, the features in the first frame image can still be used as a stable reference. By processing the first frame image and the current image, the problem of changes in target appearance and tracking drift can be better dealt with. The first frame image provides a baseline for the initial state, while the current image provides the latest target information. Combining the two can improve the robustness of tracking.

[0063] Furthermore, the target position and feature information in the previous frame have already been processed by a tracking process, and there may be certain errors. If tracking is performed only on the previous and current frames, these errors will gradually accumulate, resulting in reduced tracking accuracy. The target in the previous frame may be affected by short-term changes (such as rapid movement, momentary occlusion), which may interfere with the tracking results of the current frame. The previous frame image is easily affected by the processing results of the previous frame as a reference point, resulting in an unstable reference point for the entire tracking process, further increasing the risk of tracking drift.

[0064] Furthermore, in one embodiment, although using the previous frame and the current frame for tracking can compensate for the scale change of the target in some cases, the robustness and stability of this method are relatively weak. Therefore, in this embodiment, the first frame image and the current image are selected for processing to ensure the robustness of tracking and the adaptability to specific tasks. In addition, there are some more complex multi-frame tracking methods, such as combining three or more frames for consideration, and even using extreme cases such as seven twins. These methods can provide higher accuracy in specific scenarios, but usually increase computational complexity and processing time.

[0065] In one embodiment, the first frame initial image and the current initial image acquired need to be processed by two different network flows in the future, where the network flows refer to different processing paths. The first network flow refers to a path that does not require deep fusion, and the second network flow refers to a path that requires deep fusion.

[0066] In one embodiment, when executing step S20, specifically, the first frame initial image and the current initial image can be processed through the first network flow. The first frame initial image and the current initial image are processed by two different Transformer encoders to generate connected features (features extracted by ViT (Vision Transformer)), that is, the first frame initial image and the current initial image are respectively input into two different encoders. The features generated by the above two encoders will then be connected together to form an overall feature, which is represented as input feature data F.

[0067] In one embodiment, the above-mentioned encoder can be a weight adaptive fusion encoder, which is represented by an encoder that fuses multimodal or multimodal information based on attention weights, and its purpose is to integrate multimodal or multimodal information using attention weights. Attention weights refer to weighting the importance of different information, so that the model can better focus on important parts and improve the effect of information fusion. Among them, multimodal information refers to data from different modes (such as images from different cameras, or images of the same camera at different times).

[0068] In one embodiment, when step S30 is executed, specifically, step S30 may include the following steps:

[0069] Step S31, performing depth estimation processing on the first frame initial image and the current initial image respectively to generate a corresponding first frame depth image and a current depth image;

[0070] Step S32: normalize the first-frame depth image and the current depth image according to the center depth value of the previous frame image to generate a corresponding first-frame depth enhanced image and a current depth enhanced image;

[0071] Step S33: perform feature extraction processing on the first-frame depth-enhanced image and the current depth-enhanced image to generate depth feature data.

[0072] In one embodiment, when executing step S31, specifically, the first frame initial image and the current initial image can be processed through the second network flow. First, the first frame initial image and the current initial image can be processed for depth estimation respectively through the MiDaS (Monocular Depth Awareness for Single Image) method to generate corresponding first frame depth image and current depth image. Among them, the MiDaS method is a monocular depth estimation method that can infer the depth information of the scene from a single image.

[0073] In one embodiment, when executing step S32, specifically, the center depth value of the previous frame image is used for normalization processing, and the center depth value (Dc) of the previous frame image can be obtained from the depth image of the previous frame, wherein the previous frame image refers to the previous frame relative to the current image, for example, the current image is the 141st frame image, and the previous frame image refers to the 140th frame image. The first frame depth image D s (x, y) and the current depth image D t (x, y) is normalized to generate the corresponding first frame depth enhancement image I′ s (x, y) and the current depth enhanced image I′ t (x,y). s (x,y) represents the pixel value of the first frame depth image. t (x,y) represents the pixel value of the current depth image. (x,y) represents the position in the image.

[0074] In one embodiment, the first frame depth enhanced image I′ s (x,y) can be expressed as: I′ s (x,y)=D′ s (x,y)☉I s(x,y), ⊙ represents element-by-element multiplication. The current depth-enhanced image can be expressed as: I′ t (x,y)=D′ t (x,y)☉I t (x,y), Among them, for the pixel depth value of each position in the first frame depth image and the current depth image, if the value is less than Dc, it is proportionally reduced; if the value is greater than or equal to Dc, it is proportionally increased. By setting this way, a depth contrast effect can be created around the target, making the depth information of the target area more obvious.

[0075] In one embodiment, when executing step S33, specifically, the first frame depth enhancement image and the current depth enhancement image are processed by two different weight adaptive fusion encoders to generate connected features (features extracted by ViT (Vision Transformer)), that is, the first frame depth enhancement image and the current depth enhancement image are respectively input into two different encoders. The features generated by the above two encoders will then be connected together to form an overall feature, represented as depth feature data D.

[0076] In one embodiment, when step S40 is executed, specifically, step S40 may include the following steps:

[0077] Step S41, processing the input feature data and the depth feature data respectively through the initial normalization layer of the weight adaptive fusion encoder to generate corresponding input normalized data and depth normalized data;

[0078] Step S42: Processing the input normalized data and the depth normalized data respectively through the linear layer of the weight adaptive fusion encoder to calculate the corresponding input attention parameters and depth attention parameters;

[0079] Step S43, processing the input attention parameter and the deep attention parameter respectively through the softmax layer of the weight adaptive fusion encoder to generate corresponding input attention weights and deep attention weights;

[0080] Step S44, calculating the inner product of the input attention weight and the input attention parameter median vector matrix to generate input summary data; calculating the inner product of the depth attention weight and the depth attention parameter median vector matrix to generate depth summary data;

[0081] Step S45, processing the input attention weight and the deep attention weight through the attention weight fusion layer of the weight adaptive fusion encoder to generate a corresponding hybrid attention weight value;

[0082] Step S46, processing the mixed attention weight value, the input summary data, and the input feature data through the feature-weight fusion layer of the weight adaptive fusion encoder to generate corresponding input merged feature data; processing the mixed attention weight value, the depth summary data, and the depth feature data through the feature-weight fusion layer to generate corresponding depth merged feature data;

[0083] Step S47, processing the input merged feature data and the deep merged feature data respectively through the output normalization layer of the weight adaptive fusion encoder, and inputting the processing results into the fully connected layer of the weight adaptive fusion encoder for processing, to generate corresponding input feature tensor data and deep feature tensor data;

[0084] Step S48: Perform multiple attention weight fusion processes on the input feature tensor data and the deep feature tensor data through the weight adaptive fusion encoder to generate target feature tensor data.

[0085] See also Figure 2 In one embodiment, when executing step S41, specifically, layer normalization (LayerNorm, LN) refers to a normalization technique used in deep learning models to stabilize and accelerate the training process of the model by normalizing the input of each neural network layer.

[0086] See also Figure 2 In one embodiment, the input feature data F and the depth feature data D can be respectively subjected to layer normalization processing by the initial normalization layer (LN) of the weight adaptive fusion encoder to generate the corresponding input normalized data F norm and depth normalized data D norm Among them, F norm =LayerNorm(F),D norm = LayerNorm(D). LayerNorm() represents the layer normalization operation, which is used to normalize the feature dimension of each sample. The initial normalization layer refers to Figure 2 The LN layer close to the input feature data (Feature, F) and the depth feature data (Depth Feature, D).

[0087] See also Figure 2In one embodiment, when executing step S42, specifically, in the Transformer architecture, the linear layer is usually used to map the input data from one feature space to another feature space. The output of the linear layer can be regarded as a linear transformation of the input data. In the Transformer, the attention mechanism is used to weight different parts of the input data, so as to focus on important information. In this embodiment, the input normalized data F can be calculated by the linear layer respectively. norm and depth normalized data D norm The attention parameters include the query vector, key vector, and value vector. The attention parameters determine the association and importance between different input parts.

[0088] See also Figure 2 In one embodiment, the input normalized data F norm The attention parameter can be expressed as the input attention parameter, and the input attention parameter is expressed as Q F ,K F ,V F ,Q F ,K F ,V F =Transform(F norm ). Depth normalized data D norm The attention parameter can be expressed as the deep attention parameter Q D ,K D ,V D ,Q D ,K D ,V D =Transform(D norm ). Transform() represents the operation in the Transformer linear layer, which is used to calculate the attention parameters. Figure 2 Linear(Lin) in.

[0089] See also Figure 2 In one embodiment, when executing step S43, specifically, for the calculation process of the input attention weight, the query vector Q in the input attention parameter can be calculated first. F with the transpose of the key vector The inner product of the matrix, then the softmax function can be operated on the inner product result to obtain the input attention weight A F , Here, · represents the inner product of the matrix, and T represents the matrix transpose operation. The function of the softmax function is to convert the attention score matrix into a probability distribution matrix, where each element represents the attention weight of a certain input feature to other features. Similarly, the deep attention weight A can be calculated D , Q D and K D Respectively represent the query vector and key vector in the deep attention parameters. Among them, the softmax layer refers to Figure 2 Softmax in .

[0090] See also Figure 2 In one embodiment, when executing step S44, specifically, the input summary data A′ is generated by calculating the inner product of the input attention weight and the input attention parameter median vector matrix. F , expressed as: A′ F =A F ·V F , where V F Represented as a vector of values ​​in the input attention parameters. The deep summary data A′ is generated by calculating the inner product of the deep attention weights and the matrix of value vectors in the deep attention parameters. D , A′ D =A D ·V D , where V D Represented as a value vector in the deep attention parameter. The input summary data and the deep summary data can be intermediate results, which represent the weighted summary information of each feature after the attention mechanism is applied. For example, the input summary data reflects which parts of the input features are focused on, and the deep summary data reflects which parts of the deep features are focused on.

[0091] See also Figure 2 In one embodiment, when executing step S45, specifically, the input attention weight matrix and the deep attention weight matrix are connected through the attention weight fusion layer of the weight adaptive fusion encoder to generate a mixed attention weight value matrix. The connection operation refers to connecting the input attention weight matrix and the deep attention weight matrix in a certain dimension (for example, the feature dimension). By connecting the input attention weights and the deep attention weights, and calculating the adaptive mixed attention weights, dynamic fusion of multimodal feature information can be achieved. At the same time, by directly calculating the sum of the input and deep attention weights, a simplified mixed attention weight value can be generated for subsequent feature weighting operations. Among them, the mixed attention weight value W AMF Expressed as: W AMF =A F +A DAmong them, the attention weight fusion layer refers to Figure 2 Element-wise Additon(+) connected to the softmax layer.

[0092] See also Figure 2 In one embodiment, when executing step S46, specifically, the input summary data A′ is input through the feature-weight fusion layer of the weight adaptive fusion encoder. F and the mixed attention weight W AMF Perform element-by-element multiplication and calculate the sum of the processing result and the input feature data F to generate the input merged feature data B F , B F =F+(A′ F ☉W AMF ). The depth summary data A′ is processed by the feature-weight fusion layer of the weight adaptive fusion encoder. D and the mixed attention weight W AMF Perform element-by-element multiplication and calculate the sum of the processing result and the depth feature data D to generate the depth merged feature data B D , B D =D+(A′ D ☉W AMF ). Through element-by-element multiplication, the mixed attention weight value can weight each feature of the input summary data and the deep summary data differently, so that the model pays more attention to important features and ignores unimportant features. Adding the weighted summary data to the original feature data can enhance the focused part of the original feature data while retaining the information of the original feature data. Among them, the feature-weight fusion layer refers to Figure 2 Element-wise Multiplication (×) and Element-wise Addition (+) in .

[0093] See also Figure 2 In one embodiment, when executing step S47, specifically, the input merged feature data B can be respectively normalized by the output normalization layer of the weight adaptive fusion encoder. F Merge feature data B with depth D The layer normalization is performed, and the processing results are respectively input into the fully connected layer of the weight adaptive fusion encoder for processing to generate the output of the current layer of the encoder module based on the attention weight fusion multimodal or multimodal information, which is represented as the input feature tensor data F feature and deep feature tensor data D depth_feature , F feature =FC(LayerNorm(B F )), D depth_feature=FC(LayerNorm(B D )). Among them, the output normalization layer refers to Figure 2 The LN layer close to the feature-weight fusion layer in the figure, and the fully connected layer refers to Figure 2 Fully Conneeted Layer (FC) in.

[0094] In one embodiment, when executing step S48, specifically, the input feature tensor data F can be fused by the weight adaptive encoder. feature and deep feature tensor data D depth_feature Attention weight fusion processing is performed, and the target feature tensor data is obtained according to the input feature tensor data outputted finally, and the target depth tensor data is obtained according to the depth feature tensor data outputted finally. For example, the input feature tensor data and the depth feature tensor data can be used as the input of the subsequent layer in the encoder module sequence based on the attention weight fusion multimodal or multimodal information. After multiple (for example, 12) encoder fusion processing based on the attention weight fusion multimodal or multimodal information, the features that do not pass through the deep fusion path are finally used, that is, the target feature tensor data is obtained according to the input feature tensor data outputted finally. Through the attention mechanism (multimodal fusion) of each layer, the correlation between the input features and the depth features can be dynamically learned, so as to more effectively fuse the multimodal information. The normalization and conversion operations of each layer can enhance the expressive power of the features, reduce the risk of overfitting, and improve the generalization ability of the model. Since the features (i.e., the input feature tensor data) that do not pass through the deep fusion path are finally selected as the final output, more original input feature information can be retained, and multimodal information is used to assist and improve the accuracy of the model. Therefore, the final target depth tensor data can be ignored.

[0095] In one embodiment, when step S50 is executed, specifically, step S50 may include the following steps:

[0096] Step S51, processing the target feature tensor data through a fully convolutional neural network to generate a corresponding target classification score map, normalized size parameters, and a local offset matrix;

[0097] Step S52, determining the initial position of the target by locating the peak in the target classification score map;

[0098] Step S53, correcting the initial position according to the local offset matrix to determine the final position of the target;

[0099] Step S54: scaling the set limit frame according to the normalized size parameter, and encapsulating the final position with the scaled limit frame to generate a target image.

[0100] In one embodiment, when step S51 is executed, specifically, a fully convolutional neural network (FCN) is a network structure that replaces the fully connected layer of a traditional convolutional neural network (CNN) with a convolutional layer, so that it can process input images of any size and output feature maps of the same or similar size as the input image. The fully convolutional neural network can output a target classification score map, a normalized size parameter, and a local offset matrix. Among them, the target classification score map C is a two-dimensional matrix, and each element value Cmn is in the range of [0,1], indicating the probability that the position (m,n) is the target. The normalized size parameter Σ indicates the degree of scaling, which is in the range of [0,1]. The local offset matrix Δ represents the offset value of the target position relative to the center, Among them, Hs and Ws represent the height and width of the current initial image respectively.

[0101] In one embodiment, when executing step S52, specifically, by locating the peak in the target classification score map, the initial position of the target can be determined. The position of the initial position can be expressed as (x * ,y * ), (x * ,y * ) = argmax (m,n) Cmn, argmax represents the maximization operation.

[0102] In one embodiment, when executing step S53 and step S54, specifically, the local offset matrix is ​​a two-dimensional matrix, and each position has two values, which represent the offset of the target center relative to the center of the grid unit. The initial position is corrected according to the local offset matrix to reduce the spatial quantization error. The bounding box is a pre-set candidate box used to locate and scale the target. The bounding box can be expressed as (x, y, w, h), where x represents the horizontal coordinate of the upper left corner of the bounding box, y represents the vertical coordinate of the upper left corner of the bounding box, w represents the width of the bounding box, and h represents the height of the bounding box. Subsequently, the bounding box can be scaled using a normalized size parameter. Finally, the scaled bounding box can be used to encapsulate the final position to generate a target image (b x ,b y ,b w ,b h ), expressed as: (b x ,b y ,b w ,b h )=(x * +Δx,y * +Δy,Σw,Σh), where Δx and Δy refer to the offsets, and Σw and Σh refer to the deviation distances of the width and height of the target image relative to the scaled bounding box.

[0103] In one embodiment, during the training phase of the model, by combining the focal loss Lclass, L1 loss, and generalized IOU loss, the composite loss function Ltotal can comprehensively optimize the classification ability and bounding box regression ability of the model. Among them, the L1 loss can be used to determine the center position and size of the regression bounding box to ensure that the predicted bounding box is as close as possible to the true bounding box in terms of value. The generalized IOU loss can be used to optimize the degree of overlap of the bounding box, while considering the size of the bounding rectangle, so that the predicted bounding box is not only accurate in position, but also closer to the true bounding box in shape.

[0104] It can be seen that in the above scheme, the robot's target tracking accuracy and robustness in complex scenes are significantly improved through enhanced processing of depth information and adaptive fusion of multimodal information, combined with the attention mechanism. This method can not only effectively deal with common problems such as dynamic scale changes and blurred targets, but also maintain a high target recognition and tracking stability in complex environments. Its broad application prospects, especially in the fields of autonomous driving, industrial automation and intelligent monitoring, provide important support for future technological development. Specifically, through enhanced processing of depth information, the model can better understand the three-dimensional structure and spatial position of the target, thereby improving the accuracy of target tracking. Using the attention mechanism, the model can dynamically focus on the salient features of the target, reduce background interference and false detection, and further improve the accuracy of target tracking. Through the fusion of multimodal information, the model can use other sensor information (such as depth information) to assist in recognition and tracking when the target image is blurred or partially occluded, thereby improving robustness. Through the combination of depth information and attention mechanism, the model's environmental perception ability has been significantly improved, and it can more accurately understand the surrounding environment and improve the stability of target tracking. In the field of autonomous driving, this method can provide more accurate and stable target tracking, help vehicles better identify and track targets such as pedestrians and other vehicles, and improve driving safety. In industrial automation, this method can improve the accuracy of robot target detection and tracking in dynamic environments, and is suitable for tasks such as object recognition and grasping on production lines. In the field of intelligent monitoring, this method can improve the accuracy of monitoring systems, better identify and track targets in complex scenarios, and reduce false positives and false negatives.

[0105] It should be understood that the order of execution of the steps in the above embodiment does not necessarily mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present invention.

[0106] See also Figure 3The present invention also provides a processing device for target tracking, which can be applied to the above processing method. The processing device can include a preprocessing module 100, a first extraction module 200, a second extraction module 300, a depth fusion module 400, and an image output module 500.

[0107] In one embodiment, the preprocessing module 100 may be used to preprocess the first frame image and the current image respectively to extract the target area in the image and generate the corresponding first frame initial image and the current initial image.

[0108] In one embodiment, the first extraction module 200 may be used to perform feature extraction processing on the first frame initial image and the current initial image to generate input feature data.

[0109] In one embodiment, the second extraction module 300 may be used to perform a depth fusion process on the first frame initial image and the current initial image; and perform a feature extraction process on the processed image to generate depth feature data.

[0110] In one embodiment, the deep fusion module 400 may be used to perform attention weight fusion processing on the input feature data and the deep feature data to generate target feature tensor data.

[0111] In one embodiment, the image output module 500 may be used to perform attention weight fusion processing on the input feature data and the depth feature data to generate target feature tensor data.

[0112] See also Figure 4 In one embodiment, the electronic device 1 may include a memory 12, a processor 13 and a bus, and may also include a computer program stored in the memory 12 and executable on the processor 13, such as a target tracking processing program.

[0113] In one embodiment, the memory 12 includes at least one type of readable storage medium, and the readable storage medium includes a flash memory, a mobile hard disk, a multimedia card, a card-type memory (for example, SD or DX memory, etc.), a magnetic memory, a disk, an optical disk, etc. In some embodiments, the memory 12 may be an internal storage unit of the electronic device 1, such as a mobile hard disk of the electronic device 1. In other embodiments, the memory 12 may also be an external storage device of the electronic device 1, such as a plug-in mobile hard disk, a smart memory card (Smart Media Card, SMC), a secure digital (Secure Digital, SD) card, a flash card (Flash Card), etc. equipped on the electronic device 1. Further, the memory 12 may also include both an internal storage unit of the electronic device 1 and an external storage device. The memory 12 can not only be used to store application software and various types of data installed in the electronic device 1, such as target tracking processing, but also be used to temporarily store data that has been output or is to be output.

[0114] In one embodiment, the processor 13 may be composed of an integrated circuit, for example, a single packaged integrated circuit, or a plurality of packaged integrated circuits with the same or different functions, including one or more central processing units (CPUs), microprocessors, digital processing chips, graphics processors, and combinations of various control chips. The processor 13 is the control core (ControlUnit) of the electronic device 1, and uses various interfaces and lines to connect various components of the entire electronic device 1, and executes or executes programs or modules stored in the memory 12, and calls data stored in the memory 12, so as to execute various functions of the electronic device 1 and process data.

[0115] In one embodiment, the processor 13 executes the operation module of the electronic device 1 and various installed applications. The processor 13 executes the application to implement the steps in the above-mentioned target tracking processing method.

[0116] In one embodiment, the computer program may be divided into one or more modules, one or more modules are stored in the memory 12, and executed by the processor 13 to complete the present application. One or more modules may be a series of computer program instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program in the electronic device 1.

[0117] The embodiments of the present invention disclosed above are only used to help illustrate the present invention. The embodiments do not describe all the details in detail, nor do they limit the invention to the specific embodiments described. Obviously, many modifications and changes can be made according to the content of this specification. This specification selects and specifically describes these embodiments in order to better explain the principles and practical applications of the present invention, so that those skilled in the art can understand and use the present invention well. The present invention is limited only by the claims and their full scope and equivalents.

Claims

1. A target tracking processing method, characterized in that: include: Preprocessing the first frame image and the current image respectively to extract the target area in the image, and generating the corresponding first frame initial image and current initial image; Performing feature extraction processing on the first frame initial image and the current initial image through a weight adaptive fusion encoder to generate input feature data; Performing a deep fusion process on the first frame initial image and the current initial image; and performing a feature extraction process on the processed image through the weight adaptive fusion encoder to generate deep feature data; Performing attention weight fusion processing on the input feature data and the deep feature data through the weight adaptive fusion encoder to generate target feature tensor data; The target feature tensor data is processed by a fully convolutional neural network to generate a target image.

2. The target tracking processing method according to claim 1, characterized in that: The step of performing deep fusion processing on the first frame initial image and the current initial image comprises: Performing depth estimation processing on the first frame initial image and the current initial image respectively to generate a corresponding first frame depth image and a current depth image; The first-frame depth image and the current depth image are respectively normalized according to the central depth value of the previous frame image to generate a corresponding first-frame depth enhancement image and a current depth enhancement image.

3. The target tracking processing method according to claim 1, characterized in that: The step of performing attention weight fusion processing on the input feature data and the deep feature data by the weight adaptive fusion encoder to generate target feature tensor data comprises: The input feature data and the depth feature data are processed respectively by the initial normalization layer of the weight adaptive fusion encoder to generate corresponding input normalized data and depth normalized data; The input normalized data and the depth normalized data are processed respectively by the linear layer of the weight adaptive fusion encoder to calculate the corresponding input attention parameters and the depth attention parameters; The input attention parameter and the deep attention parameter are processed respectively by the softmax layer of the weight adaptive fusion encoder to generate corresponding input attention weights and deep attention weights; Calculate the inner product of the input attention weight and the median vector matrix of the input attention parameter to generate input summary data; calculate the inner product of the depth attention weight and the median vector matrix of the depth attention parameter to generate depth summary data; The input attention weight and the deep attention weight are processed by the attention weight fusion layer of the weight adaptive fusion encoder to generate a corresponding hybrid attention weight value; The mixed attention weight value, the input summary data, and the input feature data are processed by the feature-weight fusion layer of the weight adaptive fusion encoder to generate corresponding input merged feature data; the mixed attention weight value, the depth summary data, and the depth feature data are processed by the feature-weight fusion layer to generate corresponding depth merged feature data; The input merged feature data and the deep merged feature data are processed respectively by the output normalization layer of the weight adaptive fusion encoder, and the processing results are respectively input into the fully connected layer of the weight adaptive fusion encoder for processing to generate corresponding input feature tensor data and deep feature tensor data; The weight adaptive fusion encoder performs multiple attention weight fusion processes on the input feature tensor data and the deep feature tensor data to generate target feature tensor data.

4. The target tracking processing method according to claim 3, characterized in that: The mixed attention weight value W AMF , expressed as: W AMF =A F +A D , where A F represents the input attention weight, A D represents the deep attention weight.

5. The target tracking processing method according to claim 4, characterized in that: The input merged feature data B F , expressed as: B F =F+(A′ F ⊙W AMF ), where F represents the input feature data, A' F represents the input summary data, ⊙ represents element-by-element multiplication; the deep merge feature data B D , expressed as: B D =D+(A′ D ⊙W AMF ), where D represents the deep feature data, A' D Represented as deeply aggregated data.

6. The target tracking processing method according to claim 3, characterized in that: The step of performing multiple attention weight fusion processes on the input feature tensor data and the deep feature tensor data through the weight adaptive fusion encoder to generate target feature tensor data includes: The weight adaptive fusion encoder performs multiple attention weight fusion processes on the input feature tensor data and the deep feature tensor data, and obtains the target feature tensor data according to the final output input feature tensor data.

7. The target tracking processing method according to claim 1, characterized in that: The step of processing the target feature tensor data by a fully convolutional neural network to generate a target image comprises: Processing the target feature tensor data through a fully convolutional neural network to generate a corresponding target classification score map, a normalized size parameter, and a local offset matrix; Determining an initial position of a target by locating a peak in the target classification score map; Correcting the initial position according to the local offset matrix to determine the final position of the target; The set limit frame is scaled according to the normalized size parameter, and the final position is encapsulated by the scaled limit frame to generate a target image.

8. A target tracking processing device, characterized in that: include: A preprocessing module is used to preprocess the first frame image and the current image respectively to extract the target area in the image and generate the corresponding first frame initial image and current initial image; A first extraction module is used to perform feature extraction processing on the first frame initial image and the current initial image to generate input feature data; A second extraction module is used to perform a depth fusion process on the first frame initial image and the current initial image; and perform a feature extraction process on the processed image to generate depth feature data; A deep fusion module, used to perform attention weight fusion processing on the input feature data and the deep feature data to generate target feature tensor data; as well as The image output module is used to process the target feature tensor data to generate a target image.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the steps of the target tracking processing method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the steps of the target tracking processing method according to any one of claims 1 to 7 are implemented.

Images