Target tracking method and apparatus

Abstract

The present disclosure provides a target tracking method and device, which can be applied to the field of computer vision. The method comprises: performing multi-channel feature extraction on a to-be-detected image of a current frame to obtain feature maps corresponding to a plurality of channels respectively, wherein the to-be-detected image comprises a detection target; performing target detection on the feature maps corresponding to the plurality of channels respectively to obtain target bounding box features of the detection target and target appearance features of the detection target; and determining a matching result of the detection target and a tracking trajectory based on the target bounding box features of the detection target, the target appearance features of the detection target and trajectory features of the tracking trajectory, to obtain trajectory bounding box features and trajectory appearance features.

Description

Technical Field

[0001] This disclosure relates to the field of computer vision, and more specifically to a target tracking method, apparatus, device, medium, and program product. Background Technology

[0002] Object tracking has always been a crucial problem in the field of computer vision. It is defined as the continuous tracking of each target of interest in a video sequence and the generation of its trajectory. The key is to accurately locate and track the target from consecutive image frames while maintaining consistency in the target's identity. Many related technologies achieve object tracking by extracting the target's bounding box information to establish a data association between the target and its trajectory.

[0003] In realizing the concept disclosed herein, the inventors discovered at least the following problems in the related technology: less information is extracted from the target during the tracking process, and the target detection and tracking are not accurate enough in scenarios where the camera is not fixed or in high-speed motion. Summary of the Invention

[0004] In view of the above problems, this disclosure provides a target tracking method, apparatus, device, medium and program product.

[0005] According to a first aspect of this disclosure, a target tracking method is provided, comprising: performing multi-channel feature extraction on a target image of the current frame to obtain feature maps corresponding to multiple channels respectively, wherein the target image includes a target; performing target detection on the feature maps corresponding to the multiple channels respectively to obtain target bounding box features and target appearance features of the target; determining a matching result between the target and the tracking trajectory based on the target bounding box features, the target appearance features, and the trajectory features of the tracking trajectory, wherein the tracking trajectory is determined based on historical target detections that have been continuously tracked in at least one historical frame, and the trajectory features include trajectory bounding box features determined by the historical bounding box features of the historical target detections and trajectory appearance features determined by the historical appearance features.

[0006] According to embodiments of this disclosure, target detection is performed on feature maps corresponding to multiple channels to obtain target bounding box features and target appearance features of the detected target. This includes: performing channel-unified processing on the feature maps corresponding to multiple channels to obtain multiple target feature maps with the same number of channels; inputting the multiple target feature maps into a first feature extraction model to obtain target bounding box features of the detected target; and inputting the multiple target feature maps into a second feature extraction model to obtain target appearance features of the detected target.

[0007] According to embodiments of this disclosure, inputting multiple target feature maps into a second feature extraction model to obtain target appearance features of the detected target includes: inputting multiple target feature maps into a first appearance extraction sub-model to obtain initial appearance features of the detected target; performing temporal information analysis on the initial appearance features of the detected target and the historical appearance features of at least one neighboring historical detected target to obtain target appearance features of the detected target, wherein the neighboring historical detected target is the historical detected target included in the image to be detected in the M nearest historical frames to the current frame, and M is an integer greater than or equal to 1.

[0008] According to embodiments of this disclosure, inputting multiple target feature maps into a second feature extraction model to obtain target appearance features of the detected target includes: inputting multiple target feature maps into a second appearance extraction sub-model to obtain target appearance features of the detected target, wherein the second appearance extraction sub-model includes a feature extraction layer and a mapping layer, and the feature extraction layer includes at least six convolutional sub-layers.

[0009] According to embodiments of this disclosure, the target bounding box features include: target bounding box confidence and target bounding box position; the trajectory bounding box features include: trajectory bounding box position and trajectory velocity and trajectory acceleration corresponding to the trajectory bounding box position; wherein, determining the matching result between the detected target and the tracking trajectory based on the target bounding box features of the detected target, the target appearance features of the detected target, and the trajectory features of the tracking trajectory includes: when the target bounding box confidence is greater than a preset confidence, calculating the distance between the target appearance features of the detected target and the trajectory appearance features to obtain the appearance distance; predicting the predicted bounding box position of the tracking trajectory in the current frame based on the trajectory bounding box position and the trajectory velocity and trajectory acceleration corresponding to the trajectory bounding box position; calculating the distance between the predicted bounding box position and the target bounding box position to obtain the bounding box distance; obtaining the matching cost matrix based on the appearance distance and the bounding box distance; and determining the matching result between the detected target and the tracking trajectory based on the matching cost matrix.

[0010] According to embodiments of this disclosure, the target bounding box features include: target bounding box confidence, target bounding box position, and target bounding box size information; the trajectory bounding box features include: trajectory bounding box position, trajectory velocity corresponding to the trajectory bounding box position, trajectory acceleration, and trajectory bounding box size information; wherein, determining the matching result between the detected target and the tracking trajectory based on the target bounding box features of the detected target, the target appearance features of the detected target, and the trajectory features of the tracking trajectory includes: when the target bounding box confidence is determined to be less than or equal to a preset confidence, predicting the predicted bounding box position and predicted bounding box size information of the tracking trajectory in the current frame based on the trajectory bounding box position, the trajectory velocity corresponding to the trajectory bounding box position, the trajectory acceleration, and the trajectory bounding box size information; determining a predicted sub-image based on the predicted bounding box position and predicted bounding box size information; determining the current sub-image based on the target bounding box position and target bounding box size information; determining the distance between the predicted bounding box features and the target bounding box features based on the pixel set included in the predicted sub-image and the pixel set included in the current sub-image; and determining the matching result between the detected target and the tracking trajectory based on the distance.

[0011] According to embodiments of this disclosure, the method further includes: when it is determined that the matching result represents a match between the detected target and the tracking trajectory, replacing the trajectory bounding box feature in the trajectory feature with the target bounding box feature; obtaining an updated appearance feature based on the target appearance feature, the trajectory appearance feature and a preset appearance weight value; and replacing the trajectory appearance feature in the trajectory feature with the updated appearance feature.

[0012] According to embodiments of this disclosure, the method further includes: when it is determined that the matching result indicates that the detected target and the tracking trajectory do not match and the confidence of the target bounding box of the detected target is greater than a preset confidence, determining the trajectory features of the new tracking trajectory based on the target appearance features and the target bounding box features; and using the current frame as the lost frame of the tracking trajectory so that the tracking trajectory is deleted when the lost frames of the tracking trajectory are consecutive frames and the number of lost frames is greater than a preset frame number threshold.

[0013] According to embodiments of this disclosure, the first feature extraction model and the second feature extraction model are trained in the following manner: multiple sample target feature maps of a sample image are input into an initial first feature extraction model to obtain sample bounding box features of the sample targets included in the sample image; a first loss value is obtained based on the sample bounding box features and the ground truth bounding box features; multiple sample target feature maps are input into an initial second feature extraction model to obtain sample appearance features of the sample targets; a second loss value is obtained based on the predicted label and the ground truth label of the sample appearance features; a target loss value is determined based on the first loss value, a first weight value corresponding to the first loss value, a second loss value, and a second weight value corresponding to the second loss value; the initial first feature extraction model and the initial second feature extraction model are trained based on the target loss value to obtain the first feature extraction model and the second feature extraction model.

[0014] A second aspect of this disclosure provides a target tracking apparatus, characterized in that the apparatus comprises: an extraction module for extracting multi-channel features from a current frame's image to be detected, obtaining feature maps corresponding to each of the multiple channels, wherein the image to be detected includes a target; a detection module for detecting a target by performing target detection on the feature maps corresponding to each of the multiple channels, obtaining target bounding box features and target appearance features of the detected target; and a matching module for determining a matching result between the detected target and the tracking trajectory based on the target bounding box features, the target appearance features, and the trajectory features of the tracking trajectory, wherein the tracking trajectory is determined based on historical detected targets included in at least one historical frame's image to be detected, and the trajectory features include target boundary features determined by the historical bounding box features of the historical detected targets and trajectory appearance features determined by the historical appearance features.

[0015] A third aspect of this disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more computer programs, wherein the one or more processors execute the one or more computer programs to implement the steps of the method described above.

[0016] A fourth aspect of this disclosure also provides a computer-readable storage medium having a computer program or instructions stored thereon, which, when executed by a processor, implement the steps of the above-described method.

[0017] The fifth aspect of this disclosure also provides a computer program product, including a computer program or instructions that, when executed by a processor, implement the steps of the above-described method.

[0018] According to the target tracking method disclosed herein, multi-channel feature extraction is employed to extract features from the image to be detected at multiple scales, obtaining feature maps corresponding to each channel. This achieves the extraction of both detailed and global features of the image to be detected. Furthermore, during target detection, not only are the target bounding box features extracted, but also the target appearance features. This allows for matching the tracking trajectory with the detected target based on information from both appearance and bounding box features during subsequent matching. Because target detection is performed using feature maps corresponding to multiple channels, and the matching result between the tracking trajectory and the detected target is determined based on both target bounding box and appearance features, this method at least partially solves the technical problem of insufficient accuracy in target detection and tracking in related technologies, thereby improving the accuracy of target tracking. Attached Figure Description

[0019] The foregoing contents, as well as other objects, features, and advantages of this disclosure, will become clearer from the following description of embodiments with reference to the accompanying drawings, in which:

[0020] Figure 1 The illustrations depict application scenarios of target tracking methods, apparatus, devices, media, and program products according to embodiments of the present disclosure.

[0021] Figure 2 A flowchart illustrating a target tracking method according to an embodiment of the present disclosure is shown schematically;

[0022] Figure 3 The diagram schematically illustrates a model structure for determining target bounding box features and target appearance features of a detected target according to an embodiment of the present disclosure;

[0023] Figure 4 The diagram schematically illustrates a model structure for determining the appearance features of a target to be detected, according to another embodiment of the present disclosure.

[0024] Figure 5 A flowchart illustrating a target tracking method according to another embodiment of the present disclosure is shown schematically;

[0025] Figure 6 A schematic block diagram of a target tracking device according to an embodiment of the present disclosure is shown; and

[0026] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a target tracking method according to an embodiment of the present disclosure. Detailed Implementation

[0027] The embodiments of the present disclosure will now be described with reference to the accompanying drawings. However, it should be understood that these descriptions are exemplary only and are not intended to limit the scope of the disclosure. In the following detailed description, numerous specific details are set forth to provide a thorough understanding of the embodiments of the present disclosure for ease of explanation. However, it will be apparent that one or more embodiments may be practiced without these specific details. Furthermore, descriptions of well-known structures and techniques are omitted in the following description to avoid unnecessarily obscuring the concepts of the present disclosure.

[0028] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit this disclosure. The terms “comprising,” “including,” etc., as used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0029] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art, unless otherwise defined. It should be noted that the terms used herein are to be interpreted in a manner consistent with the context of this specification, and not in an idealized or overly rigid way.

[0030] When using expressions such as "at least one of A, B and C", they should generally be interpreted in accordance with the meaning that is commonly understood by those skilled in the art (e.g., "a system having at least one of A, B and C" should include, but is not limited to, a system having A alone, a system having B alone, a system having C alone, a system having A and B, a system having A and C, a system having B and C, and / or a system having A, B and C, etc.).

[0031] It should be noted that the target tracking method and apparatus disclosed herein can be used in the field of computer vision, or in any field other than computer vision, such as the field of artificial intelligence. This disclosure does not limit the application field of the target tracking method and apparatus.

[0032] In the technical solution disclosed herein, the user information (including but not limited to user personal information, user image information, user device information, such as location information) and data (including but not limited to data used for analysis, stored data, and displayed data) involved are all information and data authorized by the user or fully authorized by all parties. Furthermore, the collection, storage, use, processing, transmission, provision, disclosure, and application of related data all comply with relevant laws, regulations, and standards, necessary confidentiality measures have been taken, they do not violate public order and good morals, and corresponding operation entry points are provided for users to choose to authorize or refuse.

[0033] In scenarios involving automated decision-making using personal information, the methods, devices, and systems provided in this disclosure all offer users corresponding entry points for choosing to agree to or reject the automated decision-making results. If the user chooses to reject, the process proceeds to the expert decision-making stage. Here, "automated decision-making" refers to the activity of automatically analyzing and evaluating an individual's behavioral habits, interests, or economic, health, and credit status through computer programs, and then making a decision. Here, "expert decision-making" refers to the activity of making decisions by personnel who specialize in a particular field, possess specialized experience, knowledge, and skills, and have reached a certain level of professional expertise.

[0034] The research revealed that current target tracking algorithms rely solely on motion information from the trajectory and the target's bounding box to correlate data between the target and the trajectory. In scenarios with moving camera angles and high-speed motion, these algorithms lack precision in target detection and tracking, which can negatively impact performance. Furthermore, extracting feature information from only a single frame without fully utilizing the temporal information of the video further contributes to poor tracking results.

[0035] Embodiments of this disclosure provide a target tracking method, comprising: extracting multi-channel features from a target image in the current frame to obtain feature maps corresponding to each of the multiple channels, wherein the target image includes a target; performing target detection on the feature maps corresponding to each of the multiple channels to obtain target bounding box features and target appearance features of the target; determining a matching result between the target and the tracking trajectory based on the target bounding box features, the target appearance features, and the trajectory features of the tracking trajectory, wherein the tracking trajectory is determined based on historical targets included in at least one historical frame of the target image, and the trajectory features include target boundary features determined by the historical bounding box features of the historical targets and trajectory appearance features determined by the historical appearance features.

[0036] Figure 1 The illustrations depict application scenarios of target tracking methods, apparatus, devices, media, and program products according to embodiments of the present disclosure.

[0037] like Figure 1 As shown, application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 serves as a medium for providing communication links between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links, or fiber optic cables, etc.

[0038] Users can use the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the server 105 via the network 104 to receive or send messages, etc. Various communication client applications can be installed on the first terminal device 101, the second terminal device 102, and the third terminal device 103, such as shopping applications, web browser applications, search applications, instant messaging tools, email clients, social media platform software, etc. (for example only).

[0039] The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be various electronic devices with displays and support web browsing, including but not limited to smartphones, tablets, laptops, and desktop computers.

[0040] Server 105 can be a server that provides various services, such as a backend management server that supports websites browsed by users using the first terminal device 101, the second terminal device 102, and the third terminal device 103 (this is just an example). The backend management server can analyze and process data such as received user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to user requests) to the terminal devices.

[0041] It should be noted that the target tracking method provided in this embodiment can generally be executed by server 105. Correspondingly, the target tracking device provided in this embodiment can generally be located in server 105. The target tracking method provided in this embodiment can also be executed by a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105. Correspondingly, the target tracking device provided in this embodiment can also be located in a server or server cluster that is different from server 105 and capable of communicating with the first terminal device 101, the second terminal device 102, the third terminal device 103, and / or server 105.

[0042] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0043] The following will be based on Figure 1 The described scene, through Figures 2-5 The target tracking method of the disclosed embodiments is described in detail.

[0044] Figure 2 A flowchart illustrating a target tracking method according to an embodiment of the present disclosure is shown schematically.

[0045] like Figure 2 As shown, the method includes operations S210 to S230.

[0046] In operation S210, multi-channel feature extraction is performed on the image to be detected in the current frame to obtain feature maps corresponding to each of the multiple channels, wherein the image to be detected includes the target to be detected.

[0047] In operation S220, target detection is performed on the feature maps corresponding to each of the multiple channels to obtain the target bounding box features and the target appearance features of the detected target.

[0048] In operation S230, based on the target bounding box features of the detected target, the target appearance features of the detected target, and the trajectory features of the tracking trajectory, the matching result between the detected target and the tracking trajectory is determined. The tracking trajectory is determined based on historical detected targets that have been continuously tracked in at least one historical frame. The trajectory features include trajectory bounding box features determined by the historical bounding box features of the historical detected targets and trajectory appearance features determined by the historical appearance features.

[0049] According to embodiments of this disclosure, the current frame can be an image frame in the video to be detected that is being processed by the target tracking method. Image frames preceding the current frame are historical frames relative to the current frame.

[0050] According to embodiments of this disclosure, before processing the video to be detected, the video can be split into consecutive image frames to be detected, and the frame number of each image frame in the video can be marked. A preprocessing algorithm can be used to adjust the resolution of the original image frames to 800 pixels × 1440 pixels, and then the average RGB value of all image frames in the video is subtracted from all image frames.

[0051] According to embodiments of this disclosure, each image to be detected in the video to be detected can be processed sequentially by the target tracking method according to the labeled frame number.

[0052] According to the embodiments of this disclosure, the method of multi-channel feature extraction for the image to be detected is not limited. It can be implemented through the DarkNET53 network, or through FPN (Feature Pyramid Networks), ResNet (Residual Networks), etc.

[0053] According to embodiments of this disclosure, the feature maps corresponding to each of the multiple channels can have different resolutions in addition to having different numbers of channels. For example, the feature maps mentioned above are high-resolution feature maps of 256×100×180, 512×50×90, and 1024×25×45, where 256, 512, and 1024 are the number of channels, respectively.

[0054] According to embodiments of this disclosure, there may be one or more detection targets, and there may also be one or more tracking trajectories.

[0055] According to embodiments of this disclosure, when performing target detection on feature maps corresponding to multiple channels, the bounding box features and appearance features of the target can be extracted from the multiple feature maps respectively, thereby obtaining the target bounding box features and appearance features of all detected targets included in the image to be detected.

[0056] According to embodiments of this disclosure, the tracking trajectory can be a detection target that has appeared in at least one historical frame of the image to be detected.

[0057] According to embodiments of this disclosure, the trajectory features of the tracking trajectory may include trajectory bounding box features and trajectory appearance features. Specifically, if the image to be detected in the nearest historical frame to the current frame contains a historical detection target matching the tracking trajectory, the trajectory bounding box feature is the historical bounding box feature of that historical detection target. If the image to be detected in the nearest historical frame to the current frame does not contain a historical detection target matching the tracking trajectory, but other historical frames contain a detection target matching the image to be detected in the tracking trajectory, the trajectory bounding box feature is the historical bounding box of the detection target in the other historical frames closest to the current frame. Similarly, the trajectory appearance features and trajectory bounding box features are both obtained from the relevant features of the historical detection targets in the most recently appearing historical frame of the tracking trajectory.

[0058] According to embodiments of this disclosure, by performing feature matching between the target bounding box features of the detected target, the target appearance features of the detected target, and the trajectory features of the tracking trajectory determined from the image to be detected in historical frames, it is determined whether the detected target matches the tracking trajectory, thereby achieving continuous tracking and recognition of the tracking trajectory.

[0059] According to the target tracking method disclosed herein, multi-channel feature extraction is employed to extract features from the image to be detected at multiple scales, obtaining feature maps corresponding to each channel. This achieves the extraction of both detailed and global features of the image to be detected. Furthermore, during target detection, not only are the target bounding box features extracted, but also the target appearance features. This allows for matching the tracking trajectory with the detected target based on information from both appearance and bounding box features during subsequent matching. Because target detection is performed using feature maps corresponding to each channel, and the matching result between the tracking trajectory and the detected target is determined based on both bounding box and appearance features, this method at least partially solves the technical problem of insufficient accuracy in target detection and tracking in related technologies, thereby improving the accuracy of target tracking.

[0060] According to embodiments of this disclosure, target detection is performed on feature maps corresponding to multiple channels to obtain target bounding box features and target appearance features of the detected target, which may include the following steps.

[0061] The feature maps corresponding to each of the multiple channels are processed in a unified manner to obtain multiple target feature maps with the same number of channels; the multiple target feature maps are input into the first feature extraction model to obtain the target bounding box features of the detected target; the multiple target feature maps are input into the second feature extraction model to obtain the target appearance features of the detected target.

[0062] According to embodiments of this disclosure, the number of channels in the feature maps corresponding to multiple channels can be unified. For example, the number of channels can be unified from 256, 512, and 1024 to 256 through 1×1 convolution.

[0063] According to the embodiments of this disclosure, the implementation methods of the first feature extraction model and the second feature extraction model are not limited, and both can be implemented by neural networks, such as convolutional neural networks.

[0064] According to embodiments of this disclosure, the target bounding box features may include: target bounding box size information, the coordinates of the target bounding box center in the image (i.e., the bounding box position), target bounding box confidence level, etc.

[0065] According to embodiments of this disclosure, inputting multiple target feature maps into a second feature extraction model to obtain the target appearance features of the detected target may include the following steps.

[0066] Multiple target feature maps are input into the first appearance extraction sub-model to obtain the initial appearance features of the detected target; temporal information analysis is performed on the initial appearance features of the detected target and the historical appearance features of at least one neighboring historical detected target to obtain the target appearance features of the detected target, wherein the neighboring historical detected target is the historical detected target included in the image to be detected in the M nearest historical frames to the current frame, and M is an integer greater than or equal to 1.

[0067] According to embodiments of this disclosure, the first feature extraction model may include a first appearance extraction sub-model and a temporal processing sub-model.

[0068] According to embodiments of this disclosure, a temporal processing sub-model is used to perform temporal information analysis on the initial appearance features of the input detection target and the historical appearance features of at least one neighboring historical detection target to obtain the final target appearance features of the detection target, thereby making the final target appearance features of the detection target contain temporal information.

[0069] According to embodiments of this disclosure, the temporal processing sub-model can be implemented using ConvLSTM (Convolutional Long Short Term Memory) neurons, a variant of LSTM (Long Short Term Memory) neurons. LSTM is a special type of recurrent neural network that can use time series data to analyze input data, thereby introducing temporal information. ConvLSTM optimizes the neuron structure on top of LSTM, achieving the same effect while reducing the number of parameters, thus accelerating training speed.

[0070] According to an embodiment of this disclosure, if the current frame is the first frame of the input video, then the initial appearance features of all detected targets in the current frame are used as the target appearance features of the detected targets.

[0071] According to embodiments of this disclosure, the specific value of M can be determined based on the frame number of the current frame. For example, if the current frame is the third frame, meaning there are two historical frames before the current frame, then M can be 2 or 1.

[0072] According to embodiments of this disclosure, temporal information is introduced when extracting the target appearance features of the detected target, making the target appearance features of the detected target more accurate, improving the accuracy of the detected target, and thus improving the tracking performance.

[0073] Figure 3 The diagram schematically illustrates a model structure for determining the target bounding box features and the target appearance features of a detected target according to an embodiment of the present disclosure.

[0074] like Figure 3 As shown, the image to be detected 301 of the current frame is input into the image feature extraction layer 302 to obtain feature maps corresponding to multiple channels. The feature maps corresponding to multiple channels are then input into the stems layer 303, where channel normalization is performed to obtain multiple target feature maps with the same number of channels. These multiple target feature maps are then input into the first feature extraction model 304, the second feature extraction model 305, and the third feature extraction model 306 to obtain the target bounding box features and the target appearance features of the detected target. Finally, the target bounding box features, the target appearance features, and the trajectory features 307 of the tracking trajectory are input into the matching module 308 to determine the matching result 309 between the detected target and the tracking trajectory.

[0075] According to embodiments of this disclosure, the first feature extraction model 304 may include the following processing layers: Reg_conv, Reg_preds, and Obj_preds. Reg_conv is implemented by a convolutional neural network and is used to extract the target bounding box features of the detected targets included in multiple target feature maps; Reg_preds and Obj_preds may also be implemented by convolutional neural networks and are used to map the extracted target bounding box features of the detected targets from the feature space to the prediction space.

[0076] According to embodiments of this disclosure, the second feature extraction model 305 may include the following processing layers: Redi_conv, ConvlLSTM, and Reid_preds. Redi_conv is the first appearance extraction sub-model, which may include: two 3×3 convolutional sub-layers, a BN (Batch Normalization) sub-layer, and a SiLU (Sigmoid Gated Linear Unit) sub-layer. The two 3×3 convolutional sub-layers are responsible for extracting relevant information about the initial appearance features of the detected target; the BN layer is responsible for batch data normalization, performing normalization on all data obtained by the convolutional sub-layers within the same batch in this network structure. This structure aims to normalize the data to a uniform interval, thereby reducing data divergence, making the network model training process more stable, and reducing the complexity of the neural network learning process; the SiLU sub-layer is an activation function with characteristics of sigmoid and ReLU (Rectified Linear Unit), used to appropriately map feature values ​​to the prediction interval. Compared to the ReLU sublayer, the activation function used in the SiLU sublayer has a smoother curve near 0, thus retaining more input information.

[0077] According to embodiments of this disclosure, the Reid_preds layer consists of a 1×1 convolutional layer used to map data from the feature space to the prediction space. The ConvlLSTM is a temporal processing sub-model used to perform temporal processing on the historical appearance features and initial appearance features of nearby historical detected targets, thereby outputting the target appearance features of the detected targets in the current frame, including temporal information.

[0078] According to embodiments of this disclosure, the third feature extraction model 306 may include Cls_conv and Cls_preds, wherein both Cls_conv and Cls_preds can be obtained by a convolutional neural network, Cls_conv is used to limit the tracking trajectory category and the upper limit of the tracking trajectory, and Cls_preds is used to perform feature mapping.

[0079] According to embodiments of this disclosure, in addition to the bounding box information of the target, appearance information and timing information of the target are also introduced during target detection, thereby improving the detection accuracy of the target and thus improving the overall tracking performance.

[0080] According to embodiments of this disclosure, inputting multiple target feature maps into a second feature extraction model to obtain the target appearance features of the detected target may further include the following steps.

[0081] Multiple target feature maps are input into the second appearance extraction sub-model to obtain the target appearance features of the detected target. The second appearance extraction sub-model includes a feature extraction layer and a mapping layer. The feature extraction layer includes at least six convolutional sub-layers.

[0082] According to embodiments of this disclosure, the second appearance extraction sub-model may include at least six convolutional sub-layers, which deepen the appearance feature extraction structure, thereby enabling the extraction of more accurate appearance features.

[0083] Figure 4 The diagram schematically illustrates a model structure for determining the appearance features of a target to be detected, according to another embodiment of the present disclosure.

[0084] like Figure 4 As shown, the second appearance extraction sub-model can include a convs layer and a pred layer. The convs layer can include six 3×3 convolutional sub-layers, namely 3×3conv2d, BN sub-layer and SiLU sub-layer. The pred layer can be composed of a 1×1 convolutional layer, which is used to map the data from the feature space to the prediction space.

[0085] According to embodiments of this disclosure, during the research process, considering the inference time factor of target tracking algorithms, excessively deep network structures can affect the inference speed of the model and the immediacy of the tracking method. Comparative experiments were conducted on second appearance extraction sub-models with different depths (i.e., different numbers of convolutional sub-layers) under conditions where the impact on inference speed is relatively small. The tracking results were compared using commonly used metrics for evaluating target tracking results, such as IDF1 (Identification F1-Score), MOTA (Multiple Object Tracking Accuracy), and IDsw (Identification Switch). It was found that the second appearance extraction sub-model with six convolutional sub-layers achieved the best tracking effect for detected targets.

[0086] According to embodiments of this disclosure,

[0087] The target bounding box features include: target bounding box confidence and target bounding box position; the trajectory bounding box features include: trajectory bounding box position and trajectory velocity and trajectory acceleration corresponding to the trajectory bounding box position; based on the target bounding box features of the detected target, the target appearance features of the detected target, and the trajectory features of the tracking trajectory, the matching result between the detected target and the tracking trajectory is determined, which may include the following operations.

[0088] If the confidence level of the target bounding box is greater than a preset confidence level, the distance between the appearance features of the detected target and the appearance features of the trajectory is calculated to obtain the appearance distance. Based on the trajectory bounding box position, the trajectory velocity and trajectory acceleration corresponding to the trajectory bounding box position, the predicted bounding box position of the tracking trajectory in the current frame is predicted. The distance between the predicted bounding box position and the target bounding box position is calculated to obtain the bounding box distance. Based on the appearance distance and the bounding box distance, the matching cost matrix is ​​obtained. Based on the matching cost matrix, the matching result between the detected target and the tracking trajectory is determined. According to embodiments of this disclosure, the detected target and the tracking trajectory can be matched using different matching methods based on the confidence level of the detected target's bounding box.

[0089] According to embodiments of this disclosure, the preset confidence level is not limited, but can be limited according to actual needs.

[0090] According to the embodiments of this disclosure, the method of calculating the appearance distance is not limited, and can be cosine distance calculation or Pearson correlation coefficient calculation, etc., wherein the cosine distance calculation method is as shown in the following formula (1).

[0091] (1)

[0092] in, This refers to the appearance features of the target being detected. The trajectory appearance features are used to track the trajectory.

[0093] According to embodiments of this disclosure, the method for predicting the bounding box position of the current frame is not limited, and can be performed using a Kalman filter algorithm.

[0094] According to the embodiments of this disclosure, the method for calculating the bounding box distance is not limited, and it can be obtained by the Mahalanobis distance calculation method, as shown in formula (2).

[0095] (2)

[0096] in, This represents the Mahalanobis distance between the i-th tracking trajectory and the j-th detected target. Let the vector represent the bounding box position of the j-th detected target be... Let the vector represent the predicted bounding box position of the i-th tracking trajectory be... for and The covariance matrix between them.

[0097] According to embodiments of this disclosure, the apparent distance between each detected target and each tracking trajectory can be used as a parameter for the corresponding position to obtain an initial matching cost matrix; and the bounding box distance between the detected target and the tracking trajectory corresponding to each position in the initial matching cost matrix is ​​compared to whether it is less than a preset bounding box threshold; thus, if the bounding box distance is less than the preset bounding box distance threshold, the parameter for that position is set to ∞; and if the bounding box distance is greater than the preset bounding box distance threshold, the parameter for that position is not processed and remains the corresponding bounding box distance, thereby obtaining the final matching cost matrix.

[0098] According to embodiments of this disclosure, the tracking trajectory matching each detected target can be determined based on the matching cost matrix using the KM algorithm (Kuhn-Munkres).

[0099] According to embodiments of this disclosure, the matching cost matrix is ​​determined by combining the appearance distance with the bounding box distance, and the final matching result of the detected target and the tracking trajectory is determined, which can improve the matching accuracy, i.e., improve the tracking accuracy.

[0100] According to embodiments of this disclosure, the target bounding box features include: target bounding box confidence, target bounding box position, and target bounding box size information; the trajectory bounding box features include: trajectory bounding box position, trajectory velocity corresponding to the trajectory bounding box position, trajectory acceleration, and trajectory bounding box size information; wherein, determining the matching result between the detected target and the tracking trajectory based on the target bounding box features of the detected target, the target appearance features of the detected target, and the trajectory features of the tracking trajectory can include the following operations.

[0101] If the confidence level of the target bounding box is less than or equal to a preset confidence level, the predicted bounding box position and size of the tracking trajectory in the current frame are predicted based on the trajectory bounding box position, the trajectory velocity, trajectory acceleration, and trajectory bounding box size information corresponding to the trajectory bounding box position. Based on the predicted bounding box position and size information, a predicted sub-image is determined. Based on the target bounding box position and size information, the current sub-image is determined. Based on the pixel set included in the predicted sub-image and the pixel set included in the current sub-image, the distance between the predicted bounding box features and the target bounding box features is determined. Based on the distance, the matching result between the detected target and the tracking trajectory is determined.

[0102] According to embodiments of this disclosure, if the confidence level of the target bounding box is less than or equal to a preset confidence level, it can be considered that the detected target may be in a more extreme scenario, such as a high occlusion scenario, which makes the appearance features of the inspected target unreliable.

[0103] According to embodiments of this disclosure, the Kalman filter algorithm can be used to predict the predicted bounding box position and predicted bounding box size information of the current frame.

[0104] According to embodiments of this disclosure, the target bounding box size information may include the bounding box width, the bounding box width, and the bounding box aspect ratio. The trajectory bounding box size information may include the trajectory bounding box width, the trajectory bounding box width, and the trajectory bounding box aspect ratio.

[0105] According to embodiments of this disclosure, the trajectory velocity and trajectory acceleration corresponding to the trajectory bounding box position can be determined by the historical bounding box positions of historically detected targets in multiple historical frames matched with the tracking trajectory.

[0106] According to embodiments of this disclosure, when there are multiple tracking trajectories and multiple detection targets, matching can be performed between unmatched tracking trajectories and unmatched detection targets.

[0107] According to embodiments of this disclosure, during matching, a predicted sub-image corresponding to the predicted bounding box position and a current sub-image corresponding to the bounding box position can be determined from the image to be detected in the current frame. The distance between the predicted bounding box features and the bounding box features can be determined by the set of pixels included in the predicted sub-image and the current sub-image, and the specific calculation formula can be shown in the following formula (3).

[0108] (3)

[0109] Where IoU(A, B) is the distance between the predicted bounding box features and the bounding box features, A is the set of pixels in the predicted sub-image, and B is the set of pixels in the current sub-image.

[0110] According to embodiments of this disclosure, a matching cost matrix can be constructed by the distance between the trajectory bounding box features of each tracking trajectory and the target bounding box features of each detected target, and the tracking trajectory matching each detected target can be determined by the KM algorithm, thus obtaining the matching results between multiple detected targets and multiple tracking trajectories.

[0111] According to embodiments of this disclosure, the distance between the predicted bounding box features and the bounding box features is determined by calculating the intersection between the pixel set of the predicted sub-image and the pixel set of the current sub-image, thereby determining whether the detected target matches the tracking trajectory, which can improve matching accuracy. According to embodiments of this disclosure, the target tracking method may also include the following steps.

[0112] Given that the matching result indicates that the detected target matches the tracking trajectory, the trajectory bounding box feature is replaced with the target bounding box feature. Based on the target appearance feature, the trajectory appearance feature, and the preset appearance weight value, the updated appearance feature is obtained. The updated appearance feature is then used to replace the trajectory appearance feature in the trajectory feature.

[0113] According to the embodiments of this disclosure, when it is determined that the detection target matches the tracking trajectory, the appearance features, trajectory appearance features and preset appearance weight values ​​can be used to obtain the updated appearance features for updating, and the calculation formula can be as shown in the following formula (4).

[0114] (4)

[0115] in, As a vector representing the updated appearance features, A vector representing the appearance features of the trajectory. The vector trajectory appearance feature is used to characterize the appearance features, and 0.1 and 0.9 are preset appearance weight values.

[0116] According to embodiments of this disclosure, the updated appearance features calculated in the above manner can retain certain historical appearance features on the basis of the appearance features of the current frame, thereby improving the accuracy of matching the detected target and the tracking trajectory in subsequent frames.

[0117] According to embodiments of this disclosure, the target tracking method may further include the following steps.

[0118] If the matching result indicates a mismatch between the detected target and the tracking trajectory, and the confidence level of the detected target's bounding box is greater than a preset confidence level, the detected target is designated as a new tracking target. The trajectory features of the new tracking trajectory are determined based on the target's appearance features and the target bounding box features. The current frame is designated as a lost frame of the tracking trajectory, so that if the lost frames of the tracking trajectory are consecutive and the number of lost frames exceeds a preset frame count threshold, the tracking trajectory is deleted. According to embodiments of this disclosure, the preset frame count threshold is not limited and can be set according to actual conditions.

[0119] According to embodiments of this disclosure, a new tracking trajectory associated with the detected target can be established.

[0120] According to embodiments of this disclosure, when it is determined that the matching result indicates that the detection target and the tracking trajectory do not match, the number of lost frames of the tracking trajectory can be counted. If the lost frames are consecutive frames and the number of frames is greater than a preset frame number threshold, the tracking trajectory is discarded and the relevant information of the tracking trajectory is deleted.

[0121] According to embodiments of this disclosure, the detected target can also be used as a new tracking trajectory, and its target appearance features and target bounding box features can be determined as the trajectory features of the new tracking trajectory.

[0122] According to embodiments of this disclosure, if the tracking trajectory is not deleted, for the image to be detected in the next frame of the current frame, the tracking trajectory includes a new tracking trajectory and a tracking trajectory.

[0123] According to embodiments of this disclosure, a target tracking method is performed for each frame of the video to be detected, thereby obtaining tracking information of all tracking trajectories in the video to be detected.

[0124] According to embodiments of this disclosure, the first feature extraction model and the second feature extraction model are trained in the following manner.

[0125] Multiple sample target feature maps from the sample image are input into an initial first feature extraction model to obtain sample bounding box features of the sample targets included in the sample image; a first loss value is obtained based on the sample bounding box features and the ground truth bounding box features; multiple sample target feature maps are input into an initial second feature extraction model to obtain sample appearance features of the sample targets; a second loss value is obtained based on the predicted and ground truth labels of the sample appearance features; a target loss value is determined based on the first loss value, the first weight value corresponding to the first loss value, the second loss value, and the second weight value corresponding to the second loss value; the initial first feature extraction model and the initial second feature extraction model are trained based on the target loss value to obtain the first feature extraction model and the second feature extraction model.

[0126] According to an embodiment of this disclosure, the formula for calculating the target loss function can be as shown in the following formula (5).

[0127] (5)

[0128] Where det_loss is the first loss value and loss_id is the second loss value. As the first weight value, This is the second weight value.

[0129] According to embodiments of this disclosure, by jointly training the first feature extraction model and the second feature extraction model, the accuracy of model training can be improved, thereby making the final tracking results more accurate.

[0130] According to embodiments of this disclosure, at least one sample target included in the sample image can be pre-annotated with bounding boxes and labels to facilitate the determination of the true bounding box features and true labels of each sample target.

[0131] According to embodiments of this disclosure, the predicted identifier of the sample appearance features can be output by a second feature extraction model. By inputting multiple sample target feature maps into an initial second feature extraction model, the sample appearance features and predicted identifier of the sample targets can be obtained.

[0132] According to embodiments of this disclosure, appearance information and temporal information are introduced during the training process. This information is backpropagated through the loss function to positively constrain the prediction process of the detection box information, thereby improving the accuracy of target detection and thus improving the tracking performance of the algorithm.

[0133] Figure 5 A flowchart illustrating a target tracking method according to another embodiment of the present disclosure is shown.

[0134] like Figure 5 As shown, the method includes operations S501 to S513.

[0135] In operation S501, the video to be detected is split into consecutive image frames;

[0136] In operation S502, multi-channel feature extraction is performed on the image to be detected in the current frame to obtain feature maps corresponding to each of the multiple channels.

[0137] In operation S503, the feature maps corresponding to each of the multiple channels are processed in a unified manner to obtain multiple target feature maps with the same number of channels.

[0138] In operation S504, multiple target feature maps are input into the first feature extraction model to obtain the target bounding box features of the detected target.

[0139] In operation S505, multiple target feature maps are input into the first appearance extraction sub-model to obtain the initial appearance features of the detected target.

[0140] In operation S506, the initial appearance features of the detected target and the historical appearance features of at least one neighboring historical detected target are analyzed using time-series information to obtain the target appearance features of the detected target. The neighboring historical detected targets are the historical detected targets included in the images of the M nearest historical frames to the current frame, where M is an integer greater than or equal to 1.

[0141] In operation S507, when there are multiple targets to be detected, the targets are classified according to the confidence of the bounding box features of each target, resulting in high-confidence targets and low-confidence targets.

[0142] In operation S508, based on the target bounding box features, appearance features, and trajectory features of the high-confidence target, the high-confidence target is matched with the tracking trajectory to obtain a matching result. If the matching result indicates that a tracking trajectory matches the high-confidence target, operation S512 is executed. If the matching result indicates that no tracking trajectory matches the high-confidence target, operation S510 is executed.

[0143] According to embodiments of this disclosure, there is at least one tracking trajectory.

[0144] In operation S509, based on the target bounding box features of the low-confidence target, the target appearance features of the low-confidence target, and the trajectory features of the remaining tracking trajectories that do not match the high-confidence target, the low-confidence target and the remaining tracking trajectories are matched respectively to obtain matching results. If the matching result indicates that there is a remaining tracking trajectory that matches the low-confidence target, operation S512 is executed. If the matching result indicates that there is no remaining tracking trajectory that matches the low-confidence target, operation S511 is executed.

[0145] When operating the S510, the detected target is used as the new tracking trajectory.

[0146] When operating S511, low-confidence targets are discarded.

[0147] In operation S512, the trajectory features of the tracking trajectory are updated based on the target appearance features and target bounding box features of the detected target.

[0148] In operation S513, the image to be detected in the next frame is read, and the image to be detected in the next frame is processed based on S502~S509.

[0149] Based on the above target tracking method, this disclosure also provides a target tracking device. The following will be combined with... Figure 6 The device is described in detail.

[0150] Figure 6 A schematic block diagram of a target tracking device according to an embodiment of the present disclosure is shown.

[0151] like Figure 6 As shown, the target tracking device 600 of this embodiment includes an extraction module 610, a detection module 620, and a matching module 630.

[0152] The extraction module 610 is used to extract multi-channel features from the image to be detected in the current frame to obtain feature maps corresponding to each of the multiple channels, wherein the image to be detected includes the target to be detected.

[0153] The detection module 620 is used to perform target detection on the feature maps corresponding to multiple channels, and obtain the target bounding box features and target appearance features of the detected target.

[0154] The matching module 630 is used to determine the matching result between the detected target and the tracking trajectory based on the target bounding box features, the target appearance features, and the trajectory features of the tracking trajectory. The tracking trajectory is determined based on historical detected targets that have been continuously tracked in at least one historical frame. The trajectory features include trajectory bounding box features determined by the historical bounding box features of the historical detected targets and trajectory appearance features determined by the historical appearance features.

[0155] According to an embodiment of this disclosure, the detection module 620 includes: a processing submodule, a first extraction submodule, and a second extraction submodule.

[0156] The processing submodule is used to perform channel-unified processing on the feature maps corresponding to multiple channels to obtain multiple target feature maps with the same number of channels.

[0157] The first extraction submodule is used to input multiple target feature maps into the first feature extraction model to obtain the target bounding box features of the detected target.

[0158] The second extraction submodule is used to input multiple target feature maps into the second feature extraction model to obtain the target appearance features of the detected target.

[0159] According to embodiments of this disclosure, the second extraction submodule includes: a first extraction unit and an information analysis unit.

[0160] The first extraction unit is used to input multiple target feature maps into the first appearance extraction sub-model to obtain the initial appearance features of the detected target.

[0161] The information analysis unit is used to perform temporal information analysis on the initial appearance features of the detected target and the historical appearance features of at least one neighboring historical detected target to obtain the target appearance features of the detected target. The neighboring historical detected targets are the historical detected targets included in the images of the M nearest historical frames to the current frame, where M is an integer greater than or equal to 1.

[0162] According to embodiments of this disclosure, appearance features are extracted from multiple target feature maps to obtain the appearance features of each detected target, including: a second extraction unit.

[0163] The second extraction unit is used to input multiple target feature maps into the second appearance extraction sub-model to obtain the target appearance features of the detected target. The second appearance extraction sub-model includes a feature extraction layer and a mapping layer. The feature extraction layer includes at least six convolutional sub-layers.

[0164] According to embodiments of this disclosure, the target bounding box features include: target bounding box confidence and target bounding box position; the trajectory bounding box features include: trajectory bounding box position and trajectory velocity and trajectory acceleration corresponding to the trajectory bounding box position. The matching module 630 includes: a first calculation submodule, a first prediction submodule, a second calculation submodule, a matrix determination submodule, and a first result determination submodule.

[0165] The first calculation submodule is used to calculate the distance between the appearance features of the detected target and the appearance features of the trajectory when the confidence of the target bounding box is greater than the preset confidence, and obtain the appearance distance.

[0166] The first prediction submodule is used to predict the predicted bounding box position of the tracking trajectory in the current frame based on the trajectory bounding box position, the trajectory velocity and trajectory acceleration corresponding to the trajectory bounding box position.

[0167] The second calculation submodule is used to calculate the distance between the predicted bounding box position and the target bounding box position to obtain the bounding box distance.

[0168] The matrix determination submodule is used to obtain the matching cost matrix based on the appearance distance and the bounding box distance.

[0169] The first result determination submodule is used to determine the matching result between the detected target and the tracking trajectory based on the matching cost matrix.

[0170] According to embodiments of this disclosure, the target bounding box features include: target bounding box confidence, target bounding box position, and target bounding box size information; the trajectory bounding box features include: trajectory bounding box position, trajectory velocity corresponding to the trajectory bounding box position, trajectory acceleration, and trajectory bounding box size information. The matching module 630 includes: a second prediction submodule, a first image determination submodule, a second image determination submodule, a third calculation submodule, and a first result determination submodule.

[0171] The second prediction submodule is used to predict the predicted bounding box position and predicted bounding box size information of the tracking trajectory in the current frame based on the trajectory bounding box position, the trajectory velocity, trajectory acceleration and trajectory bounding box size information corresponding to the trajectory bounding box position, when the confidence of the target bounding box is less than or equal to the preset confidence.

[0172] The first image determination submodule is used to determine the predicted sub-image based on the predicted bounding box position and predicted bounding box size information.

[0173] The second image determination submodule is used to determine the current sub-image based on the target bounding box position and target bounding box size information.

[0174] The third calculation submodule is used to determine the distance between the predicted bounding box features and the target bounding box features based on the pixel set included in the predicted sub-image and the pixel set included in the current sub-image.

[0175] The first result determination submodule is used to determine the matching result between the detected target and the tracking trajectory based on distance.

[0176] According to embodiments of this disclosure, the target tracking device 600 further includes: a first replacement module, an updated feature determination module, and a second replacement module.

[0177] The first replacement module is used to replace the trajectory bounding box features in the trajectory features with the target bounding box features when the matching result indicates that the detected target matches the tracking trajectory.

[0178] The updated feature determination module is used to obtain updated appearance features based on target appearance features, trajectory appearance features, and preset appearance weight values.

[0179] The second replacement module is used to replace the trajectory appearance features in the trajectory features with updated appearance features.

[0180] According to embodiments of this disclosure, the target tracking device 600 further includes: a new target determination module, a feature determination module, and a deletion module.

[0181] The new target determination module is used to treat the current frame as a lost frame in the tracking trajectory, so that if the lost frames of the tracking trajectory are consecutive frames and the number of lost frames is greater than a preset frame number threshold, the detected target will be used as a new tracking target. The feature determination module is used to determine the trajectory features of the new tracking trajectory based on the target appearance features and the target bounding box features.

[0182] The deletion module is used to treat the current frame as a lost frame of the tracking trajectory, so that the tracking trajectory can be deleted if the lost frames of the tracking trajectory are consecutive frames and the number of lost frames is greater than a preset frame number threshold.

[0183] According to embodiments of the present disclosure, the target tracking device 600 further includes: a first input module, a first loss value determination module, a second input module, a second loss value determination module, a target loss value determination module, and a training module.

[0184] The first input module is used to input multiple sample target feature maps of the sample image into the initial first feature extraction model to obtain the sample bounding box features of the sample targets included in the sample image.

[0185] The first loss value determination module is used to obtain the first loss value based on the sample bounding box features and the ground truth bounding box features.

[0186] The second input module is used to input multiple sample target feature maps into the initial second feature extraction model to obtain the sample appearance features of the sample target.

[0187] The second loss value determination module is used to obtain the second loss value based on the predicted and true labels of the sample appearance features.

[0188] The target loss value determination module is used to determine the target loss value based on a first loss value, a first weight value corresponding to the first loss value, a second loss value, and a second weight value corresponding to the second loss value.

[0189] The training module is used to train the initial first feature extraction model and the initial second feature extraction model based on the target loss value, so as to obtain the first feature extraction model and the second feature extraction model.

[0190] According to embodiments of this disclosure, any plurality of modules among the extraction module 610, detection module 620, and matching module 630 can be combined into one module, or any one of these modules can be split into multiple modules. Alternatively, at least a portion of the functionality of one or more of these modules can be combined with at least a portion of the functionality of other modules and implemented in one module. According to embodiments of this disclosure, at least one of the extraction module 610, detection module 620, and matching module 630 can be at least partially implemented as hardware circuitry, such as a field-programmable gate array (FPGA), a programmable logic array (PLA), a system-on-a-chip, a system-on-a-substrate, a system-on-package, an application-specific integrated circuit (ASIC), or implemented in hardware or firmware by any other reasonable means of integrating or packaging circuitry, or implemented in software, hardware, or firmware, or in any suitable combination of any of these three implementation methods. Alternatively, at least one of the extraction module 610, detection module 620, and matching module 630 can be at least partially implemented as a computer program module, which, when run, can perform corresponding functions.

[0191] Figure 7 A block diagram schematically illustrates an electronic device suitable for implementing a target tracking method according to an embodiment of the present disclosure.

[0192] like Figure 7As shown, an electronic device 700 according to an embodiment of the present disclosure includes a processor 701, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 702 or a program loaded from a storage portion 708 into a random access memory (RAM) 703. The processor 701 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or an associated chipset and / or a special-purpose microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 701 may also include onboard memory for caching purposes. The processor 701 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0193] RAM 703 stores various programs and data required for the operation of electronic device 700. Processor 701, ROM 702, and RAM 703 are interconnected via bus 704. Processor 701 performs various operations of the method flow according to embodiments of the present disclosure by executing programs in ROM 702 and / or RAM 703. It should be noted that programs may also be stored in one or more memories other than ROM 702 and RAM 703. Processor 701 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

[0194] According to embodiments of this disclosure, the electronic device 700 may further include an input / output (I / O) interface 705, which is also connected to a bus 704. The electronic device 700 may also include one or more of the following components connected to the input / output (I / O) interface 705: an input section 706 including a keyboard, mouse, etc.; an output section 707 including a cathode ray tube (CRT), liquid crystal display (LCD), etc., and a speaker, etc.; a storage section 708 including a hard disk, etc.; and a communication section 709 including a network interface card such as a LAN card, modem, etc. The communication section 709 performs communication processing via a network such as the Internet. A drive 710 is also connected to the input / output (I / O) interface 705 as needed. A removable medium 711, such as a disk, optical disk, magneto-optical disk, semiconductor memory, etc., is installed on the drive 710 as needed so that computer programs read from it can be installed into the storage section 708 as needed.

[0195] This disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments; or it may exist independently and not assembled into the device / apparatus / system. The computer-readable storage medium carries one or more programs that, when executed, implement the method according to the embodiments of this disclosure.

[0196] According to embodiments of this disclosure, the computer-readable storage medium can be a non-volatile computer-readable storage medium, such as including, but not limited to: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this disclosure, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. For example, according to embodiments of this disclosure, the computer-readable storage medium may include ROM 702 and / or RAM 703 and / or one or more memories other than ROM 702 and RAM 703 described above.

[0197] Embodiments of this disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowchart. When the computer program product is run on a computer system, the program code is used to cause the computer system to implement the target tracking method provided in the embodiments of this disclosure.

[0198] When the computer program is executed by the processor 701, it performs the functions defined in the system / apparatus of this disclosure embodiments. According to embodiments of this disclosure, the systems, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0199] In one embodiment, the computer program may rely on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may also be transmitted and distributed in the form of signals over a network medium, and may be downloaded and installed via the communication section 709, and / or installed from a removable medium 711. The program code contained in the computer program can be transmitted using any suitable network medium, including but not limited to: wireless, wired, etc., or any suitable combination thereof.

[0200] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 709, and / or installed from the removable medium 711. When the computer program is executed by the processor 701, it performs the functions defined in the system of this disclosure embodiment. According to embodiments of this disclosure, the systems, devices, apparatuses, modules, units, etc., described above can be implemented by computer program modules.

[0201] According to embodiments of this disclosure, program code for executing the computer programs provided in embodiments of this disclosure can be written in any combination of one or more programming languages. Specifically, these computational programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, Python, "C", or similar programming languages. The program code can execute entirely on a user's computing device, partially on a user's device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving remote computing devices, the remote computing device can be connected to the user's computing device via any type of network, including a local area network (LAN) or a wide area network (WAN), or it can be connected to an external computing device (e.g., via the Internet using an Internet service provider).

[0202] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in a block diagram or flowchart, and combinations of blocks in a block diagram or flowchart, may be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0203] Those skilled in the art will understand that the features described in the various embodiments of this disclosure can be combined and / or combined in various ways, even if such combinations or combinations are not explicitly described in this disclosure. In particular, the features described in the various embodiments of this disclosure can be combined and / or combined in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or combinations fall within the scope of this disclosure.

[0204] The embodiments of this disclosure have been described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of this disclosure. Although various embodiments have been described above, this does not mean that the measures in the various embodiments cannot be used advantageously in combination. Various substitutions and modifications can be made by those skilled in the art without departing from the scope of this disclosure, and all such substitutions and modifications should fall within the scope of this disclosure.

Claims

1. A target tracking method, characterized in that, The method includes: Multi-channel feature extraction is performed on the image to be detected in the current frame to obtain feature maps corresponding to each of the multiple channels, wherein the image to be detected includes the target to be detected; Target detection is performed on the feature maps corresponding to each of the multiple channels to obtain the target bounding box features and the target appearance features of the detected target. Based on the target bounding box features of the detected target, the target appearance features of the detected target, and the trajectory features of the tracking trajectory, the matching result between the detected target and the tracking trajectory is determined, wherein the tracking trajectory is determined based on historical detected targets that have been continuously tracked in at least one historical frame, and the trajectory features include trajectory bounding box features determined by the historical bounding box features of the historical detected targets and trajectory appearance features determined by the historical appearance features. The step of performing target detection on the feature maps corresponding to each of the multiple channels to obtain the target bounding box features and the target appearance features of the detected target includes: The feature maps corresponding to each of the multiple channels are subjected to channel unification processing to obtain multiple target feature maps with the same number of channels; The multiple target feature maps are input into the first feature extraction model to obtain the target bounding box features of the detected target; The multiple target feature maps are input into the second feature extraction model to obtain the target appearance features of the detected target; The step of inputting multiple target feature maps into a second feature extraction model to obtain the target appearance features of the detected target includes: Multiple target feature maps are input into the first appearance extraction sub-model to obtain the initial appearance features of the detected target; temporal information analysis is performed on the initial appearance features of the detected target and the historical appearance features of at least one neighboring historical detected target to obtain the target appearance features of the detected target, wherein the neighboring historical detected target is the historical detected target included in the image to be detected in the M nearest historical frames to the current frame, and M is an integer greater than or equal to 1; Alternatively, multiple target feature maps can be input into a second appearance extraction sub-model to obtain the target appearance features of the detected target. The second appearance extraction sub-model includes a feature extraction layer and a mapping layer, and the feature extraction layer includes at least six convolutional sub-layers.

2. The method according to claim 1, characterized in that, The target bounding box features include: target bounding box confidence and target bounding box position; the trajectory bounding box features include: trajectory bounding box position and trajectory velocity and trajectory acceleration corresponding to the trajectory bounding box position; The step of determining the matching result between the detected target and the tracking trajectory based on the target bounding box features, the target appearance features, and the trajectory features of the tracking trajectory includes: If the confidence level of the target bounding box is determined to be greater than the preset confidence level... Calculate the distance between the target appearance features and the trajectory appearance features of the detected target to obtain the appearance distance; Based on the trajectory bounding box position, the trajectory velocity and trajectory acceleration corresponding to the trajectory bounding box position, the predicted bounding box position of the tracking trajectory in the current frame is predicted; Calculate the distance between the predicted bounding box position and the target bounding box position to obtain the bounding box distance; Based on the appearance distance and the bounding box distance, the matching cost matrix is ​​obtained; Based on the matching cost matrix, the matching result between the detected target and the tracking trajectory is determined.

3. The method according to claim 1, characterized in that, The target bounding box features include: target bounding box confidence, target bounding box position, and target bounding box size information; the trajectory bounding box features include: trajectory bounding box position, trajectory velocity corresponding to the trajectory bounding box position, trajectory acceleration, and trajectory bounding box size information. The step of determining the matching result between the detected target and the tracking trajectory based on the target bounding box features, the target appearance features, and the trajectory features of the tracking trajectory includes: If the confidence level of the target bounding box is determined to be less than or equal to the preset confidence level... Based on the trajectory bounding box position, the trajectory velocity corresponding to the trajectory bounding box position, the trajectory acceleration, and the trajectory bounding box size information, the predicted bounding box position and predicted bounding box size information of the tracking trajectory in the current frame are predicted; Based on the predicted bounding box position and predicted bounding box size information, a predicted sub-image is determined; The current sub-image is determined based on the target bounding box position and the target bounding box size information; Based on the set of pixels included in the predicted sub-image and the set of pixels included in the current sub-image, the distance between the predicted bounding box feature and the target bounding box feature is determined; Based on the distance, the matching result between the detected target and the tracking trajectory is determined.

4. The method according to claim 1, characterized in that, The method further includes: When the matching result indicates that the detected target matches the tracking trajectory, Replace the trajectory bounding box features in the trajectory features with the target bounding box features; Based on the target appearance features, the trajectory appearance features, and the preset appearance weight value, the updated appearance features are obtained; The updated appearance features replace the trajectory appearance features in the trajectory features.

5. The method according to claim 1, characterized in that, The method further includes: If the matching result indicates that the detected target does not match the tracking trajectory and the confidence level of the target bounding box of the detected target is greater than the preset confidence level, the detected target will be used as a new tracking target. The trajectory features of the new tracking trajectory are determined based on the target appearance features and the target bounding box features; The current frame is taken as the lost frame of the tracking trajectory, so that the tracking trajectory is deleted if the lost frames of the tracking trajectory are consecutive frames and the number of lost frames is greater than a preset frame number threshold.

6. The method according to claim 2, characterized in that, The first feature extraction model and the second feature extraction model are trained in the following way: Multiple sample target feature maps of the sample image are input into the initial first feature extraction model to obtain the sample bounding box features of the sample targets included in the sample image; Based on the sample bounding box features and the ground truth bounding box features, a first loss value is obtained; The feature maps of multiple sample targets are input into the initial second feature extraction model to obtain the sample appearance features of the sample targets; Based on the predicted and true identifiers of the sample appearance features, a second loss value is obtained; Based on the first loss value, the first weight value corresponding to the first loss value, the second loss value, and the second weight value corresponding to the second loss value, a target loss value is determined; The initial first feature extraction model and the initial second feature extraction model are trained based on the target loss value to obtain the first feature extraction model and the second feature extraction model.

7. A target tracking device, characterized in that, The device includes: The extraction module is used to extract multi-channel features from the image to be detected in the current frame to obtain feature maps corresponding to each of the multiple channels, wherein the image to be detected includes the target to be detected; The detection module is used to perform target detection on the feature maps corresponding to each of the multiple channels, and obtain the target bounding box features and the target appearance features of the detected target. A matching module is used to determine the matching result between the detected target and the tracking trajectory based on the target bounding box features of the detected target, the target appearance features of the detected target, and the trajectory features of the tracking trajectory. The tracking trajectory is determined based on historical detected targets that have been continuously tracked in at least one historical frame. The trajectory features include trajectory boundary features determined by the historical bounding box features of the historical detected targets and trajectory appearance features determined by the historical appearance features. The processing submodule is used to perform channel-unified processing on the feature maps corresponding to each of the multiple channels to obtain multiple target feature maps with the same number of channels. The first extraction submodule is used to input multiple target feature maps into the first feature extraction model to obtain the target bounding box features of the detected target; The second extraction submodule is used to input multiple target feature maps into the second feature extraction model to obtain the target appearance features of the detected target; The second extraction submodule includes: The first extraction unit is used to input multiple target feature maps into the first appearance extraction sub-model to obtain the initial appearance features of the detected target; the information analysis unit is used to perform time-series information analysis on the initial appearance features of the detected target and the historical appearance features of at least one neighboring historical detected target to obtain the target appearance features of the detected target, wherein the neighboring historical detected target is the historical detected target included in the image to be detected in the M nearest historical frames to the current frame, and M is an integer greater than or equal to 1; Alternatively, a second extraction unit is used to input multiple target feature maps into a second appearance extraction sub-model to obtain the target appearance features of the detected target, wherein the second appearance extraction sub-model includes a feature extraction layer and a mapping layer, and the feature extraction layer includes at least six convolutional sub-layers.

Images