Target tracking method and device and computer storage medium

Abstract

The invention provides a target tracking method and device and a computer storage medium. The target tracking method comprises the steps of obtaining real-time scene data; extracting feature data in the real-time scene data; using the feature data to initialize a current target query; splicing the tracking target queries according to the current target query to obtain a fused target query; and inputting the fusion target query into a multi-layer decoder, and outputting target information in the real-time scene data. By fusing the target query, the context information of the input data can be better understood, so that the task performance is improved.

Description

Technical Field

[0001] The present application relates to the field of automation technology, and in particular to a target tracking method, device, and computer storage medium. Background Art

[0002] In recent years, autonomous driving and assisted driving technologies have developed rapidly. Perception tasks, as part of the system's mid- and upper-stream modules, experience the most rapid technological updates and iterations. Within this module, the detection and tracking of dynamic targets like vehicles and pedestrians, as well as static targets like lane lines and road edges, are critical tasks. Stable and accurate output of target location, direction, speed, and other information ensures the optimal and safest driving path for the vehicle, ensuring safe operation.

[0003] Perception tasks place stringent demands on the accuracy, stability, and real-time performance of target recognition and detection. Existing technical solutions require different networks and detection heads to handle different tasks. This results in the system having multiple networks to handle different tasks, leading to problems such as limited computing resources and poor real-time performance. Summary of the Invention

[0004] To solve the above technical problems, the present application proposes a target tracking method, which includes: acquiring real-time scene data; extracting feature data from the real-time scene data; initializing a current target query using the feature data; splicing the tracking target query according to the current target query to obtain a fused target query; inputting the fused target query into a multi-layer decoder, and outputting the target information in the real-time scene data.

[0005] The feature data is bird's-eye view feature data or label true value data.

[0006] Among them, the step of inputting the fused target query into a multi-layer decoder and outputting the target information in the real-time scene data includes: inputting the fused target query into a multi-layer decoder, and outputting the dynamic target detection box and the static target detection box in the real-time scene data through a regression branch; inputting the fused target query into a multi-layer decoder, and outputting the dynamic target category and the static target category in the real-time scene data through a classification branch.

[0007] Among them, the target tracking method also includes: inputting the feature data into the position encoding layer, extracting the feature position and feature value of the feature data; inputting the fused target query into a multi-layer decoder, and outputting the target information in the real-time scene data, including: inputting the fused target query, the feature position and the feature value into a multi-layer decoder, and outputting the target information in the real-time scene data.

[0008] The method of initializing the current target query using the feature data includes: obtaining a learnable sampling point set, wherein the sampling point set includes a first number of dynamic target queries and a second number of static target queries; and performing position / feature initialization on the dynamic target query and the static target query respectively using the feature data.

[0009] Among them, the step of inputting the fused target query into a multi-layer decoder and outputting the target information in the real-time scene data includes: performing self-attention information interaction on the static target query and the dynamic target query in the fused target query respectively; reaggregating the interaction information of the static target query and the interaction information of the dynamic target query to obtain an aggregated target query; performing cross-attention information interaction on the aggregated target query with the feature position and the feature value to extract the target information in the real-time scene data.

[0010] Among them, after inputting the fused target query into a multi-layer decoder and outputting the target information in the real-time scene data, the target tracking method further includes: obtaining a newly appeared target based on the target information and the tracking target query; obtaining a confidence score of the newly appeared target; and when the confidence score is higher than a first preset threshold, placing the newly appeared target into a tracking target set.

[0011] Wherein, after inputting the fusion target query into the multi-layer decoder and outputting the target information in the real-time scene data, the target tracking method further includes: comparing the target information with the tracking targets in the tracking target set to obtain the continuously tracked target appearing in the real-time scene data; when the confidence score of the continuously tracked target is lower than a second preset threshold, setting a buffer disappearance time for the continuously tracked target; when the confidence scores of the continuously tracked targets are all lower than the second preset threshold within the buffer disappearance time, removing the continuously tracked target from the tracking target set; wherein, the first preset threshold is higher than the second preset threshold.

[0012] To solve the above technical problems, the present application proposes a target tracking device, which includes a memory and a processor coupled to the memory; wherein the memory is used to store program data, and the processor is used to execute the program data to implement the above target tracking method.

[0013] In order to solve the above technical problems, the present application proposes a computer storage medium, which is used to store program data. When the program data is executed by a computer, it is used to implement the above target tracking method.

[0014] Compared to existing technologies, the present invention has the following advantages: a target tracking device acquires real-time scene data; extracts feature data from the real-time scene data; uses the feature data to initialize a current target query; concatenates tracking target queries based on the current target query to generate a fused target query; and inputs the fused target query into a multi-layer decoder, which outputs target information from the real-time scene data. By fusing target queries, the context of the input data can be better understood, thereby improving task performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0016] Figure 1 This is a flowchart of the first embodiment of the target tracking method provided by the present application;

[0017] Figure 2 This is a schematic diagram of the target tracking framework provided by this application;

[0018] Figure 3 is a schematic diagram of one layer of the multi-layer decoder provided by this application;

[0019] Figure 4 is a flowchart of a second embodiment of the target tracking method provided by this application;

[0020] Figure 5 is a flowchart of the third embodiment of the target tracking method provided by this application;

[0021] Figure 6 This is a schematic structural diagram of an embodiment of a target tracking device provided by the present application;

[0022] Figure 7 It is a structural diagram of an embodiment of a computer storage medium provided by this application. DETAILED DESCRIPTION

[0023] The following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present invention without making any creative efforts shall fall within the scope of protection of the present invention.

[0024] To solve the above technical problems, the present application proposes a target tracking method. In this embodiment, the target tracking method of the present application is applied to a target tracking device, wherein the target tracking device of the present application can be a server or a system composed of a server and a local terminal. Accordingly, the various parts of the target tracking device, such as the various units, subunits, modules, and submodules, can be all set in the server, or can be set separately in the server and the local terminal.

[0025] Furthermore, the above-mentioned server can be hardware or software. When the server is hardware, it can be implemented as a distributed server cluster consisting of multiple servers, or it can be implemented as a single server. When the server is software, it can be implemented as multiple software or software modules, such as software or software modules for providing distributed servers, or it can be implemented as a single software or software module, which is not specifically limited here. In some possible implementations, the target tracking method of the embodiment of the present application can be implemented by a processor calling computer-readable instructions stored in a memory.

[0026] See also Figure 1 , Figure 1 This is a flowchart of the first embodiment of the target tracking method provided by this application.

[0027] like Figure 1 As shown, the specific steps are as follows:

[0028] Step S11: Acquire real-time scene data.

[0029] Specifically, in the embodiments of the present application, the scene data can be obtained by real-time scene video or images captured by a camera, or point cloud data captured by a laser radar. The scene data can be a single-frame scene image, a single-frame laser point cloud data, or a combination of multiple frames of scene images and multiple frames of laser point cloud data.

[0030] Step S12: extracting feature data from the real-time scene data.

[0031] In one embodiment of the present application, the feature data is bird's-eye view feature data, or label true value data.

[0032] Bird's-eye view (BEV) feature data and ground truth data are important data formats for training and evaluating perception models. A bird's-eye view provides an overhead perspective of the vehicle's surroundings, clearly displaying information such as road structure, obstacle locations, and lane markings. Bird's-eye view feature data is a bird's-eye view representation generated by processing raw data collected by sensors (such as cameras and lidar).

[0033] Labeled truth data is annotated data used for supervised learning, typically generated by humans or high-precision equipment. It is the "standard answer" for bird's-eye view feature data and is used to train and evaluate perception models.

[0034] Step S13: Initialize the current target query using the feature data.

[0035] like Figure 2 As shown, Figure 2 This is a schematic diagram of the target tracking framework provided by this application, such as Figure 2 As shown in the figure, the target tracking device initializes a sampling point set and uses a preset number of sampling points in the sampling point set to form a target query. 75% of all target queries are used as dynamic target queries, and 25% of all target queries are used as static target queries. The query is an intermediate representation used to encode the location, category, and feature information of the target.

[0036] It should be noted that the target tracking device selects a larger number of target queries as dynamic target queries in order to improve the detection accuracy of dynamic targets based on the detection difficulty of dynamic targets.

[0037] Please continue to see Figure 2 The target tracking device divides the initialized queries into static target queries and dynamic target queries according to a preset number. The specific steps are: the target tracking device obtains a learnable sampling point set, wherein the sampling point set includes a first number of dynamic target queries and a second number of static target queries; and uses the feature data to respectively initialize the position / feature of the dynamic target queries and the static target queries.

[0038] Specifically, the target tracking device obtains an initial preset number of sampling point sets, and the preset number of sampling point sets are used to detect dynamic targets and static targets. In a specific embodiment of the present application, the first 75% of the sampling point sets are used as dynamic target queries, and the last 25% of the sampling point sets are used as static target queries.

[0039] In an embodiment of the present application, the target tracking device uses the extracted bev features to initialize all current target queries. In other embodiments of the present application, label true value initialization can be used.

[0040] In a specific embodiment of the present application, dynamic targets may be, for example, vehicles and pedestrians, and static targets may be, for example, lane lines and road signs.

[0041] Specifically, the target tracking device uses bev features to position-encode the static and dynamic target queries divided by the above process, thereby encoding the static and dynamic target queries into a unified coding space to accommodate the same multi-layer decoder. The bev features in the current scene data record the predicted position of the target on the image. The target tracking device assigns this predicted position to the static and dynamic target queries, providing relevant position encoding features.

[0042] Step S14: splicing the tracking target query according to the current target query to obtain a fused target query.

[0043] Among them, the tracking target query is a query consisting of target information or target features successfully detected and tracked in the previous frame of scene data, which is used for binary matching with the static target query and dynamic target query of the current scene data.

[0044] For further information, please see Figure 2 ,The target tracking device respectively splices the static target query, the dynamic target query, and the tracking target query to obtain a fused target query.

[0045] The target tracking device initializes the tracking target query of frame t0 to empty. i＞0 Frame, the tracking target query of the target that has been successfully detected in the previous frame and the iteratively updated current target query are spliced together to generate a fused target query as the input of the Unified head multi-layer decoder.

[0046] Step S15: input the fused target query into a multi-layer decoder, and output the target information in the real-time scene data.

[0047] The target query undergoes information exchange with multiple layers of decoders to generate a predicted target query for the next frame. Specifically, the target tracking device inputs the fused target query into the multi-layer decoder, and outputs dynamic target detection frames and static target detection frames in the real-time scene data through the regression branch. The fused target query is also input into the multi-layer decoder, and the classification branch outputs dynamic target categories and static target categories in the real-time scene data.

[0048] Among them, the classification branch is used to predict the category, and the regression branch is used to predict the bounding box coordinates of the object.

[0049] Please continue to see Figure 2 , this application also includes a position encoder, taking a sinusoidal position encoder as an example, which performs position encoding on the features of the input image and inputs the encoded features into a multi-layer encoder.

[0050] Specifically, the target tracking device inputs the feature data into a position encoder to extract the feature position and feature value of the feature data. The target tracking device inputs the fused target query Q, the feature position K, and the feature value V into a multi-layer decoder, outputs the target information in the real-time scene data, and expresses it in the form of a target query.

[0051] Through position coding, dynamic target queries and static target queries are encoded into a unified query, that is, a query of length 400, which contains the encoded information of dynamic targets and static targets. It is updated in the subsequent multi-layer decoder to extract the key information of dynamic and static targets.

[0052] This application can simultaneously perform multiple tasks—target detection and data association—to achieve a truly unified, multi-task, end-to-end detection and tracking system. The system utilizes an attention mechanism to effectively capture the contextual relationships between targets, enhancing the model's performance in complex environments like occlusion and recurrence, effectively improving both detection and tracking performance.

[0053] Among them, about Figure 2 The specific framework structure of the multi-layer decoder shown in FIG. 1 is a schematic diagram of a decoder for extracting target information from real-time scene data. Figure 3 and Figure 4 As shown, Figure 3 is a schematic diagram of one layer of the multi-layer decoder provided by this application, Figure 4 This is a flowchart of the second embodiment of the target tracking method provided by this application.

[0054] like Figure 4 As shown, the specific steps are as follows:

[0055] Step S21: performing self-attention information interaction on the static target query and the dynamic target query in the fused target query respectively.

[0056] See Figure 3 This application adopts a specially designed multi-level instance attention head to perform a decoupled attention mechanism between each query. The first 300 static target query queries and the last 100 dynamic target query queries perform instance decoupled attention mechanisms between queries, and the points within the query perform self-attention operations.

[0057] Step S22: re-aggregating the interaction information of the static target query and the interaction information of the dynamic target query to obtain an aggregated target query.

[0058] Figure 3 The structure of the schematic diagram is the component inside the Unified head, Figure 3The figure shows a one-layer decoder, which adopts a decoupled aggregation multi-level attention mechanism. Through the hierarchical attention mechanism, it can well extract the key information of dynamic targets and static targets.

[0059] Specifically, the target tracking device first decouples 300 dynamic target queries and 100 static target queries among the 400 fused target queries, and performs self-attention between instances, that is, self-attention between dynamic target queries and dynamic target queries and between 20 points within each query, and self-attention between static target queries and between 20 points within each query to obtain information interaction between instances; then the interaction information of dynamic targets is aggregated with the interaction information of static targets, and then cross-attention is performed with the input K and V to learn the global feature information (more biased towards the fused interaction between dynamic targets and static targets, and there will also be information interaction between points within the query), and then obtain the updated query.

[0060] Step S23: Perform cross-attention information interaction on the aggregated target query, the feature position, and the feature value to extract target information from the real-time scene data.

[0061] The updates of static target query and dynamic target query (ie, the appearance and disappearance of the target) can be obtained according to the updated query, and thus the tracking target query will also be updated.

[0062] After the fusion target query is input into the multi-layer decoder and the target information in the real-time scene data is output, the present application also proposes an embodiment. Figure 5 , Figure 5 This is a flowchart of the third embodiment of the target tracking method provided by this application.

[0063] like Figure 5 As shown, the specific steps are as follows:

[0064] Step S31: Based on the target information and the tracking target query, a newly appeared target is obtained.

[0065] Specifically, the target tracking device obtains a newly appeared target based on the target information and the tracking target query.

[0066] Step S32: Obtain the confidence score of the newly appeared target.

[0067] Specifically, the target tracking device obtains the confidence score score of the newly appeared target.

[0068] Step S33: When the confidence score is higher than a first preset threshold, the newly appeared target is added to the tracking target set.

[0069] The target tracking device initializes the ID of the newly appeared target to -1, and when the score is greater than the preset threshold (0.4), the new target is added to the track query set, that is, the tracking target set.

[0070] Furthermore, the target tracking device compares the target information with the tracking targets in the tracking target set to obtain a continuously tracked target appearing in the real-time scene data; when the confidence score of the continuously tracked target is lower than the second preset threshold, a buffer expiration time is set for the continuously tracked target; when the confidence scores of the continuously tracked targets are all lower than the second preset threshold within the buffer expiration time, the continuously tracked target is removed from the tracking target set.

[0071] The first preset threshold is higher than the second preset threshold.

[0072] Specifically, for a target that already existed in the previous frame, that is, a tracking target with id ≥ 0, when the score is less than the threshold (0.3), the target is considered to have disappeared. However, there may be unstable detection or occlusion, resulting in a lower score, so the target tracking device will set a buffer disappearance time d_t and initialize the id of the tracking target that may have unstable detection or occlusion to 0. At this time, d_t will be accumulated. When d_t> 5, and within this time period, the score is lower than the threshold (0.3), the tracking target is abandoned, that is, the id is reset to -1.

[0073] When score ≥ threshold (0.3), the target is a target that can be tracked continuously.

[0074] Specifically, the query output by the multi-layer decoder detection passes through the classification branch and the regression branch to obtain the classification information of the predicted dynamic targets and the classification information of the static targets, respectively. This includes dynamic targets such as cars and pedestrians, static categories such as lane lines and curbs, and position coordinates, such as the box size and orientation of dynamic target vehicles, the coordinates of static targets such as lane line points, and the target detection confidence, i.e., the score.

[0075] In order to implement the target tracking method of the above embodiment, this application also provides a target tracking device, please refer to Figure 6 , Figure 6 It is a structural diagram of an embodiment of a target tracking device provided by this application.

[0076] like Figure 6 As shown, the target tracking device 600 of this embodiment includes a processor 61 , a memory 62 , an input and output device 63 , and a bus 64 .

[0077] The processor 61 , the memory 62 , and the input / output device 63 are respectively connected to the bus 64 . The memory 62 stores a computer program, and the processor 61 is used to execute the computer program to implement the target tracking method of the above embodiment.

[0078] In this embodiment, the processor 61 may also be referred to as a CPU (Central Processing Unit). The processor 61 may be an integrated circuit chip with signal processing capabilities. The processor 61 may also be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic device, a discrete gate or transistor logic device, or a discrete hardware component. The processor 61 may also be a GPU (Graphics Processing Unit), also known as a display core, visual processor, or display chip. This is a microprocessor specifically designed for image processing on computers, workstations, game consoles, and some mobile devices (such as tablets and smartphones). The purpose of a GPU is to convert and drive the display information required by the computer system and provide horizontal scanning signals to the display to control the correct display of the display. It is a key component connecting the display to the computer motherboard. A graphics card, as a key component of a computer host, is responsible for outputting display graphics. A general-purpose processor may be a microprocessor, or the processor 61 may be any conventional processor.

[0079] This application also provides a computer storage medium, such as Figure 7 As shown, the computer storage medium 700 is used to store a computer program 71. When the computer program 71 is executed by the processor, it is used to implement the method described in the embodiment of the target tracking method of the present application.

[0080] The method involved in the target tracking method embodiment of the present application, when implemented in the form of a software functional unit and sold or used as an independent product, can be stored in a device, such as a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: various media that can store program code, such as a U disk, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a disk or an optical disk.

[0081] In the several embodiments provided in this application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation methods described above are only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation. For example, units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0082] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0083] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0084] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application, or the part that contributes to the prior art, or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for enabling a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of the methods of each embodiment of the present application.

[0085] The above description is only an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent structure or equivalent process transformation made by using the contents of the description and drawings of the present invention, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A target tracking method, characterized in that: The target tracking method comprises: Get real-time scene data; Extracting feature data from the real-time scene data; Initializing a current target query using the feature data; splicing the tracking target query according to the current target query to obtain a fused target query; The fused target query is input into a multi-layer decoder, and target information in the real-time scene data is output.

2. The target tracking method according to claim 1, characterized in that The feature data is bird's-eye view feature data, or label true value data.

3. The target tracking method according to claim 1, wherein: Inputting the fused target query into a multi-layer decoder and outputting target information in the real-time scene data comprises: Inputting the fused target query into a multi-layer decoder, and outputting a dynamic target detection frame and a static target detection frame in the real-time scene data through a regression branch; The fused target query is input into a multi-layer decoder, and the dynamic target category and the static target category in the real-time scene data are output through a classification branch.

4. The target tracking method according to claim 3, characterized in that: The target tracking method further includes: Inputting the feature data into a position encoding layer to extract the feature position and feature value of the feature data; Inputting the fused target query into a multi-layer decoder and outputting target information in the real-time scene data includes: The fused target query, the feature position, and the feature value are input into a multi-layer decoder, and target information in the real-time scene data is output.

5. The target tracking method according to claim 4, characterized in that: The initializing the current target query using the feature data includes: Acquire a learnable sampling point set, wherein the sampling point set includes a first number of dynamic target queries and a second number of static target queries; The feature data is used to perform position / feature initialization on the dynamic target query and the static target query respectively.

6. The target tracking method according to claim 5, characterized in that: Inputting the fused target query into a multi-layer decoder and outputting target information in the real-time scene data includes: Performing self-attention information interaction on the static target query and the dynamic target query in the fused target query respectively; Reaggregating the interaction information of the static target query and the interaction information of the dynamic target query to obtain an aggregated target query; The aggregated target query is subjected to cross-attention information interaction with the feature position and the feature value to extract target information from the real-time scene data.

7. The target tracking method according to claim 1, characterized in that: After inputting the fused target query into a multi-layer decoder and outputting target information in the real-time scene data, the target tracking method further includes: Acquire a newly appeared target based on the target information and the tracking target query; Obtaining a confidence score for the newly appeared target; When the confidence score is higher than a first preset threshold, the newly appeared target is added to a tracking target set.

8. The target tracking method according to claim 7, characterized in that: After inputting the fused target query into a multi-layer decoder and outputting target information in the real-time scene data, the target tracking method further includes: Comparing the target information with the tracking targets in the tracking target set to obtain a continuously tracked target appearing in the real-time scene data; When the confidence score of the continuously tracked target is lower than a second preset threshold, setting a buffer expiration time for the continuously tracked target; When the confidence scores of the continuously tracked targets are all lower than the second preset threshold within the buffer expiration time, the continuously tracked targets are removed from the tracked target set; The first preset threshold is higher than the second preset threshold.

9. A target tracking device, characterized in that: The target tracking device includes a memory and a processor coupled to the memory; The memory is used to store program data, and the processor is used to execute the program data to implement the target tracking method according to any one of claims 1 to 8.

10. A computer storage medium, characterized in that The computer storage medium is used to store program data, and when the program data is executed by a computer, it is used to implement the target tracking method according to any one of claims 1 to 8.

Images