Target tracking method and device, model training method and device, equipment and medium
By using an end-to-end object detection model training method, binding labels and calculating loss values, the process of multi-object tracking is simplified, solving the problems of high time cost and complex post-processing in existing technologies, and improving processing efficiency and accuracy.
Patent Information
- Application Number
- CN202410452558.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-04-15
- Publication Date
- 2025-10-24
AI Technical Summary
In existing multi-target tracking technologies, the two-stage paradigm method based on KF requires two independent algorithm models, which is time-consuming and involves complex post-processing, affecting the algorithm's speed and generalization ability.
An end-to-end object detection model is adopted. By performing object detection on multiple frames of images during training, binding labels and calculating loss values, the process of association and matching between two frames is eliminated, and backpropagation training is performed directly, simplifying it into a single-stage model.
It improves the processing efficiency of target tracking, reduces complex post-processing, and enhances algorithm speed and generalization ability, especially improving tracking accuracy in occluded and dense scenes.
Smart Images
Figure CN120833465A_ABST
Abstract
Description
TECHNICAL FIELD
[0001] The present disclosure relates to the technical field of artificial intelligence, and particularly relates to a target tracking method and model training method, device, equipment and medium. BACKGROUND
[0002] MOT (Multiple Object Tracking) refers to detecting and tracking multiple targets such as pedestrians, vehicles and animals in a video under the condition that the number of targets is unknown in advance.
[0003] KF (Kalman Filter) is an algorithm for optimally estimating the state of a system by using linear system state equations and observing data of system input and output.
[0004] In related technologies, MOT is implemented by using a two-stage paradigm (target detection + KF tracking) method, that is, target detection is first performed on an image, and then post-processing is performed on the target detection result by using a KF tracking algorithm to realize target tracking. The two processes need to be performed in sequence, and therefore two independent algorithm models are needed to support them. However, this method needs to construct and improve the two independent algorithm models respectively, which consumes a high time cost, and at the same time, there is a complex post-processing process, which affects the algorithm speed and generalization ability, and the processing efficiency of target tracking needs to be improved. SUMMARY
[0005] To solve the above technical problems, the present disclosure provides a target tracking method and model training method, device, equipment and medium.
[0006] In a first aspect, an embodiment of the present disclosure provides a training method of a target detection model, comprising:
[0007] obtaining a training set, the training set comprising multiple frames of images, and the images being labeled with detection box ground truths of targets;
[0008] inputting each frame of image into a target detection model to perform target detection, and obtaining multiple prediction boxes of the each frame of image;
[0009] for a first target in the each frame of image, matching a first detection box ground truth of the first target with the multiple prediction boxes to obtain a first prediction box corresponding to the first detection box ground truth; the first target is a target without a bound label;
[0010] calculating a loss of the first detection box ground truth and the corresponding first prediction box to obtain a first loss value, and binding a label to the first target and the first prediction box;
[0011] For a second target in the each frame image, a second loss is calculated for a second detection box ground truth of the second target and a corresponding second prediction box, to obtain a second loss value; the second target is a target with a bound label;
[0012] The first loss value and / or the second loss value are used for back propagation to train the target detection model.
[0013] In a second aspect, the embodiments of the present disclosure provide a target tracking method, comprising:
[0014] Obtaining a target detection image sequence, the target detection image sequence comprising a plurality of target detection images;
[0015] Inputting the target detection image sequence into a target detection model for target detection, to obtain a target detection result in each target detection image, and to obtain a label of each target detection result;
[0016] Determining each target detection result with the same label in different target detection images as an associated target detection result.
[0017] In a third aspect, the embodiments of the present disclosure provide a training device of a target detection model, comprising:
[0018] An obtaining module, configured to obtain a training set, the training set comprising a plurality of images, and each target in the images being labeled with a detection box ground truth;
[0019] A prediction module, configured to input each image into a target detection model for target detection, to obtain a plurality of prediction boxes of the each image;
[0020] A matching module, configured to, for a first target in the each image, match a first detection box ground truth of the first target with the plurality of prediction boxes, to obtain a first prediction box corresponding to the first detection box ground truth; the first target being a target without a bound label;
[0021] A first loss calculation module, configured to calculate a loss for the first detection box ground truth and a corresponding first prediction box, to obtain a first loss value, and to bind a label for the first target and the first prediction box;
[0022] A second loss calculation module, configured to, for a second target in the each image, calculate a second loss for a second detection box ground truth of the second target and a corresponding second prediction box, to obtain a second loss value; the second target being a target with a bound label;
[0023] A training module, configured to perform back propagation according to the first loss value and / or the second loss value, to train the target detection model.
[0024] In a fourth aspect, the embodiments of the present disclosure provide a target tracking device, comprising:
[0025] an acquisition module configured to acquire a sequence of images to be detected, the sequence of images to be detected comprising a plurality of images to be detected;
[0026] a detection module configured to input the sequence of images to be detected into the target detection model to perform target detection, to obtain a target detection result in each image to be detected, and to obtain a label of each target detection result;
[0027] an association module configured to determine each target detection result with the same label in different images to be detected as an associated target detection result.
[0028] In a fifth aspect, the embodiments of the present disclosure provide an electronic device, comprising: a processor; a memory configured to store executable instructions of the processor; and the processor configured to read the executable instructions from the memory and execute the instructions to implement the method of the first aspect or the second aspect.
[0029] In a sixth aspect, the embodiments of the present disclosure provide a computer-readable storage medium, the storage medium storing a computer program, the computer program being executed by a processor to implement the method of the first aspect or the second aspect.
[0030] In a seventh aspect, the embodiments of the present disclosure provide a vehicle, comprising the target tracking device of the fourth aspect or the electronic device of the fifth aspect.
[0031] The technical solution provided by the embodiments of the present disclosure has the following advantages compared with the prior art: by performing target detection on multiple images in the training process to obtain multiple prediction boxes of each image, for a first target in each image that is not bound with a label, matching a first detection box ground truth of the first target with the multiple prediction boxes to obtain a first prediction box corresponding to each first detection box ground truth, and binding the first target and the first prediction box with a label, and then, when calculating a loss, calculating the loss for each first detection box ground truth and its corresponding first prediction box, each second detection box ground truth in each image that is bound with a label and its corresponding second prediction box, obtaining a loss value, and performing back propagation according to the loss value to train the target detection model, thus, by training the end-to-end target detection model in the above manner, the training method by binding labels eliminates the process of association and matching between two frames, and by using the end-to-end target detection model for dynamic target tracking, compared with the two-stage paradigm (target detection + KF tracking) method in the existing MOT scene, the process of association and matching between two frames is eliminated, and KF tracking is no longer needed, thus solving the problem of complex post-processing process in the traditional KF-based tracking method, which affects the algorithm speed and generalization ability, and improving the processing efficiency of target tracking. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.
[0033] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the embodiments or the description of the prior art. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.
[0034] Figure 1 A flowchart of a method for training a target detection model provided by an embodiment of the present disclosure;
[0035] Figure 2 A flowchart of another method for training a target detection model provided by an embodiment of the present disclosure;
[0036] Figure 3 A flowchart of a target tracking method provided by an embodiment of the present disclosure;
[0037] Figure 4 A schematic diagram of the structure of a training device for an object detection model provided in an embodiment of the present disclosure;
[0038] Figure 5 A schematic diagram of the structure of a target tracking device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION
[0039] In order to more clearly understand the above-mentioned objectives, features and advantages of the present disclosure, the scheme of the present disclosure will be further described below. It should be noted that the embodiments of the present disclosure and the features therein can be combined with each other in the absence of conflict.
[0040] In the following description, many specific details are set forth to facilitate a full understanding of the present disclosure, but the present disclosure may also be implemented in other ways different from those described herein; it is obvious that the embodiments in the specification are only part of the embodiments of the present disclosure, rather than all of the embodiments.
[0041] Figure 1 This is a flow chart of a method for training a target detection model provided in an embodiment of the present disclosure. The method provided in an embodiment of the present disclosure can be executed by a training device for a target detection model, which can be implemented in software and / or hardware and can be integrated on any electronic device with computing capabilities.
[0042] like Figure 1As shown, the training method of the target detection model provided by the embodiments of the present disclosure can include:
[0043] In step 101, a training set is obtained, the training set including multiple image frames, and each target detection box ground truth is labeled on the image.
[0044] The method of the embodiments of the present disclosure can be used in a MOT (Multiple Object Tracking) scene, in which multiple targets to be detected in a video are detected and tracked by assigning an identifier.
[0045] In this embodiment, the training set is used to train an end-to-end target detection model to realize target tracking through the end-to-end target detection model. Optionally, the multiple image frames are consecutive multiple image frames, such as multiple video frames in a video, for example, a video obtained by capturing the surrounding environment during vehicle driving. Each target detection box ground truth is labeled on each image, wherein the targets include but are not limited to vehicles, pedestrians, animals, etc.
[0046] In step 102, each image is input into the target detection model for target detection to obtain multiple predicted boxes of each image.
[0047] In this embodiment, feature extraction is performed on the multiple image frames to obtain image features of the multiple image frames, and the image features of each image are input into the target detection model for target detection to obtain multiple predicted boxes of each image.
[0048] Optionally, the target detection model can adopt a neural network based on a self-attention mechanism, for example, a DETR (Detection Transformer) model, which is a Transformer-based end-to-end target detection network.
[0049] As an example, for 10 targets in an image frame and their corresponding 10 detection box ground truths, the image frame is input into the target detection model for target detection to obtain 900 predicted boxes. In this example, taking DETR as an example, the predicted boxes are Object Queries, which are vector representations of the predicted boxes of the detected targets output by the target detection model.
[0050] In step 103, for a first target in each image, the first detection box ground truth of the first target is matched with the multiple predicted boxes to obtain a first predicted box corresponding to the first detection box ground truth.
[0051] In this embodiment, the first target is a target without a bound label, and the first target includes a target that first appears in multiple images or a target that reappears after being unbound, for example, a target in the first image in the multiple images, which is a first target without a bound label.
[0052] Optionally, a bipartite graph matching algorithm (Hungarian algorithm) is used to match the first detection box ground truth of each first target with a plurality of prediction boxes, and a first prediction box corresponding to the first detection box ground truth of each first target is determined from the plurality of prediction boxes.
[0053] In step 104, a loss is calculated for the first detection box ground truth and the first prediction box corresponding thereto, a first loss value is obtained, and the first target and the first prediction box are bound with a label.
[0054] In this embodiment, for each first target, a preset loss function is used to calculate a loss according to the first detection box ground truth and the first prediction box corresponding thereto, and a first loss value is obtained. In addition, a label is assigned to each first target and the first prediction box corresponding thereto and is bound therewith, and the form of the label is not specifically limited. For example, the label is track id in DETR. By binding the first target and the corresponding first prediction box with a label, the first target and the corresponding first prediction box use the label in subsequent images.
[0055] In step 105, for each second target in each image, a loss is calculated for the second detection box ground truth of the second target and the second prediction box corresponding thereto, and a second loss value is obtained.
[0056] In this embodiment, the second target is a target that has been bound with a label in the current image, and the first target in the previous image is extended to the current image after being bound with a label. For example, target A in the first image obtains a corresponding prediction box a through step 103, and target A and prediction box a are bound with a label after step 104. Target A still exists in the second image, and target A is a second target that has been bound with a label in the second image. The detection box ground truth of target A in the second image is a second detection box ground truth, and prediction box a is a second prediction box corresponding to target A in the second image.
[0057] In the embodiment, for each second target, a preset loss function is used to calculate a loss value according to the second detection frame true value and the corresponding second prediction frame. That is, in the step, for the second target that has been bound with the label, the foregoing matching step is no longer performed, and the loss is directly calculated. Specifically, after the prediction frame is bound with the label, the prediction frame is added to a preset set to inherit and pass. The prediction frame carries the bound label to the current frame image, and the loss is calculated for each prediction frame and the target in the current frame image that has the same label. The target in the current frame image that has not been bound with the label refers to the foregoing step of the first target.
[0058] In an embodiment of the present disclosure, for the third target, the number of consecutive image frames in which the third target does not appear is obtained. If the number of image frames is greater than a threshold value, the binding relationship between the third target and the label is released. The third target is a target that has been bound with the label but does not appear in the current frame image. The threshold value can be set as needed. In the embodiment, for the target that has been bound with the label but does not exist in the current frame image, if the number of consecutive image frames in which the target does not appear is greater than the threshold value, the binding relationship between the third target and the label is released, and the label is deleted. If the number of consecutive image frames in which the target does not appear is less than or equal to the threshold value, the label is set to be empty but is still retained, so as to cope with the occlusion situation.
[0059] In step 106, the first loss value and / or the second loss value is used for back propagation to train the target detection model.
[0060] In the embodiment, after the first loss value and / or the second loss value is obtained, the total loss is calculated according to the first loss value and / or the second loss value, and the total loss is used for back propagation to adjust the parameters of the target detection model, so as to train the target detection model.
[0061] As an example, when the current frame image only contains the first target, for example, for the first frame image, the first loss value is used for back propagation to train the target detection model.
[0062] As another example, when the current frame image only contains the second target, the second loss value is used for back propagation to train the target detection model.
[0063] As still another example, when the current frame image contains the first target and the second target, the total loss is calculated according to the first loss value and the second loss value, and the total loss is used for back propagation to train the target detection model.
[0064] According to the technical scheme of the embodiment of the present disclosure, in the training process, the target detection is performed on the multiple images to obtain multiple prediction boxes of each image. For a first target in each image without a bound label, the first detection box ground truth of the first target is matched with the multiple prediction boxes to obtain a first prediction box corresponding to each first detection box ground truth, and the first target and the first prediction box are bound with a label. Then, when calculating the loss, the loss is calculated for each first detection box ground truth and the corresponding first prediction box, each second detection box ground truth in each image bound with a label and the corresponding second prediction box, to obtain a loss value, and the loss value is back propagated to train the target detection model. Thus, the end-to-end target detection model trained in the above manner saves the process of association and matching between two frames through the training method of binding labels. When the end-to-end target detection model is used for dynamic target tracking, compared with the two-stage paradigm (target detection + KF tracking) in the existing MOT scene, the process of association and matching between two frames is saved, and KF tracking is no longer needed. The problem of complex post-processing process in the traditional KF-based tracking method affecting the algorithm speed and generalization ability is solved, and the processing efficiency of target tracking is improved.
[0065] Based on the above embodiment, Figure 2 Another flowchart of a method for training a target detection model is provided in the embodiment of the present disclosure, as shown in Figure 2 The method comprises the following steps:
[0066] In step 201, a training set containing multiple images is obtained, and the corresponding multi-modal features of each image are extracted to input the multi-modal features into a target detection model for target detection to obtain multiple prediction boxes of each image.
[0067] In this embodiment, for the image at time T in the multiple images, the camera image collected by the camera at time T is subjected to feature extraction to obtain image features of the camera image, and the radar point cloud collected by the radar at time T is subjected to feature extraction to obtain radar features of the radar point cloud. Then, the image features and the radar features are subjected to feature fusion to generate multi-modal features, and the multi-modal features are input into a target detection model for target detection.
[0068] The vehicle is provided with sensors of two modalities of cameras and radars, the radar can adopt a laser radar, and the feature fusion can be realized based on a CMT (Cross Modal Transformer, a multi-modal fusion model based on Transformer). In the feature fusion stage, the image and the laser radar point cloud are processed through a backbone network to obtain multi-modal features.
[0069] Optionally, the target detection model comprises a neural network based on an attention mechanism, and the multi-modal features are input into the target detection model for target detection, which comprises: acquiring image coordinate encoding and point cloud coordinate encoding, adding the image coordinate encoding and the point cloud coordinate encoding to image features and radar features respectively to obtain multi-modal features after position encoding, realizing encoding of 3D coordinates into the multi-modal features through the coordinate encoding, and then inputting the multi-modal features after position encoding into the target detection model for target detection, for example, decoding the multi-modal features by using a decoder of a Transformer to obtain a result of target detection, and specifically, a Query generated based on position prior interacts with the multi-modal features in the decoder of the Transformer to predict a category and a bounding box of a target.
[0070] The generation of the image coordinate encoding is exemplified as follows: for a given image feature, each image feature corresponds to a group of points in a camera frustum coordinate system, a maximum detection distance D is set, n points are sampled within the distance range of D on the ray, and 3D coordinates of each sampling point on the ray to a radar coordinate system are obtained through a coordinate transformation matrix from each camera to the laser radar, and then the 3D coordinates are input into an MLP (Multilayer Perceptron) layer to obtain the image coordinate encoding.
[0071] The generation of the point cloud coordinate encoding is exemplified as follows: in a bev (Bird's Eye View) space, sampling points in the bev space are generated according to a set range of length and width and a sampling density, and the sampling points are input into an MLP layer to obtain the coordinate encoding of the point cloud.
[0072] The position encoding of the Query is exemplified as follows: n anchor points are initialized for the Query by uniformly sampling from [0, 1], and after the normalized coordinates are converted to a 3D world space according to a set bev range, the position encoding of the Query is obtained by inputting the normalized coordinates into an MLP layer.
[0073] In this way, the 3D position is implicitly encoded into the multi-modal features, avoiding the bias existing in explicit cross-view feature alignment, and the robustness is high, and the performance of the model can be ensured in the case of missing laser radar.
[0074] Further, to ensure the reliability of the model in the case of single camera failure, complete camera failure and laser radar failure, single modal data is randomly used in the training process with a certain probability, for example, single image features, single radar features and multi-modal features are used with a certain probability, respectively, to ensure that the model is trained under single modal and multi-modal, so that the model can work under single modal and multi-modal without adjusting the network weight, and the robustness of the method is ensured.
[0075] Step 202, for the first target in each frame of image, matching the first detection box ground truth of the first target with a plurality of prediction boxes to obtain a first prediction box corresponding to the first detection box ground truth.
[0076] Step 203, calculating the loss of the first detection box ground truth and the first prediction box corresponding thereto to obtain a first loss value, and binding the first target and the first prediction box with a label.
[0077] Step 204, for the second target in each frame of image, calculating the loss of the second detection box ground truth of the second target and the second prediction box corresponding thereto to obtain a second loss value.
[0078] Step 205, in the process of each parameter adjustment, performing back propagation according to the first loss value and / or the second loss value corresponding to one frame of image in the plurality of frames of image to adjust the parameters of the target detection model.
[0079] In the embodiment, for the Nth frame of image in the plurality of frames of image, performing back propagation according to the first loss value and / or the second loss value corresponding to the Nth frame of image to adjust the parameters of the target detection model, and after the back propagation in the foregoing step is completed, performing back propagation according to the first loss value and / or the second loss value corresponding to the N+1th frame of image to adjust the parameters of the target detection model, wherein N is a positive integer greater than or equal to 1.
[0080] In the embodiment of the present disclosure, by inputting only one frame of image each time, performing loss calculation and back propagation based on the frame of image, and then performing the training step of the next frame, compared with the existing training method of inputting multiple frames of data at one time and performing forward propagation calculation, and then uniformly calculating the loss and performing back propagation after the forward process of all frames is completed, the training speed is improved, and the training method is easier to expand, and longer time sequence can be added.
[0081] Based on the above embodiment, Figure 3 A flowchart of a target tracking method provided by the embodiment of the present disclosure is shown in Figure 3 The method comprises the following steps:
[0082] Step 301, acquiring a to-be-detected image sequence, the to-be-detected image sequence comprising a plurality of to-be-detected images.
[0083] Step 302, inputting the to-be-detected image sequence into a target detection model to perform target detection, obtaining a target detection result in each to-be-detected image, and obtaining a label of each target detection result. The target detection model can be obtained by training in step 106.
[0084] Step 303, determining each target detection result with the same label in different to-be-detected images as an associated target detection result.
[0085] In this embodiment, the end-to-end target detection model trained in the foregoing embodiment is used, which can perform target detection on an input image sequence, obtain target detection results in each image in the image sequence, and obtain labels of each target detection result. Each target detection result with the same label in different images is determined as an associated target detection result, and target detection results with different labels are determined as unrelated target detection results, so as to realize target tracking. Compared with the two-stage paradigm (target detection + KF tracking) in the existing MOT scene, the end-to-end target detection model trained in the foregoing embodiment eliminates the process of association and matching between two frames through the binding of Query, instead of reassigning a new label to each Object Query in each frame by the original DETR. The target tracking is realized through the transmission of Query, and KF tracking is no longer needed. The problem of complex post-processing in the traditional KF-based matching mode is solved, the speed and generalization ability of the algorithm are seriously affected, the processing efficiency of target tracking is improved, and for occlusion and dense scenes, the situation of identity exchange is reduced, and the accuracy of target tracking is improved.
[0086] The embodiments of the present disclosure also provide a target detection model training device and a target tracking device.
[0087] Figure 4 A structure diagram of a target detection model training device provided by the embodiments of the present disclosure is shown in FIG. 1, which includes an acquisition module 41, a prediction module 42, a matching module 43, a first loss calculation module 44, a second loss calculation module 45, and a training module 46. Figure 4
[0088] The acquisition module 41 is configured to acquire a training set, wherein the training set includes multiple images, and each target in the images is labeled with a detection box ground truth.
[0089] The prediction module 42 is configured to input each image into a target detection model to perform target detection, and obtain multiple prediction boxes of each image.
[0090] The matching module 43 is configured to match a first detection box ground truth of a first target in each image with the multiple prediction boxes to obtain a first prediction box corresponding to the first detection box ground truth, wherein the first target is a target without a bound label.
[0091] The first loss calculation module 44 is configured to calculate a loss of the first detection box ground truth and the first prediction box corresponding thereto to obtain a first loss value, and bind a label of the first target and the first prediction box.
[0092] The second loss calculation module 45 is configured to calculate a loss value of a second target in each frame of image by calculating a second detection box ground truth of the second target and a corresponding second prediction box, wherein the second target is a target with a bound label.
[0093] The training module 46 is configured to perform back propagation according to the first loss value and / or the second loss value to train the target detection model.
[0094] In an embodiment of the present disclosure, the device further comprises an unbinding module configured to, for a third target, acquire a number of consecutive image frames in which the third target does not appear, wherein the third target is a target with a bound label but does not appear in a current frame of image; and if the number of image frames is greater than a threshold value, unbind the third target and the label.
[0095] In an embodiment of the present disclosure, the prediction module 42 is specifically configured to perform feature extraction on a camera image to obtain image features of the camera image, perform feature extraction on a radar point cloud to obtain radar features of the radar point cloud, and perform feature fusion on the image features and the radar features to generate multi-modal features, and input the multi-modal features into the target detection model for target detection.
[0096] In an embodiment of the present disclosure, the target detection model comprises a neural network based on a self-attention mechanism, and the prediction module 42 is specifically configured to acquire image coordinate encoding and point cloud coordinate encoding, add the image coordinate encoding and the point cloud coordinate encoding to the image features and the radar features respectively to obtain multi-modal features after position encoding, and input the multi-modal features after position encoding into the target detection model for target detection.
[0097] In an embodiment of the present disclosure, the training module 46 is specifically configured to, in each parameter adjustment process, perform back propagation according to the first loss value and / or the second loss value corresponding to one frame of image in the multiple frames of image to adjust parameters of the target detection model.
[0098] Figure 5 A structural schematic diagram of a target tracking device provided by an embodiment of the present disclosure is shown in FIG. 1. Figure 5 As shown in FIG. 1, the target tracking device comprises an acquisition module 51, a detection module 52, and an association module 53.
[0099] The acquisition module 51 is configured to acquire a to-be-detected image sequence, wherein the to-be-detected image sequence comprises a plurality of to-be-detected images.
[0100] The detection module 52 is configured to input the to-be-detected image sequence into the target detection model to perform target detection, to obtain a target detection result in each to-be-detected image, and to obtain a label of each target detection result.
[0101] The association module 53 is configured to determine target detection results with the same label in different to-be-detected images as associated target detection results.
[0102] The apparatus provided in the embodiments of the present disclosure can perform the corresponding method provided in the embodiments of the present disclosure, and has the corresponding function modules and advantages of performing the method. The contents not described in detail in the apparatus embodiments of the present disclosure can be referred to the description in any method embodiment of the present disclosure.
[0103] The embodiments of the present disclosure also provide an electronic device, which includes one or more processors and a memory.
[0104] The processor can be a central processing unit (CPU) or other forms of processing units having data processing and / or instruction execution capabilities, and can control other components in the electronic device to perform desired functions.
[0105] The memory can include one or more computer program products, which can include various forms of computer readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may, for example, include random access memory (RAM), cache, and / or the like. The non-volatile memory may, for example, include read-only memory (ROM), hard disk, flash memory, and / or the like. One or more computer program instructions can be stored on the computer readable storage medium, and the processor can run the program instructions to implement the method of the embodiments of the present disclosure described above and / or other desired functions. Various contents such as input signals, signal components, noise components, and the like can also be stored in the computer readable storage medium.
[0106] In one example, the electronic device can further include an input device and an output device, which are interconnected through a bus system and / or other forms of connection mechanism. In addition, the input device can further include, for example, a keyboard, a mouse, and the like. The output device can output various information to the outside, including the determined distance information, direction information, and the like. The output device can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and the like.
[0107] Of course, for simplicity, only some of the components in the electronic device related to the present disclosure are shown, and components such as buses, input / output interfaces, and the like are omitted. In addition, according to specific application circumstances, the electronic device can further include any other appropriate components.
[0108] In addition to the method and the device described above, the embodiments of the present disclosure can also be a computer program product, which includes computer program instructions, and the computer program instructions cause the processor to perform any method provided by the embodiments of the present disclosure when the processor runs the computer program instructions.
[0109] The computer program product can be written in any combination of one or more programming languages to perform the operations of the embodiments of the present disclosure, including an object-oriented programming language, such as Java, C++, and the like, and a conventional procedural programming language, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device, or entirely on the remote computing device or server.
[0110] In addition, the embodiments of the present disclosure can also be a computer readable storage medium, which stores computer program instructions, and the computer program instructions cause the processor to perform any method provided by the embodiments of the present disclosure when the processor runs the computer program instructions.
[0111] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium may, for example, include but is not limited to an electrical, a magnetic, an optical, an electromagnetic, an infrared, or a semiconductor system, device or apparatus, or any combination of the above. More specific examples (a non-exhaustive list) of the readable storage medium include an electrical connection having one or more wires, a portable disc, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or a flash memory), an optical fiber, a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0112] It should be noted that, in this paper, the relationship terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or device. Without more limitations, the element defined by the statement "including a" does not exclude the presence of another identical element in the process, method, article or device including the element.
[0113] The foregoing is merely illustrative of the various implementations of the present disclosure and the general principles thereof. Numerous modifications can be made to these illustrations, and equivalents can be substituted therefor, without departing from the scope of the present disclosure. The specific embodiments commensurate with the specific application are intended to be illustrative only and not limiting of the scope of the application as set forth in the following claims.
Claims
1. A method for training a target detection model, the method comprising: The method comprises: obtaining a training set, the training set comprising multiple image frames, and each image being labeled with a ground truth bounding box of each target; inputting each image into a target detection model to perform target detection, and obtaining multiple predicted bounding boxes of each image; for a first target in each image, matching a first ground truth bounding box of the first target with the multiple predicted bounding boxes to obtain a first predicted bounding box corresponding to the first ground truth bounding box; the first target is a target without a bound label; calculating a loss value of the first ground truth bounding box and the first predicted bounding box corresponding thereto, and binding the first target and the first predicted bounding box with a label; for a second target in each image, calculating a loss value of a second ground truth bounding box of the second target and a second predicted bounding box corresponding thereto; the second target is a target with a bound label; performing back propagation according to the first loss value and / or the second loss value to train the target detection model.
2. The method of claim 1, wherein, The method further comprises: for a third target, obtaining a number of consecutive image frames in which the third target does not appear; the third target is a target with a bound label but does not appear in a current image frame; if the number of image frames is greater than a threshold value, unbinding the third target and the label.
3. The method of claim 1, wherein, The inputting each image into a target detection model to perform target detection comprises: performing feature extraction on a camera image to obtain image features of the camera image; performing feature extraction on a radar point cloud to obtain radar features of the radar point cloud; performing feature fusion on the image features and the radar features to generate multi-modal features, and inputting the multi-modal features into the target detection model to perform target detection.
4. The method of claim 3, wherein, The target detection model comprises a neural network based on a self-attention mechanism, and the inputting the multi-modal features into the target detection model to perform target detection comprises: obtaining image coordinate encoding and point cloud coordinate encoding; adding the image coordinate encoding and the point cloud coordinate encoding to the image features and the radar features respectively to obtain multi-modal features after position encoding; inputting the multi-modal features after position encoding into the target detection model to perform target detection.
5. The method of claim 1, wherein, The performing back propagation according to the first loss value and / or the second loss value to train the target detection model comprises: in each parameter adjustment process, performing back propagation according to the first loss value and / or the second loss value corresponding to one image in the multiple images to adjust parameters of the target detection model.
6. A target tracking method characterized by, The method comprises: obtaining a to-be-detected image sequence, the to-be-detected image sequence comprising multiple to-be-detected images; inputting the to-be-detected image sequence into a target detection model to perform target detection, obtaining a target detection result in each to-be-detected image, and obtaining a label of each target detection result; the target detection model is trained by the method of claim 1; determining target detection results with the same label in different to-be-detected images as associated target detection results. 7.A device for training a target detection model, characterized in that, The method comprises: an obtaining module configured to obtain a training set, the training set comprising multiple image frames, and each image being labeled with a ground truth bounding box of each target; a prediction module, configured to input each frame of image into a target detection model to perform target detection, and obtain a plurality of prediction boxes of the each frame of image; a matching module, configured to, for a first target in the each frame of image, match a first detection box ground truth of the first target with the plurality of prediction boxes, and obtain a first prediction box corresponding to the first detection box ground truth; the first target is a target without a bound label; a first loss calculation module, configured to calculate a loss of the first detection box ground truth and the first prediction box corresponding thereto, and obtain a first loss value, and bind a label for the first target and the first prediction box; a second loss calculation module, configured to, for a second target in the each frame of image, calculate a loss of a second detection box ground truth of the second target and a second prediction box corresponding thereto, and obtain a second loss value; the second target is a target with a bound label; a training module, configured to perform back propagation according to the first loss value and / or the second loss value, to train the target detection model.
8. A target tracking device, characterized by, comprising: an acquisition module, configured to acquire a to-be-detected image sequence, the to-be-detected image sequence comprising a plurality of to-be-detected images; a detection module, configured to input the to-be-detected image sequence into a target detection model to perform target detection, and obtain a target detection result in each to-be-detected image, and obtain a label of each target detection result; the target detection model is trained by the method of claim 1; an association module, configured to determine target detection results with the same label in different to-be-detected images as associated target detection results.
9. An electronic device, comprising: comprising: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory, and execute the instructions to implement the method of any one of claims 1-6.
10. A computer-readable storage medium, characterized in that, The storage medium stores a computer program, and the computer program is executed by the processor to implement the method of any one of claims 1-6.
11. A vehicle characterized by comprising: comprising: the apparatus of claim 8, or the electronic device of claim 9.