A multi-target tracking method, device, vehicle and storage medium
By fusing convolutional kernels of different sizes to construct a multi-target tracking model with large convolutional kernels, and combining a backbone network and modules, the transfer learning technique is used to solve the problems of efficiency and insufficient receptive field of existing models, thus achieving efficient and accurate multi-target tracking.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- CHONGQING CHANGAN TECH CO LTD
- Filing Date
- 2023-07-27
- Publication Date
- 2026-06-09
AI Technical Summary
Existing multi-target tracking models suffer from high computational cost and large parameter count. Using large convolutional kernels results in low efficiency, while using small convolutional kernels results in a small effective receptive field, making it impossible to balance model efficiency and effective receptive field.
A target convolutional kernel is adopted. By fusing two convolutional kernels of different sizes, a target convolutional kernel larger than the first size threshold is constructed. Combined with the backbone network, detection module, classification module and feature module, the model training is optimized using transfer learning technology, and the number of tracking targets is dynamically adjusted.
It improves the efficiency of multi-target tracking models, enhances the effective receptive field, reduces computational load and parameter count, quickly obtains accurate multi-target tracking results, and improves the execution efficiency of multi-target tracking tasks.
Smart Images
Figure CN116740145B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of target tracking technology, specifically to a multi-target tracking method, device, vehicle, and storage medium. Background Technology
[0002] Multi-object tracking refers to assigning an ID to each object in a video and obtaining the movement trajectory of each ID. As an important computer vision task, the development of multi-object tracking now relies heavily on convolutional neural networks.
[0003] With the development of deep learning, large convolutional kernels have gradually been phased out because they are inefficient (the number of parameters and computational cost of convolution is proportional to the square of the kernel size), and increasing the kernel size actually worsens accuracy. Using small convolutional kernels means that neural networks can be made deeper. ResNet solved the optimization problem of deep small convolutional kernels, theoretically resulting in a large maximum receptive field for the model. However, the actual effective depth of the ResNet residual network structure is not deep, so the effective receptive field is not large. As a result, models based on this architecture perform well on ImageNet, but perform poorly on downstream tasks such as detection and segmentation.
[0004] In summary, existing tracking models using large convolutional kernels suffer from high computational cost and a large number of parameters, resulting in poor model efficiency; while tracking models using small convolutional kernels suffer from a small effective receptive field, resulting in poor model performance.
[0005] Therefore, existing technologies still need to be improved and enhanced. Summary of the Invention
[0006] This application provides a multi-target tracking method, apparatus, vehicle, and storage medium to address the problems in related technologies where tracking models using large convolutional kernels are inefficient, tracking models using small convolutional kernels have small effective receptive fields, and existing tracking models cannot balance model efficiency and effective receptive field.
[0007] To achieve the above objectives, this application adopts the following technical solution:
[0008] A multi-target tracking method includes the following steps:
[0009] Obtain the image to be detected, wherein the image to be detected is a non-first frame image in a series of frame images;
[0010] The image to be detected is input into a trained multi-target tracking model to obtain the recognition information corresponding to several targets in the image to be detected. The backbone network of the multi-target tracking model includes a target convolution kernel. The size of the target convolution kernel is greater than or equal to a first size threshold, and the target convolution kernel is obtained by fusing two convolution kernels of different sizes.
[0011] Obtain identification information corresponding to several tracking targets, match the identification information of each target with the identification information of each tracking target, and determine the identity identifier corresponding to each target based on the matching result.
[0012] Based on the above technical means, the target convolution kernel used in the multi-target tracking model of this application embodiment is a large convolution kernel, which has the advantage of a larger effective receptive field. In addition, since the target convolution kernel is obtained by fusing two convolution kernels of different sizes, it can effectively improve the disadvantages of large convolution kernels in terms of computational load and parameter quantity, improve the model efficiency of the multi-target tracking model, and quickly obtain accurate multi-target tracking results.
[0013] Optionally, in one embodiment of this application, the size of one of the two convolutional kernels is greater than or equal to the first size threshold, and the size of the other convolutional kernel is less than or equal to the second size threshold, wherein the first size threshold is greater than the second size threshold.
[0014] Based on the above technical means, the embodiments of this application pre-set the size requirements of two convolution kernels, thereby ensuring that the target convolution kernel can simultaneously have the advantages of small convolution kernels (low computational cost and few parameters) and large convolution kernels (large effective receptive field).
[0015] Optionally, in one embodiment of this application, the backbone network includes two convolutional kernels during the training phase; the backbone network is used to output a feature image corresponding to each training sample based on each training sample, wherein outputting the feature image corresponding to each training sample based on each training sample includes:
[0016] Obtain the feature images output by the two convolutional kernels based on the training samples;
[0017] The feature images output by the two convolutional kernels are fused to obtain the feature image corresponding to the training sample.
[0018] Based on the above technical means, the embodiments of this application use two convolutional kernels of different sizes for training during the network training stage, which can effectively capture features of different scales, thereby obtaining better network training results.
[0019] Optionally, in one embodiment of this application, the method for obtaining the target convolutional kernel includes:
[0020] Obtain the two trained convolutional kernels;
[0021] Adjust the smallest convolutional kernel to the same size as the largest convolutional kernel;
[0022] The target convolutional kernel is obtained by adding two convolutional kernels of the same size together.
[0023] Based on the above technical means, the embodiments of this application fuse two trained convolutional kernels into one convolutional kernel during the inference stage, which can reduce the number of parameters of the convolutional kernel and thus effectively improve the inference speed.
[0024] Optionally, in one embodiment of this application, the identification information of each target includes the target's location information, category information, and feature information; the multi-target tracking model further includes a detection module, a classification module, and a feature module respectively connected to the backbone network;
[0025] The backbone network is used to output a target feature image based on the image to be detected;
[0026] The detection module is used to output the location information corresponding to each target based on the target feature image;
[0027] The classification module is used to output the category information corresponding to each target based on the target feature image;
[0028] The feature module is used to output the feature information corresponding to each target based on the target feature image.
[0029] Based on the above technical means, the embodiments of this application construct a complete multi-target tracking model through a backbone network, a detection module, a classification module, and a feature module, which can accurately identify each target in the image to be detected and improve the execution efficiency of multi-target tracking tasks.
[0030] Optionally, in one embodiment of this application, the training method of the multi-target tracking model includes:
[0031] Obtain the network parameters of the pre-trained backbone network and import the network parameters into the untrained multi-target tracking model;
[0032] The imported multi-target tracking model is then trained.
[0033] Based on the above technical means, the embodiments of this application use transfer learning technology, which can reduce the training time of multi-object tracking models and enable multi-object tracking models to quickly achieve optimal training results.
[0034] Optionally, in one embodiment of this application, determining the identity identifier corresponding to each of the targets based on the matching results includes:
[0035] For each of the aforementioned targets, if the target is successfully matched with any of the tracking targets, the identity identifier corresponding to the target is determined based on the successfully matched tracking targets;
[0036] If the target fails to match any of the tracking targets, a new tracking target and a new identity identifier are determined based on the target.
[0037] Based on the above technical means, the embodiments of this application will determine the identity of each target according to the matching relationship between each target in the image to be detected and each tracking target, and dynamically adjust the number of tracking targets, which can better realize multi-target tracking.
[0038] A second aspect of this application provides a multi-target tracking device, comprising:
[0039] An acquisition module is used to acquire an image to be detected, wherein the image to be detected is a non-first frame image in a series of frame images;
[0040] The input module is used to input the image to be detected into a trained multi-target tracking model to obtain the recognition information corresponding to several targets in the image to be detected. The backbone network of the multi-target tracking model includes a target convolution kernel. The size of the target convolution kernel is greater than or equal to a first size threshold, and the target convolution kernel is obtained by fusing two convolution kernels with different sizes.
[0041] The matching module is used to obtain the identification information corresponding to several tracking targets, match the identification information of each target with the identification information of each tracking target, and determine the identity identifier corresponding to each target based on the matching result.
[0042] A third aspect of this application provides a vehicle, the vehicle including a memory, a processor, and a multi-target tracking program stored in the memory and executable on the processor, wherein when the processor executes the multi-target tracking program, it implements the steps of the multi-target tracking method as described in any of the preceding claims.
[0043] A fourth aspect of this application provides a computer-readable storage medium storing a multi-target tracking program, which, when executed by a processor, implements the steps of the multi-target tracking method as described in any of the preceding claims.
[0044] The beneficial effects of this application are:
[0045] The multi-object tracking model in this application uses a large convolutional kernel for the target, which has the advantage of a larger effective receptive field. In addition, since the target convolutional kernel is obtained by fusing two convolutional kernels of different sizes, it can effectively improve the disadvantages of large convolutional kernels in terms of computational cost and number of parameters, improve the model efficiency of the multi-object tracking model, and quickly obtain accurate multi-object tracking results.
[0046] This application uses two convolutional kernels of different sizes during the network training phase, which can effectively capture features at different scales, thus achieving better network training results. During the inference phase, the two trained convolutional kernels are merged into one, reducing the number of parameters and effectively improving inference speed.
[0047] This application constructs a complete multi-object tracking model through a backbone network, detection module, classification module, and feature module, and uses transfer learning technology to reduce the training time of the multi-object tracking model, enabling the multi-object tracking model to quickly achieve the optimal training effect and better perform multi-object tracking tasks.
[0048] This application determines the identity of each target based on the matching relationship between each target in the image to be detected and each tracked target, and dynamically adjusts the number of tracked targets, which can better achieve multi-target tracking.
[0049] Additional aspects and advantages of this application will be set forth in part in the description which follows, and in part will be obvious from the description, or may be learned by practice of this application. Attached Figure Description
[0050] The above and / or additional aspects and advantages of this application will become apparent and readily understood from the following description of the embodiments taken in conjunction with the accompanying drawings, wherein:
[0051] Figure 1 This is a flowchart illustrating the multi-target tracking method according to an embodiment of this application;
[0052] Figure 2 This is a schematic diagram of a large convolutional kernel according to an embodiment of this application;
[0053] Figure 3 This is a schematic diagram of the identity shortcut in an embodiment of this application;
[0054] Figure 4 This is a schematic diagram illustrating the process of using transfer learning techniques in the training of a multi-target tracking model according to an embodiment of this application.
[0055] Figure 5 This is a schematic diagram of the structure of a multi-target tracking device according to an embodiment of this application;
[0056] Figure 6This is a block diagram illustrating the internal structure of a vehicle as provided in an embodiment of this application. Detailed Implementation
[0057] The embodiments of this application are described in detail below. Examples of the embodiments are shown in the accompanying drawings, wherein the same or similar reference numerals denote the same or similar elements or elements having the same or similar functions throughout. The embodiments described below with reference to the accompanying drawings are exemplary and intended to explain this application, and should not be construed as limiting this application.
[0058] The following description, with reference to the accompanying drawings, illustrates a multi-target tracking method, apparatus, vehicle, and storage medium according to embodiments of this application. Addressing the issues mentioned in the background section regarding the low efficiency of tracking models using large convolutional kernels and the small effective receptive field of tracking models using small convolutional kernels, where existing tracking models cannot simultaneously achieve both model efficiency and effective receptive field, this application provides a multi-target tracking method. In this method, a target image is acquired, wherein the target image is a non-first frame image in a series of frames; the target image is input into a trained multi-target tracking model to obtain identification information corresponding to several targets in the target image; the backbone network of the multi-target tracking model includes target convolutional kernels, the size of which is greater than or equal to a first size threshold, and the target convolutional kernel is obtained by fusing two convolutional kernels of different sizes; the identification information corresponding to several tracking targets is acquired; the identification information of each target is matched with the identification information of each tracking target; and the identity identifier corresponding to each target is determined based on the matching result. The multi-target tracking model of this invention uses a large target convolutional kernel, which has the advantage of a larger effective receptive field. Furthermore, since the target convolution kernel is obtained by fusing two convolution kernels of different sizes, it can effectively improve the shortcomings of large convolution kernels in terms of computational cost and large number of parameters, improve the model efficiency of multi-target tracking models, and quickly obtain accurate multi-target tracking results.
[0059] For example, pre-trained small convolutional kernels (5x5) and ultra-large convolutional kernels (31x31) are fused to obtain target convolutional kernels. These target convolutional kernels form the backbone network, and a multi-target tracking model is built upon this backbone network. In practical applications, assuming multi-target tracking is to be performed on video data captured by a camera, the tracking targets A, B, and C are pre-determined based on the first frame of the video data, and the corresponding identification information for each target A, B, and C is determined. The identification information may include the target's location information, category information, and one or more of various feature information. Each video frame other than the first frame can be used as the image to be detected. The image to be detected is input into the pre-trained multi-target tracking model, and the multi-target tracking model outputs the targets a, b, and c in the image to be detected, along with their corresponding identification information. Taking target a as an example, the identification information of target a is matched with the identification of tracking targets A, B, and C respectively. Based on the matching results, tracking target A is determined to be the matching object of target a. Then, the identity of target a is determined based on tracking target A, thereby realizing the tracking of tracking target A.
[0060] Specifically, Figure 1 This is a flowchart illustrating a multi-target tracking method provided in an embodiment of this application.
[0061] like Figure 1 As shown, this multi-target tracking method includes the following steps:
[0062] Step S100: Obtain the image to be detected, wherein the image to be detected is a non-first frame image in a series of frame images.
[0063] Specifically, multi-target tracking refers to identifying and tracking multiple targets across a series of consecutive image frames, and can be applied to fields such as security monitoring and autonomous driving. In practical applications, this embodiment first requires acquiring consecutive image frames to perform the multi-target tracking task, such as video footage from a camera in a public place, video footage from a vehicle camera used in autonomous driving, or video footage from a camera on a navigation robot. Any frame in the consecutive image frames, except for the first frame, can be used as the image to be detected and analyzed to determine the motion trajectory of each tracked target.
[0064] Step S200: Input the image to be detected into the trained multi-target tracking model to obtain the recognition information corresponding to several targets in the image to be detected. The backbone network of the multi-target tracking model includes target convolution kernels. The size of the target convolution kernels is greater than or equal to a first size threshold, and the target convolution kernels are obtained by fusing two convolution kernels of different sizes.
[0065] Specifically, this embodiment pre-constructs a multi-object tracking model for performing multi-object tracking tasks. This model is pre-trained and has learned complex input-output mapping relationships. The multi-object tracking model consists of a backbone network. The backbone network includes target convolutional kernels for feature extraction. These target convolutional kernels differ from traditional convolutional kernels in that their size is greater than or equal to a first size threshold, classifying them as large convolutional kernels. Furthermore, the target convolutional kernel is obtained by fusing two convolutional kernels of different sizes. Therefore, the target convolutional kernel can simultaneously possess the advantages of a larger effective receptive field and less computation and parameters, thereby improving the performance of the multi-object tracking model and obtaining more accurate target tracking results. In practical applications, after inputting the image to be detected into the trained multi-object tracking model, the model can identify multiple targets in the image and their identification information, such as location information, category information, and various feature information.
[0066] In one embodiment, one of the two convolutional kernels has a size greater than or equal to a first size threshold, and the other convolutional kernel has a size less than or equal to a second size threshold, wherein the first size threshold is greater than the second size threshold.
[0067] Specifically, this embodiment pre-sets the size requirements of the two convolution kernels. The smaller convolution kernel needs to be below the first size threshold, and the larger convolution kernel needs to be above the second size threshold, thereby ensuring that the target convolution kernel can simultaneously have the advantages of small convolution kernel in terms of computational cost and number of parameters, and the advantages of large convolution kernel in terms of large effective receptive field.
[0068] For example, the larger of the two convolutional kernels can be a very large convolutional kernel, such as a depth-wise large convolutional kernel with a size of 31x31. Depth-wise convolution refers to convolution through each channel, meaning one kernel is responsible for one channel, and each channel is convolved by only one kernel. The smaller convolutional kernel can be a small convolutional kernel with a size of 3x3 or 5x5.
[0069] In one embodiment, the backbone network structure further includes residual blocks to increase network depth. Specifically, the structure of the residual block (identity shortcut) is as follows: Figure 2 As shown, it can achieve dimensionality consistency before and after data processing (64 dimensions in the figure). Therefore, when using large convolutional kernels, residual blocks help increase the number of points.
[0070] In one embodiment, the backbone network includes two convolutional kernels during the training phase; the backbone network is used to output a feature image corresponding to each training sample, and the output of the feature image corresponding to each training sample includes:
[0071] Obtain the feature images output by the two convolutional kernels based on the training samples;
[0072] The feature images output by the two convolutional kernels are fused to obtain the feature images corresponding to the training samples.
[0073] In summary, during the network training phase, two convolutional kernels of different sizes in the backbone network are not fused; each can extract features independently. Only after the backbone network is trained are the two convolutional kernels fused into a single kernel, resulting in the target convolutional kernel. When training samples are input into the backbone network, each sample is fed into one of the two convolutional kernels. Each kernel extracts features from the training sample and outputs a feature image. Finally, the fused image of the two feature images is used as the feature image of the training sample. For example, as shown... Figure 3 As shown, during training, the small convolutional kernel (5x5) and the super-large convolutional kernel (31x31) use the same input data, and their output data are merged. This embodiment uses two convolutional kernels of different sizes during the network training phase, which can effectively capture features at different scales, thereby obtaining better network training results.
[0074] In one embodiment, the method for obtaining the target convolutional kernel includes:
[0075] Obtain the two trained convolutional kernels;
[0076] Adjust the smallest convolutional kernel to the same size as the largest convolutional kernel;
[0077] The target convolutional kernel is obtained by adding two convolutional kernels of the same size.
[0078] Specifically, when the backbone network training is complete, the two convolutional kernels of different sizes are also trained. At this point, the size of the smaller convolutional kernel is adjusted according to the size of the larger kernel, so that the two convolutional kernels are the same size. Then, the two convolutional kernels of the same size are added together to obtain a single convolutional kernel, namely the target convolutional kernel. In this embodiment, the two trained convolutional kernels are fused into a target convolutional kernel during the inference phase, which can effectively reduce the number of parameters of the convolutional kernel, thereby effectively improving the inference speed.
[0079] For example, obtain a trained small convolutional kernel (3x3 or 5x5) and a very large convolutional kernel (31x31). Pad the small convolutional kernel with zeros at the edges according to the size of the very large convolutional kernel so that it has the same size as the very large convolutional kernel. Then add the two convolutional kernels of the same size to obtain the target convolutional kernel (31x31).
[0080] In one embodiment, the identification information of each target includes the target's location information, category information, and feature information; the multi-target tracking model also includes a detection module, a classification module, and a feature module, which are respectively connected to the backbone network.
[0081] The backbone network is used to output a target feature image based on the image to be detected;
[0082] The detection module is used to output the location information corresponding to each target based on the target feature image;
[0083] The classification module is used to output the category information corresponding to each target based on the target feature image;
[0084] The feature module is used to output the feature information corresponding to each target based on the target feature image.
[0085] Specifically, based on the established backbone network, a detection head, a classification head, and a feature module are connected at the output of the backbone network through different convolutional layers, thereby constructing a multi-object tracking model. The detection module outputs the location information of each object in the image to be detected; for example, a bounding box can be used to represent the object's location. The classification module outputs the category of each object in the image to be detected; for example, an identification number (ID) can be used to represent the object's category. The feature module outputs the feature information of each object; for example, a ReID feature module can be used to represent the multi-dimensional features of the object. This embodiment constructs a complete multi-object tracking model through the backbone network, detection module, classification module, and feature module, which can accurately identify each object in the image to be detected, improving the execution efficiency of multi-object tracking tasks.
[0086] In one embodiment, the training method for the multi-target tracking model includes:
[0087] Obtain the network parameters of the pre-trained backbone network and import the network parameters into the untrained multi-object tracking model;
[0088] Train the imported multi-target tracking model.
[0089] Specifically, such as Figure 4 As shown, this embodiment uses transfer learning technology. First, the backbone network is pre-trained, and then the network parameters obtained after pre-training are imported into the untrained multi-object tracking model. Then, the imported multi-object tracking model is trained, thereby reducing the training time of the multi-object tracking model and enabling the multi-object tracking model to quickly achieve the optimal training effect.
[0090] For example, the backbone network can be pre-trained using the ImageNet dataset, and then the trained weights can be imported into a multi-object tracking model. Then, pre-prepared multi-object tracking data can be obtained, labeled, and the multi-object tracking model can be trained using both the original and labeled data.
[0091] In one embodiment, during the training phase of the multi-object tracking model, the loss function value of the multi-object tracking model is determined based on the loss function values corresponding to the detection module, classification module, and feature module, respectively.
[0092] Specifically, the labeled data for multi-object tracking can include the target's bounding box, label category, and ID number. The loss function value for the detection module is determined by the bounding box; the loss function value for the classification module is determined by the label category; and since the features extracted by the multi-object tracking network from targets with the same ID number should be similar, the loss function value for the feature module can be determined by the ID number. Finally, the loss function values corresponding to the three modules are combined to determine the overall loss function value of the multi-object tracking model, achieving better model training results.
[0093] Step S300: Obtain the identification information corresponding to several tracking targets respectively, match the identification information of each target with the identification information of each tracking target, and determine the identity identifier corresponding to each target according to the matching result.
[0094] Specifically, for each target in the image to be detected, the identification information of the target is matched with the identification information of each tracked target. If the match is successful, the identity of the target can be determined based on the successfully matched tracked target, thereby inferring the motion trajectory of the successfully matched tracked target and realizing target tracking.
[0095] In one embodiment, obtaining the identification information corresponding to several tracking targets includes:
[0096] The first frame image in a series of images is obtained and input into a multi-target tracking model to obtain several tracking targets and their corresponding recognition information.
[0097] Specifically, in practical applications, each tracked target and its corresponding identification information are obtained by inputting the first frame image into a trained multi-target tracking model. Then, an ID number is assigned to each tracked target, and a target pool can be constructed based on the ID numbers of each target for performing multi-target tracking tasks.
[0098] In one embodiment, the target identification information includes a predicted bounding box and ReID features, the tracking target identification information includes a labeled bounding box and labeled ReID features, and the method for obtaining the matching result of each target with each tracked target includes:
[0099] Obtain the intersection-union ratio (IoU) between the predicted bounding box of the target and the labeled bounding box of each tracked target;
[0100] Obtain the cosine distance between the ReID features of the target and the labeled ReID features of each tracked target;
[0101] Based on the intersection-union ratio and cosine distance of each tracked target, the matching result between the target and each tracked target is determined.
[0102] Specifically, for any target in the image to be detected, the predicted bounding box of the target is matched with the labeled bounding box of the tracked target using Interchange of Union (IOU), and the ReID features of the target are matched with the labeled ReID features of the tracked target using cosine distance. The predicted bounding box can be obtained using Kalman filtering, and the matching algorithm can be the Hungarian algorithm. This cascaded matching method can accurately obtain the matching results between the target and each tracked target.
[0103] In one embodiment, determining the identity identifier corresponding to each target based on the matching result includes:
[0104] For each target, if the target is successfully matched with any other tracked target, the identity identifier corresponding to the target is determined based on the successfully matched tracked target;
[0105] If the target fails to match any of the tracking targets, a new tracking target and a new identity identifier are determined based on the target.
[0106] Specifically, for a target that is successfully matched in the image to be detected, its identity is determined based on the tracking target it is matched with; for a target that is not successfully matched in the image to be detected, it is treated as a newly appearing target, given a new identity, and then added to the tracking target pool. This embodiment determines the identity of each target based on the matching relationship between each target in the image to be detected and each tracking target, and dynamically adjusts the number of tracking targets to better achieve multi-target tracking.
[0107] In one embodiment, the multi-target tracking method further includes:
[0108] For each tracking target, if the tracking target fails to match any of the other targets, the tracking target is removed.
[0109] Specifically, for tracking targets in the tracking target pool that do not match any targets in the image to be detected, this embodiment considers the tracking target as lost and removes it from the tracking target pool to avoid subsequent invalid matching and wasting system resources.
[0110] In summary, the multi-object tracking model in this application uses a large convolutional kernel, which has the advantage of a larger effective receptive field. Furthermore, since the target convolutional kernel is obtained by fusing two convolutional kernels of different sizes, it effectively mitigates the drawbacks of large convolutional kernels, such as high computational cost and large number of parameters, thereby improving the model efficiency of the multi-object tracking model and quickly obtaining accurate multi-object tracking results. Secondly, this application uses two convolutional kernels of different sizes for training during the network training phase, which can effectively capture features at different scales, resulting in better network training performance. During the inference phase, fusing the two trained convolutional kernels into one reduces the number of parameters, thus effectively improving inference speed. In addition, this application constructs a complete multi-object tracking model through a backbone network, detection module, classification module, and feature module, and uses transfer learning techniques to reduce the training time of the multi-object tracking model, enabling it to quickly reach optimal training results and better perform multi-object tracking tasks. Furthermore, this application determines the identity of each target based on the matching relationship between each target in the image to be detected and each tracked target, and dynamically adjusts the number of tracked targets, thus achieving better multi-object tracking.
[0111] Next, a multi-target tracking device according to an embodiment of this application is described with reference to the accompanying drawings.
[0112] like Figure 5 As shown, the multi-target tracking device 10 includes: an acquisition module 100, an input module 200, and a matching module 300.
[0113] Specifically, the acquisition module 100 is used to acquire the image to be detected, wherein the image to be detected is a non-first frame image in a series of frame images;
[0114] The input module 200 is used to input the image to be detected into a trained multi-target tracking model to obtain the recognition information corresponding to several targets in the image to be detected. The backbone network of the multi-target tracking model includes target convolution kernels. The size of the target convolution kernels is greater than or equal to a first size threshold, and the target convolution kernels are obtained by fusing two convolution kernels of different sizes.
[0115] The matching module 300 is used to acquire the identification information corresponding to several tracking targets, match the identification information of each target with the identification information of each tracking target, and determine the identity identifier corresponding to each target based on the matching result.
[0116] It should be noted that the foregoing explanation of the multi-target tracking method embodiment also applies to the multi-target tracking device of this embodiment, and will not be repeated here.
[0117] Figure 6 A schematic diagram of the structure of a vehicle provided in an embodiment of this application. The vehicle may include:
[0118] The memory 601, the processor 602, and the computer program stored on the memory 601 and capable of running on the processor 602.
[0119] When the processor 602 executes the program, it implements the multi-target tracking method provided in the above embodiments.
[0120] Furthermore, the vehicle also includes:
[0121] Communication interface 603 is used for communication between memory 601 and processor 602.
[0122] The memory 601 is used to store computer programs that can run on the processor 602.
[0123] The memory 601 may include high-speed RAM memory, and may also include non-volatile memory, such as at least one disk storage device.
[0124] If the memory 601, processor 602, and communication interface 603 are implemented independently, then the communication interface 603, memory 601, and processor 602 can be interconnected via a bus to complete communication between them. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into address bus, data bus, control bus, etc. For ease of representation, Figure 6 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.
[0125] Optionally, in a specific implementation, if the memory 601, processor 602, and communication interface 603 are integrated on a single chip, then the memory 601, processor 602, and communication interface 603 can communicate with each other through an internal interface.
[0126] The processor 602 may be a central processing unit (CPU), an application specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of this application.
[0127] This embodiment also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the multi-target tracking method described above.
[0128] In the description of this specification, the references to terms such as "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., indicate that a specific feature, structure, material, or characteristic described in connection with that embodiment or example is included in at least one embodiment or example of this application. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described may be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification, as well as the features of different embodiments or examples.
[0129] Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include at least one of that feature. In the description of this application, "N" means at least two, such as two, three, etc., unless otherwise explicitly specified.
[0130] Any process or method description in the flowchart or otherwise herein can be understood as representing a module, segment, or portion of code comprising one or N executable instructions for implementing custom logic functions or processes, and the scope of the preferred embodiments of this application includes additional implementations in which functions may be performed not in the order shown or discussed, including substantially simultaneously or in reverse order depending on the functions involved, as should be understood by those skilled in the art to which embodiments of this application pertain.
[0131] The logic and / or steps represented in the flowchart or otherwise described herein, for example, can be considered as a ordered list of executable instructions for implementing logical functions, and can be embodied in any computer-readable medium for use by, or in conjunction with, an instruction execution system, apparatus, or device (such as a computer-based system, a processor-included system, or other system that can read and execute instructions from or in conjunction with such an instruction execution system, apparatus, or device). For the purposes of this specification, "computer-readable medium" can be any means that can contain, store, communicate, propagate, or transmit programs for use by or in conjunction with an instruction execution system, apparatus, or device. More specific examples (a non-exhaustive list) of computer-readable media include: an electrical connection having one or more wires (electronic device), a portable computer disk drive (magnetic device), random access memory (RAM), read-only memory (ROM), erasable and editable read-only memory (EPROM or flash memory), fiber optic devices, and portable optical disc read-only memory (CDROM). In addition, computer-readable media can even be paper or other suitable media on which programs can be printed, because programs can be obtained electronically by optically scanning paper or other media, then editing, interpreting or otherwise processing them as necessary, and then storing them in computer memory.
[0132] It should be understood that the various parts of this application can be implemented using hardware, software, firmware, or a combination thereof. In the above embodiments, the N steps or methods can be implemented using software or firmware stored in memory and executed by a suitable instruction execution system. If implemented in hardware, as in another embodiment, it can be implemented using any one or a combination of the following techniques known in the art: discrete logic circuits having logic gates for implementing logical functions on data signals, application-specific integrated circuits (ASICs) having suitable combinational logic gates, programmable gate arrays (PGAs), field-programmable gate arrays (FPGAs), etc.
[0133] Those skilled in the art will understand that all or part of the steps of the methods described in the above embodiments can be implemented by a program instructing related hardware, and the program can be stored in a computer-readable storage medium. When executed, the program includes one or a combination of the steps of the method embodiments.
[0134] Furthermore, the functional units in the various embodiments of this application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into a module. The integrated module can be implemented in hardware or as a software functional module. If the integrated module is implemented as a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium.
[0135] The storage medium mentioned above can be a read-only memory, a disk, or an optical disk, etc. Although embodiments of this application have been shown and described above, it is understood that the above embodiments are exemplary and should not be construed as limiting this application. Those skilled in the art can make changes, modifications, substitutions, and variations to the above embodiments within the scope of this application.
Claims
1. A multi-target tracking method, characterized in that, Includes the following steps: Obtain the image to be detected, wherein the image to be detected is a non-first frame image in a series of frame images; The image to be detected is input into a trained multi-target tracking model to obtain recognition information corresponding to several targets in the image to be detected. The backbone network of the multi-target tracking model includes a target convolutional kernel, the size of which is greater than or equal to a first size threshold, and the target convolutional kernel is obtained by fusing two convolutional kernels of different sizes. The backbone network includes two convolutional kernels during the training phase. The backbone network is used to output a feature image corresponding to each training sample based on that training sample. Outputting the feature image corresponding to each training sample includes: acquiring the feature images output by the two convolutional kernels based on the training sample; fusing the feature images output by the two convolutional kernels to obtain the feature image corresponding to the training sample; the method for acquiring the target convolutional kernel includes: acquiring two trained convolutional kernels; adjusting the smallest convolutional kernel to the same size as the largest convolutional kernel; and adding the two convolutional kernels of the same size to obtain the target convolutional kernel. A method is used to obtain identification information corresponding to several tracking targets, match the identification information of each target with the identification information of each tracking target, and determine the identity identifier corresponding to each target based on the matching result. The identification information of each target includes a predicted bounding box and ReID features, and the identification information of each tracking target includes a labeled bounding box and labeled ReID features. The method for obtaining the matching result between each target and each tracking target includes: obtaining the intersection-union ratio (IUGR) between the predicted bounding box of the target and the labeled bounding box of each tracking target; obtaining the cosine distance between the ReID features of the target and the labeled ReID features of each tracking target; and determining the matching result between the target and each tracking target based on the IUGR and cosine distance corresponding to each tracking target.
2. The multi-target tracking method as described in claim 1, characterized in that, One of the two convolutional kernels has a size greater than or equal to the first size threshold, and the other convolutional kernel has a size less than or equal to the second size threshold, wherein the first size threshold is greater than the second size threshold.
3. The multi-target tracking method as described in claim 1, characterized in that, The identification information for each target includes the target's location information, category information, and feature information; the multi-target tracking model also includes a detection module, a classification module, and a feature module, which are respectively connected to the backbone network. The backbone network is used to output a target feature image based on the image to be detected; The detection module is used to output the location information corresponding to each target based on the target feature image; The classification module is used to output the category information corresponding to each target based on the target feature image; The feature module is used to output the feature information corresponding to each target based on the target feature image.
4. The multi-target tracking method as described in claim 1 or 3, characterized in that, The training method for the multi-target tracking model includes: Obtain the network parameters of the pre-trained backbone network and import the network parameters into the untrained multi-target tracking model; The imported multi-target tracking model is then trained.
5. The multi-target tracking method as described in claim 1, characterized in that, The step of determining the identity identifier corresponding to each of the targets based on the matching results includes: For each of the aforementioned targets, if the target is successfully matched with any of the tracking targets, the identity identifier corresponding to the target is determined based on the successfully matched tracking targets; If the target fails to match any of the tracking targets, a new tracking target and a new identity identifier are determined based on the target.
6. A multi-target tracking device, characterized in that, include: An acquisition module is used to acquire an image to be detected, wherein the image to be detected is a non-first frame image in a series of frame images; An input module is used to input the image to be detected into a trained multi-target tracking model to obtain recognition information corresponding to several targets in the image to be detected. The backbone network of the multi-target tracking model includes a target convolutional kernel, the size of which is greater than or equal to a first size threshold, and the target convolutional kernel is obtained by fusing two convolutional kernels of different sizes. The backbone network includes two convolutional kernels during the training phase. The backbone network is used to output a feature image corresponding to each training sample based on each training sample. Outputting the feature image corresponding to each training sample includes: acquiring the feature images output by the two convolutional kernels based on the training sample; fusing the feature images output by the two convolutional kernels to obtain the feature image corresponding to the training sample. The method for acquiring the target convolutional kernel includes: acquiring two trained convolutional kernels; adjusting the smallest convolutional kernel to the same size as the largest convolutional kernel; and adding the two convolutional kernels of the same size to obtain the target convolutional kernel. A matching module is used to acquire identification information corresponding to several tracking targets, match the identification information of each target with the identification information of each tracking target, and determine the identity identifier corresponding to each target based on the matching result. The identification information of the target includes a predicted bounding box and ReID features, and the identification information of the tracking target includes a labeled bounding box and labeled ReID features. The method for acquiring the matching result between each target and each tracking target includes: acquiring the intersection-union ratio (IUGR) between the predicted bounding box of the target and the labeled bounding box of each tracking target; acquiring the cosine distance between the ReID features of the target and the labeled ReID features of each tracking target; and determining the matching result between the target and each tracking target based on the IUGR and cosine distance corresponding to each tracking target.
7. A vehicle, characterized in that, The vehicle includes a memory, a processor, and a multi-target tracking program stored in the memory and executable on the processor. When the processor executes the multi-target tracking program, it implements the steps of the multi-target tracking method as described in any one of claims 1-5.
8. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a multi-target tracking program, which, when executed by a processor, implements the steps of the multi-target tracking method as described in any one of claims 1-5.