Multi-target tracking method and device, equipment, storage medium and program product

Abstract

The invention discloses a multi-target tracking method and device, equipment, a storage medium and a program product, and relates to the technical field of artificial intelligence, and the method comprises the steps: carrying out the key point detection of an ith frame image in a video, so as to determine a plurality of first key point sets corresponding to the ith frame image; if the ith frame of image is not the first frame of image in the video, obtaining target motion information; the target motion information comprises at least one of the following two kinds of motion information: first motion information, related to space, of each target in a preset number of images before the ith frame of image, and second motion information, related to time, of each target in the ith frame of image; generating a plurality of second key point sets corresponding to the ith frame of image based on the target motion information and the random noise image; and according to a matching relationship between each first key point set and each second key point set, determining each target and position in the ith frame of image. According to the invention, the accuracy of multi-target tracking is improved.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multi-target tracking method, device, equipment, storage medium and program product. Background Art

[0002] Object tracking is a hot topic in the field of computer vision. Object tracking can be divided into single-target tracking and multi-target tracking based on the number of targets to be tracked. Single-target tracking can handle problems such as illumination, deformation, and occlusion by modeling the appearance or motion of the target, while multi-target tracking is much more complicated. In addition to the problems encountered in single-target tracking, other issues such as correlation matching between targets are also required.

[0003] Existing multi-target tracking solutions mainly include traditional feature solutions represented by kernel correlation filtering and deep learning solutions represented by neural networks. Among them, the multi-target tracking effect of deep learning solutions is far ahead of that of traditional feature solutions, but the multi-target tracking effect of deep learning solutions still needs to be further improved. Summary of the invention

[0004] In view of the above problems, the present application provides a multi-target tracking method, device, equipment, storage medium and program product to improve the accuracy of multi-target tracking. The specific scheme is as follows:

[0005] The first aspect of the present application provides a multi-target tracking method, comprising:

[0006] Performing key point detection on the i-th frame image in the video to determine a plurality of first key point sets corresponding to the i-th frame image; different first key point sets correspond to different targets in the i-th frame image; different key points in each first key point set represent different parts of the target corresponding to the first key point set;

[0007] If the i-th frame image is not the first frame image in the video, obtain target motion information; the target motion information includes at least one of the following two types of motion information: first motion information related to space of each target in a preset number of images before the i-th frame image, and second motion information related to time of each target in the i-th frame image;

[0008] Based on the target motion information and the random noise image, a plurality of second key point sets corresponding to the i-th frame image are generated; different second key point sets correspond to different historical tracking targets; different key points in each second key point set represent different parts of the historical tracking target corresponding to the second key point set;

[0009] According to the matching relationship between each first key point set and each second key point set, each target and position in the i-th frame image is determined.

[0010] In a possible implementation, obtaining first motion information related to space of each target in a preset number of images before the i-th frame image includes:

[0011] The preset number of images are processed based on a frame difference method to obtain first motion information related to each target in the preset number of images and the space.

[0012] In a possible implementation, obtaining the second motion information of each target in the i-th frame image related to time includes:

[0013] Based on the first position of each key point in the plurality of first key point sets in the i-th frame image, predict the second position of each key point in the plurality of first key point sets in the (i+1)th frame image;

[0014] Determine the movement speed and movement direction of each key point based on the first position and the second position of the key point;

[0015] The movement speed and movement direction of each key point corresponding to the same target are fused to obtain the second movement information related to time of each target.

[0016] In a possible implementation, generating a plurality of second key point sets corresponding to the i-th frame image based on the target motion information and the random noise image includes:

[0017] Using the target motion information as a control condition of a diffusion generation model, the random noise image is processed by the diffusion generation model to generate a plurality of second key point sets corresponding to the i-th frame image;

[0018] The diffusion generation model is obtained by conditional diffusion training using point set images corresponding to video frames as training samples, target motion information corresponding to video frames as control conditions, and randomly sampled noise images corresponding to video frames as labels.

[0019] In a possible implementation, the target motion information corresponding to the video frame includes first motion information related to space of each target in a preset number of video frames before the video frame, and second motion information related to time of each target in the video frame;

[0020] In the process of training the diffusion generation model, the first motion information or the second motion information in the target motion information corresponding to some video frames is deleted.

[0021] In a possible implementation, determining each target in the i-th frame image according to the matching relationship between each first key point set and each second key point set includes:

[0022] For each first key point set, if a second key point set matching the first key point set is found in the plurality of second key point sets, determining the target corresponding to the first key point set as the historical tracking target corresponding to the matched second key point set;

[0023] If no second key point set matching the first key point set is found in the plurality of second key point sets, it is determined that the target corresponding to the first key point set is a newly appeared target.

[0024] In a possible implementation, determining the position of each target in the i-th frame image according to the matching relationship between each first key point set and each second key point set includes:

[0025] For each first key point set, if a second key point set matching the first key point set is found in the plurality of second key point sets, weighted sum of the first key point set and the second key point set matching the first key point set is performed to obtain a fused key point set; coordinate information of a minimum circumscribed rectangle of the fused key point set is determined as the position of the historical tracking target corresponding to the matched second key point set in the i-th frame image;

[0026] If no second key point set matching the first key point set is found in the plurality of second key point sets, the coordinate information of the minimum circumscribed rectangle of the first key point set is determined as the position of the target corresponding to the first key point set in the i-th frame image.

[0027] In a possible implementation, the process of determining the matching relationship between any first key point set and any second key point set includes:

[0028] Using a graph neural network to process any of the first key point sets and any of the second key point sets to obtain a processing result, wherein the processing result indicates whether any of the first key point sets and any of the second key point sets match;

[0029] Alternatively, the area where the minimum circumscribed rectangle of any key point set in the point set image corresponding to any key point set is located is used as a template, and the point set images corresponding to the multiple second key point sets are traversed to search for the area matching the template, and the second key point set in the searched matching area is determined as the second key point set matching any key point set.

[0030] A second aspect of the present application provides a multi-target tracking device, comprising:

[0031] A detection module is used to perform key point detection on the i-th frame image in the video to determine a plurality of first key point sets corresponding to the i-th frame image; different first key point sets correspond to different targets in the i-th frame image; different key points in each first key point set represent different parts of the target corresponding to the first key point set;

[0032] a motion information acquisition module, configured to acquire target motion information if the i-th frame image is not the first frame image in the video; the target motion information includes at least one of the following two types of motion information: first motion information related to space of each target in a preset number of images before the i-th frame image, and second motion information related to time of each target in the i-th frame image;

[0033] A generating module, configured to generate a plurality of second key point sets corresponding to the i-th frame image based on the target motion information and the random noise image; different second key point sets correspond to different historical tracking targets; different key points in each second key point set represent different parts of the historical tracking target corresponding to the second key point set;

[0034] The target determination module is used to determine each target and position in the i-th frame image according to the matching relationship between each first key point set and each second key point set.

[0035] A third aspect of the present application provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the multi-target tracking method of the first aspect or any implementation of the first aspect.

[0036] A fourth aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0037] The memory is used to store computer programs;

[0038] The processor is used to execute the computer program so that the electronic device can implement the multi-target tracking method of the first aspect or any implementation of the first aspect.

[0039] A fifth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the multi-target tracking method of the first aspect or any implementation of the first aspect.

[0040] By means of the above technical scheme, the multi-target tracking method, device, equipment, storage medium and program product provided by the present application perform key point detection on the i-th frame image in the video to determine multiple first key point sets corresponding to the i-th frame image; different first key point sets correspond to different targets in the i-th frame image; different key points in each first key point set represent different parts of the target corresponding to the first key point set; if the i-th frame image is not the first frame image in the video, obtain target motion information; the target motion information includes at least one of the following two types of motion information: first motion information related to space of each target in a preset number of images before the i-th frame image, and second motion information related to time of each target in the i-th frame image; based on the target motion information and the random noise image, generate multiple second key point sets corresponding to the i-th frame image; different second key point sets correspond to different historical tracking targets; different key points in each second key point set represent different parts of the historical tracking target corresponding to the second key point set; according to the matching relationship between each first key point set and each second key point set, determine each target and position in the i-th frame image. The multi-target tracking method of the present application represents the target to be tracked in the form of a point set. In addition to detecting the key point set of the target from the i-th frame image, the key point set of the target in the i-th frame image is predicted based on the motion information of the target. Then, based on the matching relationship between the detected key point set and the predicted key point set, the target in the i-th frame image and its position are determined, thereby improving the accuracy of multi-target tracking. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] The above and other features, advantages and aspects of the embodiments of the present disclosure will become more apparent with reference to the following detailed description in conjunction with the accompanying drawings. Throughout the accompanying drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the originals and elements are not necessarily drawn to scale.

[0042] Figure 1 A flowchart of an implementation of the multi-target tracking method provided in this application;

[0043] Figure 2 An example diagram of a key point set provided for this application;

[0044] Figure 3 Another example diagram of a key point set provided for this application;

[0045] Figure 4 Another example diagram of a key point set provided for this application;

[0046] Figure 5 A flowchart for obtaining the second motion information of each target in the i-th frame image related to time provided by the present application;

[0047] Figure 6 An example diagram of the overall architecture of the multi-target tracking method provided in this application;

[0048] Figure 7 A schematic diagram of the structure of a multi-target tracking device provided by this application;

[0049] Figure 8 A schematic diagram of the structure of an electronic device provided in this application. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] In order to improve the accuracy of multi-target tracking, this application is proposed. This application relates to the field of artificial intelligence technology, specifically including computer vision, image processing, deep learning and other technical fields.

[0054] like Figure 1 As shown, a flowchart of an implementation of the multi-target tracking method provided in an embodiment of the present application may include:

[0055] Step S101: perform key point detection on the i-th frame image in the video to determine multiple (for ease of description, recorded as N) first key point sets corresponding to the i-th frame image; different first key point sets correspond to different targets in the i-th frame image; different key points in each first key point set represent different parts of the target corresponding to the first key point set.

[0056] Multi-Object Tracking (MOT) is an important task in the field of computer vision, which aims to identify and track the motion trajectories of multiple targets in a video.

[0057] The i-th (i is an integer greater than 0) frame image is any frame image in the video.

[0058] The target of tracking may be a natural person, an animal, or other objects, etc. For the convenience of description, the embodiment will be explained below by taking the tracking target as a natural person as an example.

[0059] The trained human key point prediction network can be used to process the i-th frame image to obtain multiple key point sets (referred to as the first key point sets), each of which corresponds to a target in the i-th frame image, and different key points belonging to the same key point set correspond to different parts of the same target. The number of key points in different key point sets is the same, that is, in the i-th frame image, regardless of whether the parts of the target corresponding to the key points are blocked, a preset number of key points are predicted.

[0060] The human key point prediction network may include but is not limited to any of the following network structures: DeepPose, StackedHourglass, CPN (Convolutional Pose Machines Network), HRNet (High-ResolutionNetwork), RTMPose, etc.

[0061] The human key point prediction network can be trained using images as training samples and using the human key point set annotated for the images as labels. The images used as training samples may contain only one human body or multiple human bodies.

[0062] like Figure 2 As shown, it is an example diagram of a key point set provided in an embodiment of the present application. In this example, each key point set includes 17 key points ( Figure 2 In the figure, the lines between the key points represent the connection relationship between the key points, which is fixed. Based on this example, the embodiment of the present application can detect 17 key points of each target in the i-th frame image. In other words, among the predicted multiple key point sets, each key point set includes 17 key points, regardless of whether the target corresponding to the key point set has an obscured part.

[0063] It should be noted that in the present application, the number of key points in the key point set is not limited to 17, and may also be other numbers, such as 33, which is not specifically limited in the present application. Figure 3As shown, another example diagram of a key point set provided in an embodiment of the present application is shown. In this example, each key point set includes 33 key points ( Figure 3 In the figure, the lines between the key points represent the connection relationship between the key points, which is fixed. Based on this example, the embodiment of the present application can detect 33 key points of each target in the i-th frame image. In other words, among the predicted multiple key point sets, each key point set includes 33 key points, regardless of whether the target corresponding to the key point set has an obscured part.

[0064] like Figure 4 As shown, it is another example diagram of a key point set provided by an embodiment of the present application. In this example, each key point set includes 25 key points ( Figure 4 In the figure, the lines between the key points represent the connection relationship between the key points, which is fixed. Based on this example, the embodiment of the present application can detect 25 key points of each target in the i-th frame image. In other words, among the predicted multiple key point sets, each key point set includes 25 key points, regardless of whether the target corresponding to the key point set has an obscured part.

[0065] If the i-th frame image is the first frame image in the video, the minimum bounding rectangle of each key point set can be obtained, and the coordinate information of the minimum bounding rectangle can be determined as the position of a target, and different identification information can be assigned to different targets to distinguish different targets. Further, the minimum bounding rectangles of different key point sets displayed in the i-th frame image can be marked with different colors so that the user can distinguish different targets.

[0066] Step S102: If the i-th frame image is not the first frame image in the video, obtain target motion information. The target motion information includes at least one of the following two types of motion information: first motion information related to space of each target in a preset number of images before the i-th frame image, and second motion information related to time of each target in the i-th frame image.

[0067] The first motion information only includes the motion information of the historical tracking target (i.e., the target tracked historically), and the first motion information reflects the motion of the historical tracking target in the spatial dimension; the second motion information may only include the motion information of the historical tracking target, and may also include the motion information of the newly appearing target. The second motion information at least reflects the motion of the historical tracking target in the time dimension, and may further reflect the motion of the newly appearing target in the time dimension.

[0068] Step S103: Based on the target motion information and the random noise image, generate multiple (for ease of description, recorded as M) second key point sets corresponding to the i-th frame image; different second key point sets correspond to different historical tracking targets; different key points in each second key point set represent different parts of the historical tracking target corresponding to the second key point set.

[0069] The value of each pixel in the random noise image is obtained by randomly sampling the Gaussian model. As an example, the Gaussian model can be a Gaussian mixture model or a single Gaussian model.

[0070] The target motion information and the random noise image can be processed by a diffusion generation model (hereinafter referred to as the diffusion model) to generate multiple second key point sets corresponding to the i-th frame image. Specifically, the target motion information can be used as a control condition and the random noise image can be input into the diffusion generation model together to obtain multiple second key point sets generated by the diffusion generation model.

[0071] The number M of the second key point sets generated in step S103 may be the same as or different from the number N of the first key point sets detected in step S101.

[0072] Step S104: determining each target and position in the i-th frame image according to the matching relationship between each first key point set and each second key point set.

[0073] For any first key point set and any second key point set, the matching relationship between the any first key point set and the any second key point set represents whether the any first key point set matches the any second key point set.

[0074] The multi-target tracking method provided in the embodiment of the present application represents the target to be tracked in the form of a point set. In addition to detecting the key point set of the target from the i-th frame image, the key point set of the target in the i-th frame image is predicted based on the motion information of the target. Then, based on the matching relationship between the detected key point set and the predicted key point set, the target in the i-th frame image and its position are determined, thereby improving the accuracy of multi-target tracking.

[0075] In an optional embodiment, an implementation method of obtaining the first motion information related to the space of each target in a preset number of images before the i-th frame image may be:

[0076] The preset number of images are processed based on a frame difference method to obtain first motion information related to each target in the preset number of images and the space.

[0077] Since the position of the video acquisition device used for target tracking is usually fixed, that is, the target background is unchanged, the motion information related to the target and space can be obtained by the frame difference method.

[0078] Optionally, the first motion information related to the space of each target in the two frames of images obtained by calculation before the i-th frame of image may be used.

[0079] As an example, the first motion information may be obtained by subtracting the i-1th frame image from the i-2th frame image.

[0080] As an example, the first point set image corresponding to the i-1th frame image and the second point set image corresponding to the i-2th frame image may be subtracted to obtain the first motion information. The first point set image is the same size as the i-1th frame image, and the first point set image is obtained by retaining the pixel points of the multiple first key point sets detected from the i-1th frame image, and setting the pixel points in the i-1th frame image that do not belong to the multiple first key point sets detected from the i-1th frame image to the preset grayscale values; similarly, the second point set image is the same size as the i-2th frame image, and the second point set image is obtained by retaining the pixel points of the multiple first key point sets detected from the i-2th frame image, and setting the pixel points in the i-2th frame image that do not belong to the multiple first key point sets detected from the i-2th frame image to the preset grayscale values.

[0081] Optionally, the first motion information related to the space of each target in the three frames of images may be calculated using the three frames of images before the i-th frame of image.

[0082] As an example, the i-1th frame image and the i-2th frame image can be subtracted to obtain a first difference image, and the i-2th frame image and the i-3th frame image can be subtracted to obtain a second difference image; the first difference image and the second difference image are subtracted to obtain first motion information.

[0083] As an example, the first point set image corresponding to the i-1 frame image and the second point set image corresponding to the i-2 frame image can be subtracted to obtain a first difference image, and the second point set image corresponding to the i-2 frame image and the third point set image corresponding to the i-3 frame image can be subtracted to obtain a second difference image; the first difference image and the second difference image are subtracted to obtain the first motion information. The third point set image has the same size as the i-3 frame image, and the third point set image is obtained by retaining the pixel points of the multiple first key point sets detected from the i-3 frame image, and setting the pixel points in the i-3 frame image that do not belong to the multiple first key point sets detected from the i-3 frame image to preset grayscale values.

[0084] In an optional embodiment, the second motion information of each target in the i-th frame image related to time can be obtained based on multiple first key point sets. Based on this, a flowchart for implementing the above-mentioned obtaining the second motion information of each target in the i-th frame image related to time is as follows: Figure 5 As shown, this may include:

[0085] Step S501: Based on the position of each key point in the multiple first key point sets in the i-th frame image (for the convenience of description and distinction, recorded as the first position), predict the position of each key point in the multiple first key point sets in the i+1-th frame image (for the convenience of description and distinction, recorded as the second position).

[0086] The second position of each key point in the multiple first key point sets in the i+1th frame image can be predicted based on the first position of each key point in the multiple first key point sets in the i-th frame image by Kalman filtering. The specific implementation method can refer to the existing solution, which will not be repeated here.

[0087] Step S502: Determine the movement speed and movement direction of each key point based on the first position and the second position of the key point.

[0088] For the jth (j=1, 2, 3, ..., J; J is the total number of key points in a key point set) key point P in the kth key point set kj , according to the key point P kj The distance between the first position and the second position, as well as the acquisition time difference of two adjacent frames of the video, calculate the key point P kj The speed of movement. kj The first position points to the key point P kj The vector of the second position is determined as the key point P kj direction of movement.

[0089] Step S503: The movement speed and movement direction of each key point corresponding to the same target are integrated to obtain the second movement information related to time of each target.

[0090] Optionally, the key point set of the same target can be grouped, the moving speeds of the key points belonging to the same group can be averaged to obtain the average speed of each group, and the average speeds of each group of the same target can be spliced ​​into a vector to obtain the moving speed of each target; the moving directions of the key points belonging to the same group can be summed, the moving directions of each group can be spliced ​​into a vector to obtain the moving direction of each target. The moving speed and moving direction of the same target can be spliced ​​into a vector to obtain the second moving information of each target.

[0091] Optionally, the key point set of the same target can be grouped, and the moving speeds of the key points belonging to the same group can be averaged to obtain the average speed of each group, and the moving directions of the key points belonging to the same group can be summed to obtain the moving direction of each target. The moving speed and moving direction of the same target can be spliced ​​into a vector to obtain the second moving information of each target.

[0092] As an example, Figure 2 Taking the 17 key points shown in FIG. 1 as an example, the 17 key points can be divided into 5 groups according to the head, left upper body, right upper body, left lower body and right lower body. Among them, the head corresponds to point numbers 1, 2, 3, and 4, and the 4 key points are grouped into one group; the left upper body corresponds to point numbers 5, 7, and 9, and the 3 key points are grouped into one group; the right upper body corresponds to point numbers 6, 8, and 10, and the 3 key points are grouped into one group; the left lower body corresponds to point numbers 11, 13, and 15, and the 3 key points are grouped into one group; the right lower body corresponds to point numbers 12, 14, and 16, and the 3 key points are grouped into one group.

[0093] In an optional embodiment, an implementation method of generating a plurality of second key point sets corresponding to the i-th frame image based on the target motion information and the random noise image may be:

[0094] The target motion information is used as the control condition of the diffusion generation model, and the random noise image is processed by the diffusion generation model to generate multiple second key point sets corresponding to the i-th frame image.

[0095] The diffusion generation model is obtained by conditional diffusion training with point set images corresponding to video frames as training samples, target motion information corresponding to video frames as control conditions, and randomly sampled noise images corresponding to video frames as labels.

[0096] The point set image corresponding to the video frame includes the key point set of each target in the video frame. The values ​​of the pixels belonging to the key points in the point set image are the same as the values ​​of the pixels belonging to the key points in the video frame, and the values ​​of the pixels belonging to the non-key points in the point set image are preset gray values, and the values ​​of different pixels belonging to the non-key points are the same.

[0097] Optionally, for any training sample, a time (denoted as t) can be randomly selected from the time series (1, 2, 3, ..., T).

[0098] A noisy image is obtained by random sampling from the Gaussian model, and the noisy image and the training sample are summed to obtain a noisy training sample.

[0099] The target motion information corresponding to the video frame is encoded through the first encoding module of the diffusion generation model to obtain a first encoding feature.

[0100] The position encoding module of the diffusion generation model performs sinusoidal position encoding on the time t to obtain the position encoding feature; the position encoding feature and the noisy training sample are fused to obtain the target training sample at time t; the target training sample is encoded by the second encoding module to obtain the second encoding feature; the first encoding feature and the second encoding feature are fused to obtain the fused encoding feature; the fused encoding feature is decoded by the decoding module to obtain the predicted noise image, and the parameters of the diffusion generation model are updated with the goal of predicting the noise image to be close to the noise image obtained by the aforementioned random sampling.

[0101] The parameters that need to be updated in the diffusion generation model include: parameters of the first encoding module, parameters of the second encoding module and parameters of the decoding module.

[0102] After the diffusion generation model is trained, in the testing phase, based on the diffusion generation model, the target motion information is used as a control condition to process the random noise image to generate a point set image containing multiple second key point sets. The process can be referred to the existing scheme and will not be repeated here.

[0103] Optionally, when the target motion information includes the first motion information and the second motion information, the target motion information is obtained by splicing the first motion information and the second motion information. In the above embodiment, the first motion information is in vector form and the second motion information is in matrix form. The second motion information can be stretched into a vector form and then the first motion information and the second motion information are spliced.

[0104] Optionally, the diffusion generation model can use a U-Net network or a network with other structures, such as a Transformer network, a convolutional neural network, a recurrent neural network, etc.

[0105] Optionally, the target motion information corresponding to the video frame may only include first motion information related to the space of each target in a preset number of video frames before the video frame.

[0106] Alternatively, the object motion information corresponding to the video frame may only include the second motion information of each object in the video frame that is related to time.

[0107] Alternatively, the target motion information corresponding to the video frame may include first motion information related to space of each target in a preset number of video frames before the video frame, and second motion information related to time of each target in the video frame.

[0108] Optionally, when the target motion information corresponding to a video frame includes first motion information related to space of each target in a preset number of video frames before the video frame, and second motion information related to time of each target in the video frame, during the training of the diffusion generation model, the first motion information or the second motion information in the target motion information corresponding to some video frames can be deleted, and only one type of motion information can be retained.

[0109] As an example, during the training of the diffusion generation model, a certain proportion of training samples can be randomly selected, and the first motion information in the target motion information corresponding to some of the training samples among the selected training samples can be discarded, and only the second motion information can be retained; the second motion information in the target motion information corresponding to another part of the training samples among the training samples can be discarded, and only the first motion information can be retained. This can improve the robustness of the diffusion generation model and enable it to cope with situations such as target occlusion and improve its generation capabilities.

[0110] In an optional embodiment, an implementation method of determining each target in the i-th frame image according to the matching relationship between each first key point set and each second key point set may be:

[0111] For each first key point set, if a second key point set matching the first key point set is found in multiple second key point sets, it is determined that the historical tracking target corresponding to the matched second key point set appears in the i-th frame image, that is, the target corresponding to the first key point set is the historical tracking target corresponding to the matched second key point set.

[0112] If a second key point set matching the first key point set is found in multiple second key point sets, it means that the first key point and the second key point set matching the first key point set correspond to the same target. At this time, the identification information of the target corresponding to the second key point set can be determined as the identification information of the target corresponding to the first key point set.

[0113] After determining the target, it is necessary to determine the position of the target. Optionally, the first key point set and the second key point set that matches it can be weighted summed to obtain a fused key point set; the coordinate information of the minimum bounding rectangle of the fused key point set is determined as the position of the historical tracking target corresponding to the matched second key point set in the i-th frame image, that is, the position of the target corresponding to the first key point set in the i-th frame image.

[0114] The weighted sum of the first key point set and the second key point set may specifically be: weighted sum of the key points corresponding to the same part in the first key point set and the second key point set (for example, key points with the same number).

[0115] As an example, the weight of the first keypoint set is greater than the weight of the second keypoint set.

[0116] As an example, the weight of the first key point set and the weight of the second key point set are both adjustable parameters, and the weight of the first key point set and the weight of the second key point set can be modified according to actual application requirements.

[0117] If a second key point set matching the first key point set is not found in the plurality of second key point sets, it is determined that the target corresponding to the first key point set is a newly appeared target.

[0118] If no second key point set matches the first key point set, it means that the target corresponding to the first key point set is different from the targets corresponding to the second key point sets, and the target corresponding to the second key point set is a target that has been tracked. Therefore, the target corresponding to the first key point set is a newly appeared target. Unique identification information can be assigned to the newly appeared target to distinguish the newly appeared identification information from other targets.

[0119] For a newly appearing target, the coordinate information of the minimum circumscribed rectangle of the first key point set corresponding to the newly appearing target may be determined as the position of the target corresponding to the first key point set (ie, the newly appearing target) in the i-th frame image.

[0120] In an optional embodiment, for each second key point set, if there is no first key point set matching it, it is determined that the target corresponding to the second key point set is lost once.

[0121] If the target corresponding to the second key point set is lost for a preset number of consecutive times, it is determined that the target corresponding to the second key point set is not within the tracking range. At this time, the identification information of the target corresponding to the second key point set can be deleted, and the target is no longer tracked.

[0122] In an optional embodiment, an implementation method of determining the matching relationship between any first key point set and any second key point set may be:

[0123] The graph neural network is used to process any first key point set and any second key point set to obtain a processing result, wherein the processing result represents whether any first key point set and any second key point set match.

[0124] Optionally, the graph neural network may be a graph convolutional neural network (GCNs), or a graph attention network (GAT), etc.

[0125] Optionally, a first feature of each of the N first key point sets may be extracted, and a second feature of each of the M second key point sets may be extracted.

[0126] The N first features are used as nodes, and a first relationship graph (including N nodes and the relationship between the two nodes) is constructed according to the relationship between the two first features (for example, similarity).

[0127] The M second features are used as nodes, and a second relationship graph (including the M nodes and the relationship between the two nodes) is constructed according to the relationship between the two second features (for example, similarity).

[0128] The first relationship graph and the second relationship graph are input into the graph neural network to obtain the matching relationship between each node in the first relationship graph (i.e., the first feature) and each node in the second relationship graph (i.e., the second feature) output by the graph neural network.

[0129] In an optional embodiment, another implementation manner of determining the matching relationship between any first key point set and any second key point set may be:

[0130] The area where the minimum circumscribed rectangle of any key point set in the point set image corresponding to any key point set is located is used as a template, and the point set images corresponding to multiple second key point sets are traversed to search for the area matching the template, and the second key point set in the searched matching area is determined as the second key point set matching the any key point set.

[0131] Since the structural information between the key points of the human body in the point set image corresponding to the key point set is very strong and there is no complex background interference, the matching scheme based on template matching is more effective and thus has higher tracking efficiency than determining the matching relationship between key point sets through graph neural networks.

[0132] like Figure 6 As shown, it is an example diagram of the overall architecture of the multi-target tracking method provided in the embodiment of the present application. In this example, it includes:

[0133] Key point detection branch and key point prediction branch;

[0134] Among them, the key point detection branch performs human key point detection on the video frame Frame through the human key point detection network RTMPose to obtain the point set detection result.

[0135] The key point prediction branch predicts the key point set in a diffusion-generated manner. Specifically, the diffusion model Diffusion is used based on randomly sampled noise images and control conditions (i.e. Figure 6The control condition is calculated based on the video frame Frame and / or the video frame before the video frame Frame.

[0136] Figure 6 The generation condition of the diffusion model in is the calculated motion information of n targets (Id1, Id2, Id3, ..., Idn), which is iteratively updated as the video frame Frame is updated.

[0137] The tracking result is obtained by matching the point set detection result and the point set prediction result, including: the ID (i.e. identification mark) and position of the tracked target. Figure 6 In the example shown, different targets are identified by rectangular boxes of different colors.

[0138] Corresponding to the method embodiment, the present application also provides a multi-target tracking device. A structural schematic diagram of the multi-target tracking device provided in the embodiment of the present application is as follows: Figure 7 As shown, this may include:

[0139] Detection module 701, motion information acquisition module 702, generation module 703 and target determination module 704;

[0140] The detection module 701 is used to perform key point detection on the i-th frame image in the video to determine multiple first key point sets corresponding to the i-th frame image; different first key point sets correspond to different targets in the i-th frame image; different key points in each first key point set represent different parts of the target corresponding to the first key point set;

[0141] The motion information obtaining module 702 is used to obtain target motion information if the i-th frame image is not the first frame image in the video; the target motion information includes at least one of the following two types of motion information: first motion information related to space of each target in a preset number of images before the i-th frame image, and second motion information related to time of each target in the i-th frame image;

[0142] The generating module 703 is used to generate a plurality of second key point sets corresponding to the i-th frame image based on the target motion information and the random noise image; different second key point sets correspond to different historical tracking targets; different key points in each second key point set represent different parts of the historical tracking target corresponding to the second key point set;

[0143] The target determination module 704 is used to determine each target and position in the i-th frame image according to the matching relationship between each first key point set and each second key point set.

[0144] The multi-target tracking device provided in the embodiment of the present application represents the target to be tracked in the form of a point set. In addition to detecting the key point set of the target from the i-th frame image, the key point set of the target is also predicted based on the motion information of the target. Then, based on the matching relationship between the detected key point set and the predicted key point set, the target and its position in the i-th frame image are determined, thereby improving the accuracy of multi-target tracking.

[0145] In an optional embodiment, when the motion information obtaining module 702 obtains the first motion information related to the space of each target in a preset number of images before the i-th frame image, it is used to:

[0146] The preset number of images are processed based on a frame difference method to obtain first motion information related to each target in the preset number of images and the space.

[0147] In an optional embodiment, when the motion information obtaining module 702 obtains the second motion information related to time of each target in the i-th frame image, it is used to:

[0148] Based on the first position of each key point in the plurality of first key point sets in the i-th frame image, predict the second position of each key point in the plurality of first key point sets in the (i+1)th frame image;

[0149] Determine the movement speed and movement direction of each key point based on the first position and the second position of the key point;

[0150] The movement speed and movement direction of each key point corresponding to the same target are fused to obtain the second movement information related to time of each target.

[0151] In an optional embodiment, when the generating module 703 generates a plurality of second key point sets corresponding to the i-th frame image based on the target motion information and the random noise image, it is used to:

[0152] Using the target motion information as a control condition of a diffusion generation model, the random noise image is processed by the diffusion generation model to generate a plurality of second key point sets corresponding to the i-th frame image;

[0153] The diffusion generation model is obtained by conditional diffusion training using point set images corresponding to video frames as training samples, target motion information corresponding to video frames as control conditions, and randomly sampled noise images corresponding to video frames as labels.

[0154] In an optional embodiment, the target motion information corresponding to the video frame includes first motion information related to space of each target in a preset number of video frames before the video frame, and second motion information related to time of each target in the video frame;

[0155] In the process of training the diffusion generation model, the first motion information or the second motion information in the target motion information corresponding to some video frames is deleted.

[0156] In an optional embodiment, when the target determination module 704 determines each target in the i-th frame image according to the matching relationship between each first key point set and each second key point set, it is configured to:

[0157] For each first key point set, if a second key point set matching the first key point set is found in the plurality of second key point sets, determining the target corresponding to the first key point set as the historical tracking target corresponding to the matched second key point set;

[0158] If no second key point set matching the first key point set is found in the plurality of second key point sets, it is determined that the target corresponding to the first key point set is a newly appeared target.

[0159] In an optional embodiment, when the target determination module 704 determines the position of each target in the i-th frame image according to the matching relationship between each first key point set and each second key point set, it is used to:

[0160] For each first key point set, if a second key point set matching the first key point set is found in the plurality of second key point sets, weighted sum of the first key point set and the second key point set matching the first key point set is performed to obtain a fused key point set; coordinate information of a minimum circumscribed rectangle of the fused key point set is determined as the position of the historical tracking target corresponding to the matched second key point set in the i-th frame image;

[0161] If no second key point set matching the first key point set is found in the plurality of second key point sets, the coordinate information of the minimum circumscribed rectangle of the first key point set is determined as the position of the target corresponding to the first key point set in the i-th frame image.

[0162] In an optional embodiment, when the target determination module 704 determines the matching relationship between any first key point set and any second key point set, it is used to:

[0163] Using a graph neural network to process any of the first key point sets and any of the second key point sets to obtain a processing result, wherein the processing result indicates whether any of the first key point sets and any of the second key point sets match;

[0164] Alternatively, the area where the minimum circumscribed rectangle of any key point set in the point set image corresponding to any key point set is located is used as a template, and the point set images corresponding to the multiple second key point sets are traversed to search for the area matching the template, and the second key point set in the searched matching area is determined as the second key point set matching any key point set.

[0165] The present application also provides an electronic device in an embodiment. Figure 8 As shown, it shows a structural schematic diagram of an electronic device suitable for implementing the multi-target tracking method in the embodiment of the present application. The electronic device in the embodiment of the present application can be a terminal device (for example, a mobile phone, a tablet computer, a laptop computer, a desktop computer, etc.), or a server (can be a single server, can be a server cluster, or can be a cloud server, etc.). Figure 8 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0166] like Figure 8 As shown, the electronic device may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 to a random access memory (RAM) 803. When the electronic device is powered on, various programs and data required for the operation of the electronic device are also stored in the RAM 803. The processing device 801, the ROM 802, and the RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0167] Typically, the following devices may be connected to the I / O interface 805: input devices 806 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 807 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 808 including, for example, a memory card, a hard disk, etc.; and communication devices 809. The communication device 809 may allow the electronic device to communicate with other devices wirelessly or by wire to exchange data. Although Figure 8 An electronic device having various devices is shown, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0168] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements any of the multi-target tracking methods provided in the embodiments of the present application.

[0169] A computer-readable storage medium is also provided in an embodiment of the present application. The storage medium carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any multi-target tracking method provided in an embodiment of the present application.

[0170] It should be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0171] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0172] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware, or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product. Professionals and technicians may use different methods to implement the described functions for each specific solution, but such implementation should not be considered beyond the scope of this application.

[0173] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

[0174] The various embodiments in this specification are described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the various embodiments can be referenced to each other.

[0175] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A multi-target tracking method, characterized in that: include: Performing key point detection on an i-th frame image in a video to determine a plurality of first key point sets corresponding to the i-th frame image; Different first key point sets correspond to different targets in the i-th frame image; different key points in each first key point set represent different parts of the target corresponding to the first key point set; If the i-th frame image is not the first frame image in the video, obtaining target motion information; The target motion information includes at least one of the following two types of motion information: first motion information related to space of each target in a preset number of images before the i-th frame image, and second motion information related to time of each target in the i-th frame image; Based on the target motion information and the random noise image, a plurality of second key point sets corresponding to the i-th frame image are generated; different second key point sets correspond to different historical tracking targets; Different key points in each second key point set represent different parts of the historical tracking target corresponding to the second key point set; According to the matching relationship between each first key point set and each second key point set, each target and position in the i-th frame image is determined.

2. The method according to claim 1, characterized in that: Obtaining first motion information related to space of each target in a preset number of images before the i-th frame image, including: The preset number of images are processed based on a frame difference method to obtain first motion information related to each target in the preset number of images and the space.

3. The method according to claim 1, characterized in that Obtaining second motion information of each target in the i-th frame image related to time, including: Based on the first position of each key point in the plurality of first key point sets in the i-th frame image, predict the second position of each key point in the plurality of first key point sets in the (i+1)th frame image; Determine the movement speed and movement direction of each key point based on the first position and the second position of the key point; The movement speed and movement direction of each key point corresponding to the same target are fused to obtain the second movement information related to time of each target.

4. The method according to claim 1, characterized in that: The step of generating a plurality of second key point sets corresponding to the i-th frame image based on the target motion information and the random noise image comprises: Using the target motion information as a control condition of a diffusion generation model, the random noise image is processed by the diffusion generation model to generate a plurality of second key point sets corresponding to the i-th frame image; The diffusion generation model is obtained by conditional diffusion training using point set images corresponding to video frames as training samples, target motion information corresponding to video frames as control conditions, and randomly sampled noise images corresponding to video frames as labels.

5. The method according to claim 4, characterized in that The target motion information corresponding to the video frame includes first motion information related to space of each target in a preset number of video frames before the video frame, and second motion information related to time of each target in the video frame; In the process of training the diffusion generation model, the first motion information or the second motion information in the target motion information corresponding to some video frames is deleted.

6. The method according to claim 1, characterized in that Determining each target in the i-th frame image according to the matching relationship between each first key point set and each second key point set includes: For each first key point set, if a second key point set matching the first key point set is found in the plurality of second key point sets, determining the target corresponding to the first key point set as the historical tracking target corresponding to the matched second key point set; If no second key point set matching the first key point set is found in the plurality of second key point sets, it is determined that the target corresponding to the first key point set is a newly appeared target.

7. The method according to claim 6, characterized in that Determining the position of each target in the i-th frame image according to the matching relationship between each first key point set and each second key point set includes: For each first key point set, if a second key point set matching the first key point set is found in the plurality of second key point sets, weighted sum of the first key point set and the second key point set matching the first key point set is performed to obtain a fused key point set; coordinate information of a minimum circumscribed rectangle of the fused key point set is determined as the position of the historical tracking target corresponding to the matched second key point set in the i-th frame image; If no second key point set matching the first key point set is found in the plurality of second key point sets, the coordinate information of the minimum circumscribed rectangle of the first key point set is determined as the position of the target corresponding to the first key point set in the i-th frame image.

8. The method according to claim 6 or 7, characterized in that: The process of determining the matching relationship between any first key point set and any second key point set includes: Using a graph neural network to process any of the first key point sets and any of the second key point sets to obtain a processing result, wherein the processing result indicates whether any of the first key point sets and any of the second key point sets match; Alternatively, the area where the minimum circumscribed rectangle of any key point set in the point set image corresponding to any key point set is located is used as a template, and the point set images corresponding to the multiple second key point sets are traversed to search for the area matching the template, and the second key point set in the searched matching area is determined as the second key point set matching any key point set.

9. A multi-target tracking device, characterized in that: include: A detection module, used for performing key point detection on an i-th frame image in a video to determine a plurality of first key point sets corresponding to the i-th frame image; Different first key point sets correspond to different targets in the i-th frame image; different key points in each first key point set represent different parts of the target corresponding to the first key point set; A motion information obtaining module, used for obtaining target motion information if the i-th frame image is not the first frame image in the video; The target motion information includes at least one of the following two types of motion information: first motion information related to space of each target in a preset number of images before the i-th frame image, and second motion information related to time of each target in the i-th frame image; A generating module, used for generating a plurality of second key point sets corresponding to the i-th frame image based on the target motion information and the random noise image; Different second key point sets correspond to different historical tracking targets; Different key points in each second key point set represent different parts of the historical tracking target corresponding to the second key point set; The target determination module is used to determine each target and position in the i-th frame image according to the matching relationship between each first key point set and each second key point set.

10. A computer program product, characterized in that It comprises computer-readable instructions, and when the computer-readable instructions are executed on an electronic device, the electronic device implements the multi-target tracking method as claimed in any one of claims 1 to 8.

11. An electronic device, characterized in that: The electronic device comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the multi-target tracking method as described in any one of claims 1 to 8.

12. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the multi-target tracking method as described in any one of claims 1 to 8.

Images