Object association method, device and electronic device

Abstract

The present application provides an object association method, device and electronic device, including: acquiring an image frame sequence including a first image and a second image; the acquisition time of the second image is earlier than the acquisition time of the first image; acquiring target position information and target feature information of a target object identified from the first image; acquiring historical position information, historical feature information and historical tracking information of a tracked object identified from the second image, and predicting the predicted position information of the tracked object in the first image based on the historical position information of each tracked object; inputting the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object and the historical feature information of the tracked object into a trained similarity prediction model to determine whether the target object and the tracked object are the same target, thereby improving the accuracy of establishing the association relationship between the target object and the tracked object.

Description

Technical Field

[0001] The present application relates to the field of image processing, and in particular to an object association method, device, and electronic device. Background Art

[0002] Image-based target object tracking technology refers to: associating at least one target object identified from the current video frame with the tracking object identified from the previous video frame, and determining the motion trajectory of the target object based on the position information of the target object and its associated tracking object, thereby achieving tracking.

[0003] Specifically, after identifying relevant information of at least one target object from the first video frame, the identified target object can be used as a tracking object, and the tracked object and its relevant information can be recorded. When the target object is identified from subsequent video frames, the target object can be matched with each tracked object. If the target object matches any tracked object, it is determined that the target object and any tracked object are the same object, and an association relationship between the target object and any tracked object is established. Then, based on the position information of the target object and its associated tracked objects, the target object's motion trajectory is determined, thereby achieving tracking of the target object.

[0004] It can be seen that how to establish the association relationship between the target object and the tracked object is the key to achieving target object tracking, so establishing the association relationship between the target object and the tracked object becomes an urgent problem to be solved. Summary of the Invention

[0005] In view of this, the present application provides an object association method, apparatus, and electronic device for improving the accuracy of establishing an association relationship between a target object and a tracked object.

[0006] Specifically, this application is implemented through the following technical solutions:

[0007] According to a first aspect of the present application, there is provided an object association method, the method comprising:

[0008] Acquire an image frame sequence including a first image and a second image, wherein the acquisition time of the second image is earlier than the acquisition time of the first image;

[0009] Acquiring target position information and target feature information of the target object identified from the first image;

[0010] Acquire historical position information, historical feature information, and historical tracking information of the tracked objects identified from the second image, and predict predicted position information of each tracked object in the first image based on the historical position information of the tracked object;

[0011] The target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object and the historical feature information of the tracked object are input into the trained similarity prediction model to obtain whether the target object and the tracked object are the same target.

[0012] Optionally, the number of the target objects is at least one, and the number of the tracked objects is at least one;

[0013] The step of inputting the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object, and the historical feature information of the tracked object into a trained similarity prediction model to determine whether the target object and the tracked object are the same target includes:

[0014] Inputting the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object, and the historical feature information of the tracked object into a trained similarity prediction model;

[0015] The similarity prediction model determines first feature information for representing the predicted position similarity between each target object and each tracked object based on the target position information of each target object and the predicted position information of each tracked object, and determines second feature information for representing the feature similarity between each target object and each tracked object based on the target feature information of each target object and the historical feature information of each tracked object, and convolves the historical tracking information of each tracked object to obtain an attention probability mask;

[0016] The similarity prediction model fuses the first feature information and the second feature information between each target object and each tracked object, and performs a mask operation on the fusion result and the attention probability mask to obtain the similarity between each target object and each tracked object;

[0017] The similarity prediction model determines and outputs whether each target object and each tracked object are the same target based on a trained similarity threshold and the similarity between each target object and each tracked object.

[0018] Optionally, inputting the target feature information of the target object and the historical feature information of the tracked object into a trained similarity prediction model includes:

[0019] Clustering the historical feature information of the tracked object to obtain a clustering result, the clustering result including: at least one feature category and at least one cluster corresponding to each feature category; the historical feature information in each cluster matches the feature category corresponding to the cluster;

[0020] The target feature information of the target object is spliced ​​with the cluster center of at least one cluster, and the splicing result is input into the trained similarity prediction model.

[0021] Optionally, the number of target objects is m, the number of tracked objects is n, and the number of clusters is k;

[0022] The splicing result is represented by a tensor of (m*n)*k*2dims; where m*n represents the number of matching pairs between the target object and the tracked object, k represents the number of clusters, and 2dims represents the number of dimensions of the feature information;

[0023] The determining, based on the target feature information of each target object and the historical feature information of each tracked object, second feature information for indicating feature similarity between each target object and each tracked object includes:

[0024] Perform a convolution operation on the concatenated result in a dimension corresponding to 2dims to obtain a first tensor, where the first tensor is represented by (m*n)*k*p, where p represents the number of dimensions of the convolution operation result;

[0025] Swap the elements of the dimension corresponding to k and the dimension corresponding to p in the first tensor to obtain a second tensor, which is represented by (m*n)*p*k;

[0026] A convolution operation is performed on the second tensor in the dimension corresponding to k, and the obtained result is determined as second feature information for representing the feature similarity between each target object and each tracked object.

[0027] Optionally, the similarity prediction model represents the similarity between each target object and each tracked object through a similarity matrix, and each element in the similarity matrix represents the similarity between a target object and a tracked object;

[0028] The determining and outputting whether each target object and each tracked object are the same target based on the trained similarity threshold and the similarity between each target object and each tracked object includes:

[0029] Expanding the similarity matrix using the similarity threshold so that the expanded similarity matrix includes the similarity threshold;

[0030] Setting weights for the elements in each row of the expanded similarity matrix based on the values ​​of the elements in the row, and setting weights for the elements in the column based on the values ​​of the elements in each row of the expanded similarity matrix;

[0031] Perform minimum flow calculation on the expanded similarity matrix based on the weights set for each row element and each column element to obtain the matching value between each target object and each tracked object;

[0032] Based on the matching value between each target object and each tracked object, it is determined whether each target object and each tracked object are the same target and outputted.

[0033] Optionally, the number of target objects is m, the number of tracked objects is n; the size of the similarity matrix is ​​m*n;

[0034] The method of expanding the similarity matrix by using a similarity threshold comprises:

[0035] The m*n similarity matrix is ​​expanded to obtain an M*N similarity matrix; where M represents the maximum number of detectable target objects and N represents the maximum number of trackable target objects;

[0036] The similarity threshold is used to expand the M*N similarity matrix to obtain a 2M*2N similarity matrix.

[0037] Optionally, the determining whether each target object and each tracked object are the same target based on the matching value between each target object and each tracked object includes:

[0038] For each target object, if the matching value between the target object and any tracked object is a first preset value, then the target object is determined to be the same target as the tracked object;

[0039] If the target object and all the tracked objects match to the second preset value, it means that the target object and all the tracked objects are not the same target.

[0040] Optionally, the method further includes:

[0041] For each target object, if the target object is not the same as any of the tracked objects, the target object is determined to be a newly appeared object;

[0042] For each tracked object, if the tracked object is not the same as all target objects, it is determined that the tracked object disappears in the first image.

[0043] Optionally, the similarity threshold is a model parameter of the similarity prediction model, and the similarity threshold is obtained by training the similarity prediction model.

[0044] According to a second aspect of the present application, there is provided an object association device, the device comprising:

[0045] an acquisition unit, configured to acquire an image frame sequence comprising a first image and a second image, wherein the acquisition time of the second image is earlier than the acquisition time of the first image; acquire target position information and target feature information of a target object identified from the first image; and acquire historical position information, historical feature information, and historical tracking information of a tracked object identified from the second image;

[0046] a prediction unit, configured to predict predicted position information of each tracked object in the first image based on historical position information of the tracked object;

[0047] The output unit is used to input the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object and the historical feature information of the tracked object into the trained similarity prediction model to determine whether the target object and the tracked object are the same target.

[0048] Optionally, the number of the target objects is at least one, and the number of the tracked objects is at least one;

[0049] The output unit is used to input the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object and the historical feature information of the tracked object into the trained similarity prediction model to obtain whether the target object and the tracked object are the same target. The similarity prediction model determines, based on the target position information of each target object and the predicted position information of each tracked object, a target feature information of the target object and the historical feature information of the tracked object. first feature information, and, based on the target feature information of each target object and the historical feature information of each tracked object, determining the second feature information used to represent the feature similarity between each target object and each tracked object, and convolving the historical tracking information of each tracked object to obtain an attention probability mask; the similarity prediction model fuses the first feature information and the second feature information between each target object and each tracked object, and performs a mask operation on the fusion result and the attention probability mask to obtain the similarity between each target object and each tracked object; the similarity prediction model determines and outputs whether each target object and each tracked object are the same target based on the trained similarity threshold and the similarity between each target object and each tracked object.

[0050] Optionally, when the target feature information of the target object and the historical feature information of the tracked object are input into a trained similarity prediction model, the output unit is used to cluster the historical feature information of the tracked object to obtain a clustering result, wherein the clustering result includes: at least one feature category, and at least one cluster cluster corresponding to each feature category; the historical feature information in each cluster cluster is matched with the feature category corresponding to the cluster cluster; the target feature information of the target object is spliced ​​with the cluster center of at least one cluster cluster, and the splicing result is input into the trained similarity prediction model.

[0051] Optionally, the number of target objects is m, the number of tracked objects is n, and the number of clusters is k;

[0052] The input feature information is represented by a tensor of (m*n)*k*2dims; where m*n represents the number of matching pairs between the target object and the tracked object, k represents the number of clusters, and 2dims represents the number of dimensions of the feature information;

[0053] The output unit, when determining the second feature information used to represent the feature similarity between each target object and each tracked object based on the target feature information of each target object and the historical feature information of each tracked object, is used to perform a convolution operation on the splicing result in the dimension corresponding to 2dims to obtain a first tensor, where the first tensor is represented by (m*n)*k*p, where p represents the number of dimensions of the convolution operation result; swap the elements in the dimension corresponding to k and the dimension corresponding to p in the first tensor to obtain a second tensor, where the second tensor is represented by (m*n)*p*k; perform a convolution operation on the second tensor in the dimension corresponding to k, and determine the obtained result as the second feature information used to represent the feature similarity between each target object and each tracked object.

[0054] Optionally, the similarity prediction model represents the similarity between each target object and each tracked object through a similarity matrix, the size of the similarity matrix is ​​m*n, and each element in the similarity matrix represents the similarity between a target object and a tracked object;

[0055] The output unit is used to, when determining and outputting whether each target object and each tracked object are the same target based on a trained similarity threshold and the similarity between each target object and each tracked object, expand the similarity matrix using the similarity threshold so that the expanded similarity matrix includes the similarity threshold; set weights for the row elements based on the values ​​of the elements on each row of the expanded similarity matrix, and set weights for the column elements based on the values ​​of the elements on each row of the expanded similarity matrix; perform minimum flow calculation on the expanded similarity matrix based on the weights set for the elements in each row and each column to obtain matching values ​​for each target object and each tracked object; and obtain and output the association relationship between each target object and each tracked object based on the matching values ​​for each target object and each tracked object.

[0056] Optionally, the number of target objects is m, the number of tracked objects is n; the size of the similarity matrix is ​​m*n;

[0057] The output unit is used to expand the m*n similarity matrix to obtain an M*N similarity matrix when the similarity threshold is used to expand the similarity matrix; wherein M represents the maximum number of detectable target objects, and N represents the maximum number of trackable target objects; and the M*N similarity matrix is ​​expanded using the similarity threshold to obtain a 2M*2N similarity matrix.

[0058] Optionally, the output unit, based on the matching value of each target object and each tracked object, obtains whether each target object and each tracked object are the same target, and is used to, for each target object, if the matching value between the target object and any tracked object is a first preset value, determine that the target object and any tracked object are the same target; if the matching value between the target object and all tracked objects is a second preset value, it indicates that the target object and all tracked objects are not the same target.

[0059] Optionally, the output unit is further configured to, for each target object, determine that the target object is a newly appeared object if the target object is not the same target as all the tracked objects;

[0060] For each tracked object, if the tracked object is not the same as all target objects, it is determined that the tracked object disappears in the first image.

[0061] Optionally, the similarity threshold is a model parameter of the similarity prediction model, and the similarity threshold is obtained by training the similarity prediction model.

[0062] According to a third aspect of the present application, there is provided an electronic device, comprising a readable storage medium and a processor;

[0063] Wherein, the readable storage medium is used to store machine-executable instructions;

[0064] The processor is configured to read the machine-executable instructions on the readable storage medium and execute the instructions to implement the object association method.

[0065] According to a fourth aspect of the present application, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the object association method described above is implemented.

[0066] According to a fifth aspect of the present application, a computer program is provided. The computer program is stored in a computer-readable storage medium, and when a processor executes the computer program, the processor is prompted to implement the above-mentioned object association method.

[0067] As can be seen from the above description, on the one hand, the electronic device can identify the target position information and target feature information of the target object from the first image. The electronic device identifies the historical position information, historical feature information and historical tracking information of the tracked object from the second image whose acquisition time is before the first image, and predicts the predicted position information of the tracked object in the first image based on the historical position information of each tracked object. The electronic device can input the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, and the target feature information of m target objects and the historical feature information of the tracked object into the trained similarity prediction model to obtain whether the target object and the tracked object are the same target (or whether the target object and the tracked object are associated), thereby realizing the establishment of an association relationship between the target object and the tracked object.

[0068] On the other hand, in the present application, since a supervised similarity prediction model is used to predict whether the target object and the tracked object are the same target based on the relevant information extracted from the target object and the historical tracked object, rather than manually setting a similarity threshold and manually setting a measurement function to determine whether the target object and the tracked object are the same object, the method provided by the present application to determine whether the target object and the tracked object are the same target is more accurate. BRIEF DESCRIPTION OF THE DRAWINGS

[0069] Figure 1 is a flow chart of an object association method shown in an exemplary embodiment of the present application;

[0070] Figure 2 is a schematic diagram of a clustering method shown in an exemplary embodiment of the present application;

[0071] Figure 3 This is a schematic diagram of target feature information and cluster center splicing shown in an exemplary embodiment of the present application;

[0072] Figure 4 is a schematic diagram of a similarity prediction model for similarity prediction according to an exemplary embodiment of the present application;

[0073] Figure 5 is a schematic diagram showing an exemplary embodiment of the present application for expanding a similarity matrix;

[0074] Figure 6 is a schematic diagram of performing minimum flow calculation according to an exemplary embodiment of the present application;

[0075] Figure 7 is a hardware structure diagram of an electronic device shown in an exemplary embodiment of the present application;

[0076] Figure 8It is a block diagram of an object association device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0077] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. In the following description, when referring to the drawings, identical numerals in different figures represent identical or similar elements, unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0078] The terms used in this application are for the purpose of describing specific embodiments only and are not intended to limit this application. As used in this application and the appended claims, the singular forms "a," "an," "the," and "the" are intended to include the plural forms, unless the context clearly indicates otherwise. It should also be understood that the term "and / or" as used herein refers to and encompasses any and all possible combinations of one or more of the associated listed items.

[0079] It should be understood that although the terms first, second, third, etc. may be used in this application to describe various information, such information should not be limited to these terms. These terms are only used to distinguish information of the same type from each other. For example, without departing from the scope of this application, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0080] Image-based object tracking involves associating at least one target object identified in the current video frame with tracked objects identified in previous video frames. Based on the positional information of the target object and its associated tracked objects, the target object's trajectory is determined, thereby enabling tracking. Association refers to matching the target object with each tracked object. If the target object and a tracked object are the same, the target object is associated with the tracked object.

[0081] For example, the electronic device can identify three target objects from the first video frame and use these three target objects as tracking objects. When the electronic device receives the second video frame, assuming that two target objects are identified from the second video frame, in order to track the objects, the electronic device needs to match these two target objects with the three tracking objects. For each target object, if the target object successfully matches any tracking object, it is determined that the target object and the tracking object are the same target, and a corresponding relationship between the target object and the tracking object is established. The motion trajectory of the object is determined based on the position information of the tracking object and the position information of the target object, thereby tracking the object.

[0082] It can be seen that establishing an association relationship between the target object in the first image and the tracking object in the historical frame is an essential step in achieving object tracking.

[0083] In view of this, the present application aims to propose an object association method. On the one hand, the electronic device can identify the target position information and target feature information of the target object from the first image. The electronic device identifies the historical position information, historical feature information and historical tracking information of the tracked object from a second image whose acquisition time is before the first image, and predicts the predicted position information of the tracked object in the first image based on the historical position information of each tracked object. The electronic device can input the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, and the target feature information of m target objects and the historical feature information of the tracked object into a trained similarity prediction model to obtain whether the target object and the tracked object are the same target (or whether the target object and the tracked object are associated), thereby realizing the establishment of an association relationship between the target object and the tracked object.

[0084] On the other hand, in the present application, since a supervised similarity prediction model is used to predict the similarity between the target object and the tracked object based on the relevant information extracted from the target object and the historical tracked object, and then whether the target object and the tracked object are the same target is determined based on the similarity and the similarity threshold obtained by training with the similarity prediction model, rather than manually setting the similarity threshold and manually setting the measurement function to calculate the similarity between the target object and the tracked object, the present application is more accurate in determining whether the target object and the tracked object are the same target.

[0085] Before introducing the object association method provided by this application, the concepts involved in this application are first introduced.

[0086] 1. Object

[0087] In this application, an object refers to something that needs to be tracked. For example, an object can be a vehicle, an animal, etc. This is just an example of an object and does not specifically limit it.

[0088] In addition, in this application, for the convenience of description, the object identified from the first image may be referred to as the target object, and the object identified from the historical frame before the first image may be referred to as the tracking object.

[0089] 2. Object location information

[0090] The object's position information can indicate the object's position within a video frame. This position information can be represented by the position information of a rectangular frame enclosing the object, such as the coordinates of the top-left corner of the rectangular frame, and the width and height of the rectangular frame. Alternatively, the position information can be represented by the coordinates of other vertices of the rectangular frame, the width and height of the rectangular frame, or the coordinates of each vertex of the rectangular frame. This is merely an example of the target object's position information and is not specifically limited.

[0091] 3. Feature information of the object

[0092] The feature information of an object may be information such as surface features used to represent the appearance of the object.

[0093] For example, when the object is a vehicle, the appearance feature may be a vehicle feature of the vehicle (such as license plate feature, vehicle color, brand, etc.) Here, the feature information of the object is merely exemplified and is not specifically limited.

[0094] 4. Object tracking information

[0095] In the present application, the same object can be identified from multiple video frames, and the electronic device can form tracking information based on the relevant information of the same object in the multiple video frames.

[0096] Tracking information can represent the information of the same tracked object in different video frames, so the intrinsic correlation between different video frames corresponding to the same tracked object can be represented, which helps to improve the accuracy of subsequent similarity calculation.

[0097] Specifically, the tracking information refers to information related to tracking of the tracked object, such as the track length of the tracked object, the number of lost frames of the tracked object, and the overlap between the tracked object and other tracked objects.

[0098] The track length of a tracked object refers to the total number of video frames from the time a tracked object is established to the time the tracked object is associated with an object. For example, if there are 10 video frames, and tracked object 1 is associated with an object identified from frames 1 to 4, but not associated with any object identified from frames 5 to 10, the track length of tracked object 1 is 4 frames.

[0099] The number of lost frames for a tracked object refers to the number of consecutive video frames in which the tracked object is not associated with an object since it was established. For example, assuming that tracked object 1 is associated with an object in frames 1 to 3, and is not associated with the object identified in frame 4, then the number of lost frames for tracked object 1 in frames 1-4 is 0. Assuming that tracked object 1 is not associated with the object identified in frame 5, then the number of lost frames for tracked object 1 in frame 5 is 1. Assuming that tracked object 1 is associated with an object in frames 6 to 10, then the number of lost frames for tracked object 1 in frame 6 is 2. From frames 7 to 10, the number of lost frames for tracked object 1 is 0.

[0100] The overlap of the tracked object and other tracked objects means that due to the occlusion between objects in the captured image, the rectangular boxes where the tracked object and other tracked objects are located will overlap. Therefore, the overlap of the tracked object and other tracked objects can be represented by the overlap of the rectangular boxes where the tracked object and other tracked objects are located, such as the overlapping area, etc.

[0101] The tracking information is merely described here as an example. In practical applications, any information related to the tracking of a tracking object may be referred to as tracking information, and the tracking information is not specifically limited here.

[0102] After introducing the above concepts, the object association method provided by this application is introduced in detail below.

[0103] See also Figure 1 , Figure 1 This is a flowchart of an object association method shown in an exemplary embodiment of the present application. The method can be applied to an electronic device, which can be a server, a data center, a computer, etc. The electronic device is only illustrated here for example without specific limitation.

[0104] The object association method of the present application is applicable to determining whether a target object and a tracking object are the same object, and is also applicable to determining whether a target object and each of multiple tracking objects are the same target, and is also applicable to determining whether each of multiple target objects and each of multiple tracking objects are the same target. The applicable scenarios of this method are not specifically limited here.

[0105] The object association method may include the following steps.

[0106] Step 101: An electronic device acquires an image frame sequence including a first image and a second image; wherein the acquisition time of the second image is earlier than the acquisition time of the first image.

[0107] In an optional implementation, the front-end acquisition device sends a currently acquired image to the electronic device. The electronic device may use the currently acquired image as a first image and select a second image acquired earlier than the first image from images previously acquired by the front-end acquisition device to form an image frame sequence.

[0108] In another optional implementation, the electronic device records all images captured by the front-end acquisition device. The user can specify an image for multi-target association. The electronic device uses the user-specified image as the first image and a second image captured earlier than the first image to form an image frame sequence.

[0109] Here, the method of acquiring the image frame sequence is merely exemplified and is not specifically limited.

[0110] Step 102: The electronic device obtains target position information and target feature information of the target object identified from the first image.

[0111] In an optional implementation, the front-end acquisition device performs object recognition on each captured image, obtains object location information and feature information, and transmits the information to the electronic device. The electronic device can directly obtain the target location information and target feature information of the target object recognized by the front-end acquisition device from the first image.

[0112] In another optional implementation, the electronic device may recognize the first image, and identify target position information and target feature information of the target object in the first image from the first image.

[0113] Here, “how the electronic device obtains the target position information and target feature information of the target object identified from the first image” is merely exemplarily described without any specific limitation.

[0114] Step 103: The electronic device obtains historical location information, historical feature information, and historical tracking information of the tracked object identified from the second image, and predicts predicted location information of each tracked object in the first image based on the historical location information of the tracked object.

[0115] The following describes step 103 in detail from step 1031 to step 1032.

[0116] Step 1031: The electronic device obtains historical position information, historical feature information, and historical tracking information of n tracking objects identified from the second image.

[0117] The following first introduces how the historical location information, historical feature information, and historical tracking information of the tracking object are recorded, and then introduces the implementation method of step 1031.

[0118] 1) Recording of historical location information, historical feature information, and historical tracking information of the tracked object

[0119] In this application, when an electronic device receives each video frame, in addition to obtaining the location information and feature information of the object identified from the video frame, it also needs to associate the identified object with the object identified in the previous video frame and generate tracking information for the object based on the association relationship. The electronic device can use the identified object as a tracking object and record the information of the tracking object. The tracking object information includes: the location information, feature information, and tracking information of the tracking object identified from the video frame.

[0120] For example, after receiving the first video frame, the electronic device identifies the position information and feature information of three objects, namely object 1, object 2 and object 3, and also records the overlapping information of the three objects and other tracking-related information.

[0121] After receiving the second video frame, the electronic device assumes that two objects are identified, and assumes that the first of the two objects matches object 1 and the second object matches object 2. Then, the electronic device can record the position information and feature information of object 1 and object 2 in the second video frame respectively, as well as the overlap of object 1 and object 2 in the second video frame.

[0122] Then, as of these two video frames, the electronic device can determine that the trajectory of object 1 consists of the position of object 1 in the first video frame and the position in the second video frame, the trajectory length is 2 frames, the number of lost frames is 0 frames, and the overlap can be represented by the overlap between the two video frames. The electronic device can then generate tracking information for object 1 based on the tracking-related information such as the trajectory length, the number of lost frames, and the overlap.

[0123] Similarly, the electronic device can determine that the trajectory of object 2 consists of the position of object 2 in the first video frame and the position of object 2 in the second video frame, the trajectory length is 2 frames, the number of lost frames is 0 frames, and the overlap can be represented by the overlap between the two video frames. The electronic device can then generate tracking information for object 2 based on the tracking-related information such as the trajectory length, the number of lost frames, and the overlap.

[0124] The electronic device may determine that the trajectory of object 3 is composed of the position of object 3 in the first video frame, the trajectory length is 1 frame, the number of lost frames is 0 frames, and the overlap can be represented by the overlap in the first video frame. The electronic device may then generate tracking information for object 3 based on the tracking-related information such as the trajectory length, the number of lost frames, and the overlap.

[0125] Thus, the electronic device records the position information, feature information, and tracking information of each tracked object identified from the received video frame.

[0126] 2) Implementation of step 1031

[0127] When implementing step 1021 , the electronic device may obtain historical position information, historical feature information, and historical tracking information of the tracked object identified from the second image from the recorded position information, feature information, and tracking information of each tracked object.

[0128] Still taking the above example, assuming that the third video frame is the first image, the electronic device can determine the historical location information, historical feature information and historical tracking information of the three tracking objects identified from the first two video frames from the recorded information of the tracking objects.

[0129] Specifically, the electronic device can obtain the historical location information, historical feature information, and historical tracking information of object 1 in the first and second video frames. The electronic device can also obtain the historical location information, historical feature information, and historical tracking information of object 2 in the first and second video frames. The electronic device can also obtain the historical feature information and historical tracking information of object 3 in the first video frame.

[0130] Step 1032: The electronic device predicts predicted position information of each tracked object in the first image based on the historical position information of the tracked object.

[0131] During implementation, for each tracked object, the electronic device may input the historical position information of the tracked object in at least one second image into a trained position prediction model, so that the position prediction model outputs the predicted position information of the tracked object in the first image.

[0132] For example, it is assumed that the historical frames are video frames 1 to 10, and the first image is video frame 11. It is assumed that video frames 1 to 10 all contain object 1.

[0133] The electronic device may input the historical position information of the object 1 in the video frames 1 to 10 respectively into the position prediction model, so that the position prediction model outputs the predicted position information of the object 1 in the video frame 11 .

[0134] Step 104: The electronic device inputs the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object, and the historical feature information of the tracked object into the trained similarity prediction model to determine whether the target object and the tracked object are the same target.

[0135] There is at least one target object and at least one tracking object.

[0136] The process of "determining whether the target object and the tracked object are the same object" may include the following situations:

[0137] Get whether a target object and a tracking object are the same target;

[0138] Determine whether a target object and each of the multiple tracking objects are the same target;

[0139] It is determined whether each target object among the plurality of target objects and each tracked object among the plurality of tracked objects are the same target.

[0140] Optionally, the number of target objects is m, where m is an integer greater than or equal to 1; the number of tracked objects is n, where n is an integer greater than or equal to 1.

[0141] Step 104 will be described in detail below through steps 1041 to 1044.

[0142] Step 1041: The electronic device inputs the target position information of the m target objects and the predicted position information of the n tracked objects into the trained similarity prediction model.

[0143] During implementation, in order to simplify the computational complexity of the similarity prediction model and to adapt the input data format to the similarity prediction model, the electronic device may first preprocess the target position information of the m target objects and the predicted position information of the n tracking objects, and input the preprocessed target position information of the m target objects and the predicted position information of the n tracking objects into the trained similarity prediction model.

[0144] 1) Preprocessing of target position information for m target objects

[0145] Assume that this application uses the position information of the rectangular frame where the target object is located to represent the target position information of the target object.

[0146] Assume that the position information of the rectangular frame may include: the vertex coordinates of the upper left corner of the rectangular frame (x1, y1), the width w1 of the rectangular frame, and the height h1 of the rectangular frame. Then the position information of the rectangular frame can be expressed as [x1, y1, w1, h1]

[0147] For each target object, the electronic device may perform normalization processing on the target position information [x1, y1, w1, h1] of the target object. The normalization processing formula may refer to the following formula:

[0148] x 1g =x1 / W;

[0149] y 1g =y1 / H;

[0150] w 1g =w1 / W;

[0151] h 1g =h1 / H;

[0152] Wherein, W is the width of a frame image, and H is the height of a frame image.

[0153] The target position information of the normalized target object can be expressed as [x 1g ,y 1g ,w 1g ,h 1g ].

[0154] 2) Preprocessing of predicted position information for n tracked objects

[0155] Assume that this application uses the position information of the rectangular box where the tracked object is located to represent the predicted position information of the tracked object.

[0156] Assume that the position information of the rectangular box may include: the vertex coordinates of the upper left corner of the rectangular box (x2, y2), the width w2 of the rectangular box, and the height h2 of the rectangular box. Then the predicted position information of the tracked object can be expressed as [x2, y2, w2, h2];

[0157] For each tracked object, the electronic device may perform normalization processing on the predicted position information [x2, y2, w2, h2] of the tracked object. The normalization processing formula may be referred to as follows:

[0158] x 2g =x2 / W;

[0159] y 2g =y2 / H;

[0160] w 2g =w2 / W;

[0161] h 2g =h2 / H;

[0162] Wherein, W is the width of a frame image, and H is the height of a frame image.

[0163] The predicted position information of the tracked object after normalization can be expressed as [x 2g ,y 2g ,w 2g ,h 2g ].

[0164] Then, the electronic device may form a (m*n)*8 matrix with the normalized target position information of each target object and the predicted position information of each tracked object, and then input the matrix into the trained similarity prediction model.

[0165] Where m*n represents the number of matching pairs between the target object and the tracking object. For example, assume there are two target objects (i.e., m=2), namely target object 1 and target object 2. There are three tracking objects (i.e., n=3), namely tracking object 1, tracking object 2, and tracking object 3. Then the matching pairs formed by the target object and the tracking object include 6 matching pairs (i.e., m*n), namely [target object 1, tracking object 1], [target object 1, tracking object 2], [target object 1, tracking object 3], [target object 2, tracking object 1], [target object 2, tracking object 2], [target object 2, tracking object 3].

[0166] The "8" in the above (m*n)*8 represents the number of dimensions of the position information, where the target position information is represented by [x 1g ,y 1g ,w 1g ,h 1g ]These four dimensions represent the predicted position information by [x 2g ,y 2g ,w 2g ,h 2g ]These 4 dimensions represent the position information, so the position information here is represented by 8 dimensions.

[0167] Step 1042: The target feature information of the m target objects of the electronic device and the historical feature information of the n tracked objects are input into the trained similarity prediction model.

[0168] The following describes step 1042 in detail through steps A1 and A2.

[0169] Step A1: The electronic device clusters the historical feature information of the tracked object to obtain a clustering result, wherein the clustering result includes: at least one feature category and at least one cluster corresponding to each feature category; the historical feature information in each cluster matches the feature category corresponding to the cluster.

[0170] During implementation, the feature information of an object may typically include multiple categories. For example, if the object is a car, the image of the car while driving may be a front image, a back image, or a side image of the car. In order for the subsequent similarity prediction model to calculate a more accurate feature similarity between the target object and the tracked object, the electronic device may classify the historical feature information of n tracked objects, then concatenate the target feature information of m target objects with the classified historical feature information and input the concatenated information into the similarity prediction model. This allows the similarity prediction model to calculate the similarity between the target feature information of each target object and the historical feature information of different categories for each tracked object, thereby obtaining a more accurate feature similarity between the target object and the tracked object.

[0171] Specifically, the electronic device may cluster the historical feature information of n tracked objects to obtain a clustering result, which includes: k feature categories and k clusters corresponding to the k feature categories; the historical feature information in each cluster matches the feature category corresponding to the cluster.

[0172] For example, Figure 2 As shown in the figure, the triangle represents the cluster center, and the circles surrounding the triangle form clusters. Assume that the historical feature information of n tracked objects is clustered to obtain three clusters, namely cluster 1, cluster 2, and cluster 3. The categories of these three clusters are front features, back features, and side features, respectively. Cluster 1 contains some front features of tracked objects, cluster 2 contains some back features of tracked objects, and cluster 3 contains some side features of tracked objects.

[0173] Step A2: The electronic device may concatenate the target feature information of the m target objects with the cluster centers of the k clusters, and input the concatenation result into the trained similarity prediction model.

[0174] During the splicing process, the electronic device may splice the target feature information of the m target objects with the k clusters respectively, and use the splicing results as input feature information.

[0175] For example, Figure 3 As shown, assuming k = 3, the electronic device can splice the target feature information of m target objects with cluster 1 to obtain splicing result 1, splice the target feature information of m target objects with cluster 2 to obtain splicing result 2, and splice the target feature information of m target objects with cluster 3 to obtain splicing result 3, and then splicing result 1, splicing result 2 and splicing result 3 constitute the input feature information.

[0176] The input feature information can be represented by a tensor of (m*n)*k*2dims.

[0177] Here, m*n represents the number of matching pairs between the target object and the tracked object. For details, please refer to the above description and will not be repeated here.

[0178] k represents the number of clusters.

[0179] 2dims represents the number of dimensions of feature information (including target feature information and historical feature information)

[0180] In addition, in order to reduce the computational complexity of the similarity prediction model, before clustering, the electronic device may first normalize the target feature information of the m target objects and the historical feature information of the n tracked objects.

[0181] For example, target feature information and historical feature information are generally represented by a multi-dimensional feature vector (such as a 64-dimensional feature vector or a 128-dimensional feature vector). The electronic device may normalize the feature vector F representing the target feature information and historical feature information in terms of length. The normalization formula is as follows:

[0182]

[0183] in, It is the value obtained after the eigenvector F is normalized in length.

[0184] Step A2: The electronic device may input the input feature information into the trained similarity prediction model.

[0185] During implementation, the electronic device may input the splicing result represented by the tensor (m*n)*k*2dims into a trained similarity prediction model.

[0186] Step 1043: The electronic device inputs the historical tracking information of the n tracking objects into the trained similarity prediction model.

[0187] In order to simplify the computational complexity of the similarity prediction model, the electronic device may pre-process the historical tracking information of the n tracking objects, such as performing normalization processing.

[0188] Specifically, the historical tracking information may include dimensions such as the track length, the number of lost frames, and the overlap rate of the tracked object, so the electronic device may normalize the dimensions in the historical tracking information.

[0189] For example, the electronic device can normalize the trajectory length l of the tracked object using the following formula:

[0190]

[0191] Among them, l 1g represents the normalized trajectory length, and Lmax represents the maximum allowed trajectory length.

[0192] For example, the electronic device can normalize the number of lost frames t of the tracked object using the following formula:

[0193]

[0194] Among them, t 1g It indicates the normalized number of lost frames, and Tmax indicates the maximum allowed number of lost frames.

[0195] Then, the electronic device may organize the normalized quantities into a (m*n)*q matrix and input the matrix into the trained similarity prediction model.

[0196] Here, m*n represents the number of matching pairs between the target object and the tracked object. For details, please refer to the above description and will not be repeated here.

[0197] q represents the number of dimensions of the tracking information. For example, the tracking information includes the track length, the number of lost frames, and the overlap ratio. If the tracking information includes the track length, the number of lost frames, and the overlap ratio, then q is 3.

[0198] Step 1044: The similarity prediction model can determine whether each target object and each tracked object are the same target through the following steps.

[0199] The similarity prediction model may refer to a supervised learning model, such as a neural network. This is merely an example of the similarity prediction model and is not specifically limited.

[0200] like Figure 4 As shown, the similarity prediction model can determine the position similarity between each target object and each tracked object based on the target position information of each target object and the predicted position information of each tracked object. The similarity prediction model can also determine the feature similarity between each target object and each tracked object based on the input feature information, and convolve the historical tracking information of each tracked object to obtain an attention probability mask.

[0201] The similarity prediction model can then fuse the position similarity and feature similarity between each target object and each tracked object, and perform a mask operation on the fusion result with the attention probability mask to obtain the similarity between each target object and each tracked object, and determine the similarity threshold based on the model parameters of the similarity prediction model.

[0202] Step 1044 is described in detail below through steps B1 to B6.

[0203] Step B1: The similarity prediction model may determine first feature information representing the position similarity between each target object and each tracked object based on the target position information of each target object and the predicted position information of each tracked object.

[0204] During implementation, as described above, the electronic device may use a (m*n)*8 matrix to represent the target position information of each target object and the predicted position information of each tracked object, and input the (m*n)*8 matrix into the similarity prediction model.

[0205] The similarity prediction model can perform at least one full connection calculation on the (m*n)*8 matrix, for example, three full connection calculations, to obtain a (m*n)*p position similarity feature matrix. The (m*n)*p position similarity feature matrix is ​​the first feature information, which can represent the position similarity between each target object and each tracked object.

[0206] Here, p represents the data element of the fully connected layer, or in other words, p represents the number of dimensions of the fully connected result.

[0207] Step B2: The similarity prediction model may also determine second feature information for representing feature similarity between each target object and each tracked object based on the target feature information of each target object and the historical feature information of each tracked object.

[0208] From the above description, it can be seen that the target feature information of each target object and the historical feature information of each tracked object are represented by the splicing result, so the similarity prediction model receives the splicing result used to represent the target feature information of each target object and the historical feature information of each tracked object.

[0209] As shown above, the splicing result can be represented by a tensor of (m*n)*k*2dims; where m*n represents the number of matching pairs between the target object and the tracked object, k represents the number of clusters, and 2dims represents the number of dimensions of the feature information.

[0210] The similarity prediction model can first perform a convolution operation on the splicing result on the dimension corresponding to 2dims to obtain a first tensor, and the first tensor is represented by (m*n)*k*p, where p represents the number of dimensions of the convolution operation result. Then, the similarity prediction model can exchange the elements on the dimension corresponding to k and the dimension corresponding to p in the first tensor to obtain a second tensor, and the second tensor is represented by (m*n)*p*k. Then, the similarity prediction model can perform a convolution operation on the second tensor on the dimension corresponding to k to obtain a tensor of (m*n)*p*1. Since the third dimension of the tensor is 1, the obtained tensor of (m*n)*p*1 is actually an apparent similarity feature matrix of (m*n)*p. The apparent similarity feature matrix of (m*n)*p is the second feature information, which can represent the feature similarity between each target object and each tracked object.

[0211] It should be noted that the similarity prediction model first performs a convolution operation on the input feature information on the dimension corresponding to 2dims, which can realize the similarity calculation of the target feature information and k cluster centers. The similarity prediction model exchanges the elements on the dimension corresponding to k and the dimension corresponding to p in the first tensor to obtain a second tensor, and performs a convolution operation on the second tensor on the dimension corresponding to k, which can realize the fusion of the target feature information and the similarity results of the k-type cluster centers. It can be seen that this method of the present application can perform similarity calculations on the target feature information and cluster centers of different categories, and overlap similar results, so the calculated feature similarity is more accurate.

[0212] Step B3: The similarity prediction model convolves the historical tracking information of each tracked object to obtain an attention probability mask.

[0213] It can be seen from the above description that the historical tracking information can be represented by a (m*n)*q matrix.

[0214] After receiving the (m*n)*q matrix, the similarity prediction model can perform a convolution operation on the (m*n)*q matrix (for example, a 1*1 convolution operation). Then, the similarity prediction model can map the convolution result to [0,1] to obtain the attention probability mask.

[0215] During mapping, the similarity prediction model may map the convolution result to [0, 1] through a sigmoid function (a type of function). This is merely an example of the mapping method and is not specifically limited thereto.

[0216] Step B4: The similarity prediction model may fuse the position similarity and feature similarity between each target object and each tracked object, and perform a mask operation on the fusion result and the attention probability mask to obtain the similarity between each target object and each tracked object.

[0217] The similarity prediction model can concatenate the position similarity and feature similarity between each target object and each tracked object, and then fuse the concatenated position similarity and feature similarity through at least one full connection.

[0218] The similarity prediction model can perform a mask operation on the fusion result and the above-mentioned attention probability mask (for example, the fusion result can be multiplied by the attention probability mask), and perform a full connection operation on the mask operation, and map the full connection operation result to the [0,1] interval, thereby obtaining the similarity between each target object and each tracked object.

[0219] The similarity between each target object and each tracked object can be represented by an m*n similarity matrix. The size of the similarity matrix is ​​m*n, and each element in the similarity matrix represents the similarity between a target object and a tracked object.

[0220] For example, Figure 4 The m*n similarity matrix shown in Figure 4 In the matrix shown, m=4, n=4, indicating that there are 4 tracking objects and 4 target objects in this example.

[0221] The element value of the first row and first column of the matrix is ​​0.11, which means that the similarity between the first target object and the first tracked object is 0.11.

[0222] Step B5: The similarity prediction model obtains the trained similarity threshold.

[0223] In this application, the similarity threshold is used as a learnable parameter of the similarity prediction model to construct the similarity prediction model, which means that the similarity threshold will also be adjusted during the training process of the similarity prediction model.

[0224] For example, during training, the similarity prediction model obtains a similarity matrix and a similarity threshold based on the input samples. The similarity prediction model then uses the similarity threshold to expand and fill the similarity matrix, obtaining the association result of each detected object and tracked object, and calculating the error between the obtained association result and the actual association result. The error is then passed back to the similarity prediction model, which adjusts the model parameters based on the error. Since the similarity threshold is a learnable parameter in the similarity prediction model, it is also adjusted during training. Therefore, the similarity threshold is a learnable parameter. The specific training process can be found in the following text, and this is just a brief description.

[0225] When obtaining the similarity threshold, the similarity prediction model may determine a model parameter representing the similarity threshold from the model parameters of the current model, and call the determined model parameter as the similarity threshold.

[0226] Of course, in practical applications, multiple parameters of the similarity prediction model may also be used to represent the similarity threshold. The similarity prediction model may obtain the similarity threshold based on operations performed on these multiple parameters according to preset rules.

[0227] Here, the method of determining the similarity threshold is merely exemplified and is not specifically limited.

[0228] It should be noted that: as can be seen from the above description, on the one hand, since the similarity prediction model takes into account multiple dimensions such as position similarity and feature similarity when calculating the similarity between each target object and each tracked object, the calculated similarity is more accurate. In addition, compared to the method of weighting the similarities of multiple dimensions (such as position similarity, feature similarity, etc.) to obtain similarity, this application uses a supervised similarity prediction model (such as a neural network) to fuse the similarities of multiple dimensions to obtain the final similarity, so that the calculation of the final similarity is not limited to linear calculation, making the calculation result more accurate.

[0229] On the other hand, when calculating the similarity, the similarity prediction model also takes into account the tracking information used to characterize the association relationship of the same object in different frames, and uses the tracking information as a mask to perform mask operations with the fused position similarity and feature similarity, which makes the similarity obtained by the prediction model more accurate.

[0230] Thirdly, the similarity threshold of the present application is no longer a manually set threshold, but the similarity threshold is used as a parameter of the similarity prediction model, which allows the similarity threshold to be continuously adjusted as the similarity prediction model is trained, making the similarity threshold a learnable similarity threshold, which is conducive to the subsequent determination of the association relationship between the target object and the tracked object.

[0231] Step B6: The electronic device determines and outputs whether each target object and each tracked object are the same target based on the trained similarity threshold and the similarity between each target object and each tracked object.

[0232] During implementation, the similarity prediction model outputs a similarity matrix representing the similarity between each target object and each tracked object. The size of the similarity matrix is ​​m*n, and each element in the similarity matrix represents the similarity between a target object and a tracked object.

[0233] The electronic device may use a similarity threshold to expand the similarity matrix, so that the expanded similarity matrix includes the similarity threshold, and based on the values ​​of the elements in each row of the expanded similarity matrix, weights are set for the elements in that row, and based on the values ​​of the elements in each row of the expanded similarity matrix, weights are set for the elements in that column. The electronic device may then perform a minimum flow calculation on the expanded similarity matrix based on the weights set for the elements in each row and column, obtain a matching value between each target object and each tracked object, and obtain an association relationship between each target object and each tracked object based on the matching value between each target object and each tracked object.

[0234] Step B6 will be described in detail below through steps B61 to B64.

[0235] Step B61: Expand the similarity matrix using the similarity threshold, so that the expanded similarity matrix includes the similarity threshold.

[0236] During implementation, the electronic device may expand the m*n similarity matrix into an M*N similarity matrix using a preset value for representing an invalid bit, where M represents the preset maximum number of detectable target objects, and N represents the preset maximum number of trackable target objects.

[0237] For example, Figure 5 As shown, Figure 5 Where a represents an m*n similarity matrix. The electronic device can set the value of each element from row m to row M and from column n to column N to 0, thereby forming an M*N similarity matrix. The M*N similarity matrix is ​​as follows: Figure 5 As shown in b.

[0238] Among them, the m*n similarity matrix is ​​called the valid area in the M*N matrix and participates in the subsequent minimum flow calculation. The area filled with 0 in the M*N matrix is ​​called the invalid area and does not participate in the subsequent minimum flow calculation.

[0239] The preset value representing the invalid bit is used to fill and expand the m*n similarity matrix into an M*N similarity matrix, mainly to make the length and width of the similarity matrix corresponding to each frame consistent.

[0240] After obtaining the M*N similarity matrix, the electronic device may use the similarity threshold to expand the M*N similarity matrix to obtain a 2M*2N similarity matrix.

[0241] For example, Figure 5As shown in FIG, when expanding, the electronic device can perform a pad operation (a filling operation) below and to the right of the M*N similarity matrix (as shown in FIG b), and the pad value is the similarity threshold. In other words, the electronic device sets the value of each element in the Mth row to the 2Mth row and the Nth column to the 2Nth column to the similarity threshold, thereby forming a 2M*2N similarity matrix (as shown in FIG b). Figure 5 (as shown in c in the figure).

[0242] Among them, the area filled with the similarity threshold in the 2M*2N matrix is ​​called the pad area, which participates in the subsequent minimum flow calculation.

[0243] Step B62: The electronic device sets weights for the elements in each row of the expanded similarity matrix based on the values ​​of the elements in the row, and sets weights for the elements in the column based on the values ​​of the elements in each row of the expanded similarity matrix.

[0244] When setting the weights, in order to allow the valid area and pad area in 2M*2N to participate in the subsequent minimum flow calculation, and to prevent the invalid area from participating in the minimum flow calculation, the electronic device can set the weights of the rows and columns of the valid area and pad area to 1, and set the weights of the rows and columns of the invalid area to 0.

[0245] During implementation, for each row element in 2M*2N, if the row element includes elements in the m*n similarity matrix, the weight of the row element is set to 1; if the row element does not include elements in the m*n similarity matrix but includes 0 and the similarity threshold, the weight of the row element is set to 0; if the row element only includes the similarity threshold, the weight of the row element is set to 1.

[0246] For each column element in 2M*2N, if the column element includes the elements in the m*n similarity matrix, the weight of the column element is set to 1; if the column element does not include the elements in the m*n similarity matrix but includes 0 and the similarity threshold, the weight of the column element is set to 0; if the column element only includes the similarity threshold, the weight of the column element is set to 1.

[0247] Step B63: The electronic device performs minimum flow calculation on the expanded similarity matrix based on the weights set for each row element and each column element to obtain a matching value between each target object and each tracked object.

[0248] The minimum flow calculation may be a Hungarian calculation or an EMD (Earth Mover's Distance) algorithm. The minimum flow algorithm is merely described here for example without any specific limitation.

[0249] like Figure 6As shown, during implementation, the electronic device may perform minimum flow calculation on the expanded 2M*2N similarity matrix based on the weights set for each row element and each column element, and obtain Figure 6 The matching value matrix shown in FIG. 1 , each element of the matching value matrix may represent a matching value between a tracked object and a target object.

[0250] It should be noted that in existing methods, after obtaining the similarity matrix between each target object and each tracked object, a minimum flow solution is first performed to obtain a matching value between each target object and each tracked object. Then, for each matching value between the target object and each tracked object, the matching value is compared with a preset threshold. If the matching value is greater than the preset threshold, the target object is determined to be associated with the tracked object. If the matching value is less than the preset threshold, the target object is determined to be unassociated with the tracked object.

[0251] The disadvantage of this is that the existing method does not consider the similarity threshold when solving the minimum flow, so the final association result is not very accurate.

[0252] This application uses a similarity threshold to expand and fill the similarity matrix, sets weights for each row and column, and performs a minimum flow calculation on the expanded and filled similarity matrix based on the weights. Since the learnable similarity threshold is considered in the minimum flow calculation, the matching results are more accurate.

[0253] Step B64: The similarity prediction model calculates the matching value between each target object and each tracking object to determine whether each target object and each tracking object are the same target (ie, whether each target object is associated with each tracking object).

[0254] During implementation, for each target object, if the matching value between the target object and any tracking object is a first preset value, it is determined that the target object and the any tracking object are the same target, that is, the target object is associated with the any tracking object;

[0255] If the target object matches all the tracked objects to the second preset value, it means that the target object and all the tracked objects are not the same target, that is, the target object is not associated with any of the tracked objects.

[0256] The first preset value may be represented by 1, and the second preset value may be represented by 0. Here, the first preset value and the second preset value are merely exemplarily described without specific limitation.

[0257] For example, as described above, the matching value between each target object and each tracking object can be obtained by Figure 6 The matching value matrix in is represented.

[0258] by Figure 6 Taking the matching value matrix in as an example, since the element in the 1st row and the 3rd column is the first preset value (ie, 1), it is determined that the first tracked object is associated with the third target object.

[0259] Since the element in the 2nd row and 4th column is 1, it is determined that the 2nd tracking object and the 4th target object are the same target, that is, the 2nd tracking object is associated with the 4th target object.

[0260] Since the element in the third row and second column is 1, it is determined that the third tracking object and the second target object are the same target, that is, the third tracking object is associated with the second target object.

[0261] Since the element in the 4th row and the 1st column is 1, it is determined that the 4th tracking object and the 1st target object are the same target, that is, the 4th tracking object is associated with the 1st target object.

[0262] Since the valid area elements in the fifth column are all 0 and the pad area elements are 1, it is determined that the fifth target object is not the same target as all the tracked objects, that is, the fifth target object is not associated with any tracked objects.

[0263] In addition, in the present application, in addition to determining the association relationship between each target object and the tracked object, the electronic device can also determine newly appeared objects in the first image and tracked objects that disappear in the first image.

[0264] Specifically, for each target object, if the target object is not associated with any tracked object, the electronic device may determine that the target object is a newly appeared object. For each tracked object, if the tracked object is not associated with any target object, the electronic device may determine that the tracked object has disappeared from the first image.

[0265] For example, Figure 6 Take the matching value matrix in as an example.

[0266] Since all elements in the valid area of ​​the 5th column are 0 and an element in the pad area of ​​the 5th column is 1, the electronic device determines that the 5th target object is not the same target as all tracked objects, and further determines that the 5th target object is a newly appeared object.

[0267] Since all elements in the valid area of ​​the 5th row are 0 and an element in the pad area of ​​the 5th row is 1, the electronic device determines that the 5th tracked object is not the same target as all target objects, and further determines that the 5th tracked object disappears from the first image.

[0268] In addition, in the present application, the electronic device may train the similarity prediction model in the following manner: wherein the similarity threshold is a learnable parameter in the similarity prediction model.

[0269] Before training, this application provides sample label pairs.

[0270] The sample includes: the position information and feature information of at least one target object identified in the Nth frame, the feature information of at least one tracked object identified from the video frame before the Nth frame, the predicted position information of each tracked object in the Nth frame, and the tracking information of each tracked object.

[0271] The label is the matching value between each target object and each tracked object.

[0272] The electronic device can input samples and labels into a similarity prediction model. The similarity prediction model can determine, based on the target location information of each target object and the predicted location information of each tracked object, first feature information used to represent the location similarity between each target object and each tracked object, and second feature information used to represent the feature similarity between each target object and each tracked object based on the target feature information of each target object and the historical feature information of each tracked object, and convolve the historical tracking information of each tracked object to obtain an attention probability mask. The similarity prediction model fuses the location similarity and feature similarity between each target object and each tracked object, and performs a mask operation on the fusion result and the attention probability mask to obtain a similarity matrix used to represent the similarity between each target object and each tracked object.

[0273] Then, the similarity prediction model may call the model parameters representing the similarity threshold and the obtained similarity matrix to obtain the matching value between each target object and each tracked object.

[0274] Specifically, the similarity prediction model can use the similarity threshold as a model parameter of the similarity prediction model to expand the similarity matrix, set row and column weights for the expanded similarity matrix, and use the set weights to perform minimum flow calculation on the expanded similarity matrix to obtain the matching value of each target object and each tracked object.

[0275] Then, the electronic device can use the cross entropy loss between the matching value and label of each target object and each tracked object obtained by calculating the similarity prediction model, and pass the cross entropy loss back to the similarity prediction model as an error, so that the similarity prediction model can adjust its own model parameters based on it to achieve the purpose of training.

[0276] During training, the similarity threshold is a learnable parameter of the similarity prediction model. Therefore, when the similarity prediction model adjusts its own model parameters, it also adjusts the similarity threshold. In other words, the similarity threshold is continuously learned as the model trains, making it a learnable similarity threshold.

[0277] From the above description, it can be seen that, on the one hand, in the present application, since a supervised similarity prediction model is used to predict the similarity between the target object and the tracked object based on the relevant information extracted from the target object and the historical tracked object, and then whether the target object and the tracked object are the same target is determined based on the similarity and the similarity threshold obtained by training with the similarity prediction model, rather than manually setting the similarity threshold and manually setting the measurement function to calculate the similarity between the target object and the tracked object, the present application determines whether the target object and the tracked object are the same target more accurately.

[0278] Secondly, when calculating the similarity between each target object and the tracked object, the similarity prediction model takes into account multiple dimensions such as position similarity and feature similarity when calculating the similarity between each target object and each tracked object, so the calculated similarity is more accurate. In addition, compared to the method of weighting the similarities of multiple dimensions (such as position similarity, feature similarity, etc.) to obtain similarity, this application uses a supervised similarity prediction model (such as a neural network) to fuse the similarities of multiple dimensions to obtain the final similarity, so that the calculation of the final similarity is not limited to linear calculation, making the calculation result more accurate.

[0279] In addition, when calculating similarity, the similarity prediction model also takes into account the tracking information used to characterize the association relationship of the same object in different frames, and uses the tracking information as a mask to perform mask operations with the fused position similarity and feature similarity, which makes the similarity obtained by the prediction model more accurate.

[0280] In addition, the similarity threshold of the present application is no longer a manually set threshold, but the similarity threshold is used as a parameter of the similarity prediction model, which allows the similarity threshold to be continuously adjusted as the similarity prediction model is trained, making the similarity threshold a learnable similarity threshold, which is conducive to the subsequent determination of the association relationship between the target object and the tracked object.

[0281] Third, when determining the association between each target object and each tracked object based on the similarity matrix and similarity threshold, this application uses the similarity threshold to expand and fill the similarity matrix, sets weight values ​​for each row and column, and performs a minimum flow calculation on the expanded and filled similarity matrix based on the weights. Since the learnable similarity threshold is considered in the minimum flow calculation, the matching results are more accurate.

[0282] See also Figure 7 , Figure 7 It is a hardware structure diagram of an electronic device shown in an exemplary embodiment of the present application.

[0283] The electronic device includes a communication interface 701, a processor 702, a machine-readable storage medium 703, and a bus 704. The communication interface 701, the processor 702, and the machine-readable storage medium 703 communicate with each other via the bus 704. The processor 702 can execute the object association method described above by reading and executing machine-executable instructions corresponding to the object association control logic in the machine-readable storage medium 703.

[0284] See also Figure 8 , Figure 8 This is a block diagram of an object association device shown in an exemplary embodiment of the present application. The device can be applied to an electronic device and may include the following units.

[0285] An acquisition unit 801 is configured to acquire an image frame sequence comprising a first image and a second image, wherein the second image is acquired earlier than the first image; acquire target position information and target feature information of a target object identified from the first image; and acquire historical position information, historical feature information, and historical tracking information of a tracked object identified from the second image.

[0286] A prediction unit 802 is configured to predict predicted position information of each tracked object in the first image based on historical position information of the tracked object;

[0287] The output unit 803 is used to input the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object and the historical feature information of the tracked object into the trained similarity prediction model to obtain whether the target object and the tracked object are the same target.

[0288] Optionally, the number of the target objects is at least one, and the number of the tracked objects is at least one;

[0289] The output unit 803 is used to input the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object and the historical feature information of the tracked object into the trained similarity prediction model to obtain whether the target object and the tracked object are the same target. The similarity prediction model determines, based on the target position information of each target object and the predicted position information of each tracked object, a similarity value representing the predicted position similarity between each target object and each tracked object. The first feature information of the similarity prediction model is obtained by combining the first feature information and the second feature information of the target object with the tracking object, and based on the target feature information of each target object and the historical feature information of each tracking object, determining the second feature information for representing the feature similarity between each target object and each tracking object, and convolving the historical tracking information of each tracking object to obtain an attention probability mask; the similarity prediction model fuses the first feature information and the second feature information between each target object and each tracking object, and performs a mask operation on the fusion result and the attention probability mask to obtain the similarity between each target object and each tracking object; the similarity prediction model determines and outputs whether each target object and each tracking object are the same target based on the trained similarity threshold and the similarity between each target object and each tracking object.

[0290] Optionally, the output unit 803, when inputting the target feature information of the target object and the historical feature information of the tracked object into a trained similarity prediction model, is used to cluster the historical feature information of the tracked object to obtain a clustering result, wherein the clustering result includes: at least one feature category, and at least one cluster cluster corresponding to each feature category; the historical feature information in each cluster cluster is matched with the feature category corresponding to the cluster cluster; the target feature information of the target object is spliced ​​with the cluster center of at least one cluster cluster, and the splicing result is input into the trained similarity prediction model.

[0291] Optionally, the number of target objects is m, the number of tracked objects is n, and the number of clusters is k;

[0292] The input feature information is represented by a tensor of (m*n)*k*2dims; where m*n represents the number of matching pairs between the target object and the tracked object, k represents the number of clusters, and 2dims represents the number of dimensions of the feature information;

[0293] The output unit, when determining the second feature information used to represent the feature similarity between each target object and each tracked object based on the target feature information of each target object and the historical feature information of each tracked object, is used to perform a convolution operation on the splicing result in the dimension corresponding to 2dims to obtain a first tensor, where the first tensor is represented by (m*n)*k*p, where p represents the number of dimensions of the convolution operation result; swap the elements in the dimension corresponding to k and the dimension corresponding to p in the first tensor to obtain a second tensor, where the second tensor is represented by (m*n)*p*k; perform a convolution operation on the second tensor in the dimension corresponding to k, and determine the obtained result as the second feature information used to represent the feature similarity between each target object and each tracked object.

[0294] Optionally, the similarity prediction model represents the similarity between each target object and each tracked object through a similarity matrix, the size of the similarity matrix is ​​m*n, and each element in the similarity matrix represents the similarity between a target object and a tracked object;

[0295] The output unit 803 is used to expand the similarity matrix using the similarity threshold when determining and outputting whether each target object and each tracked object are the same target based on the trained similarity threshold and the similarity between each target object and each tracked object, so that the expanded similarity matrix includes the similarity threshold; set weights for the row elements based on the values ​​of the elements on each row of the expanded similarity matrix, and set weights for the column elements based on the values ​​of the elements on each row of the expanded similarity matrix; perform minimum flow calculation on the expanded similarity matrix based on the weights set for each row element and each column element to obtain matching values ​​for each target object and each tracked object; and obtain and output the association relationship between each target object and each tracked object based on the matching values ​​for each target object and each tracked object.

[0296] Optionally, the number of target objects is m, the number of tracked objects is n; the size of the similarity matrix is ​​m*n;

[0297] The output unit 803 is used to expand the m*n similarity matrix to obtain an M*N similarity matrix when the similarity threshold is used to expand the similarity matrix; wherein M represents the maximum number of detectable target objects, and N represents the maximum number of trackable target objects; the M*N similarity matrix is ​​expanded using the similarity threshold to obtain a 2M*2N similarity matrix.

[0298] Optionally, the output unit 803 is used to determine whether each target object and each tracked object are the same target based on the matching value between each target object and each tracked object. For each target object, if the matching value between the target object and any tracked object is a first preset value, it is determined that the target object and any tracked object are the same target; if the matching value between the target object and all tracked objects is a second preset value, it indicates that the target object and all tracked objects are not the same target.

[0299] Optionally, the output unit 803 is further configured to, for each target object, determine that if the target object is not the same target as all the tracked objects, the target object is a newly appeared object;

[0300] For each tracked object, if the tracked object is not the same as all target objects, it is determined that the tracked object disappears in the first image.

[0301] Optionally, the similarity threshold is a model parameter of the similarity prediction model, and the similarity threshold is obtained by training the similarity prediction model.

[0302] In addition, the present application also provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, the object association method described above is implemented.

[0303] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0304] The computer-readable storage medium referred to herein can be any electronic, magnetic, optical, or other physical storage device that can contain or store information, such as executable instructions, data, and the like. For example, the machine-readable storage medium can be a volatile memory, a non-volatile memory, or a similar storage medium. Specifically, the computer-readable storage medium can be a RAM (Random Access Memory), a flash memory, a storage drive (such as a hard disk drive), a solid-state drive, any type of storage disk (such as a CD, DVD, etc.), or a similar storage medium, or a combination thereof.

[0305] In addition, the present application also provides a computer program, which is stored in a computer-readable storage medium, and when a processor executes the computer program, it prompts the processor to implement the above-mentioned object association method.

[0306] The implementation process of the functions and effects of each unit in the above-mentioned device is specifically described in the implementation process of the corresponding steps in the above-mentioned method, and will not be repeated here.

[0307] For the device embodiments, since they basically correspond to the method embodiments, the relevant parts can be referred to the partial description of the method embodiments. The device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the present application scheme. A person of ordinary skill in the art can understand and implement it without paying any creative work.

[0308] The above description is only a preferred embodiment of the present application and is not intended to limit the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principles of the present application shall be included in the scope of protection of the present application.

Claims

1. An object association method, characterized in that: The method comprises: Acquire an image frame sequence including a first image and a second image, wherein the acquisition time of the second image is earlier than the acquisition time of the first image; Acquiring target position information and target feature information of the target object identified from the first image; Acquiring historical position information, historical feature information, and historical tracking information of the tracked objects identified in the second image, and predicting the predicted position information of each tracked object in the first image based on the historical position information of the tracked object; the historical tracking information includes at least the track length, number of lost frames, and overlap rate of the tracked object; Inputting the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object and the historical feature information of the tracked object into a trained similarity prediction model to determine whether the target object and the tracked object are the same target; Wherein, when inputting the target feature information of the target object and the historical feature information of the tracked object into the trained similarity prediction model, it includes: Clustering the historical feature information of the tracked object to obtain a clustering result, the clustering result including: at least one feature category and at least one cluster corresponding to each feature category; the historical feature information in each cluster matches the feature category corresponding to the cluster; The target feature information of the target object is spliced ​​with the cluster center of each cluster, and the splicing result is input into the trained similarity prediction model.

2. The method according to claim 1, characterized in that The number of the target objects is at least one, and the number of the tracked objects is at least one; The step of inputting the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object, and the historical feature information of the tracked object into a trained similarity prediction model to determine whether the target object and the tracked object are the same target includes: Inputting the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object, and the historical feature information of the tracked object into a trained similarity prediction model; The similarity prediction model determines first feature information for representing the predicted position similarity between each target object and each tracked object based on the target position information of each target object and the predicted position information of each tracked object, and determines second feature information for representing the feature similarity between each target object and each tracked object based on the target feature information of each target object and the historical feature information of each tracked object, and convolves the historical tracking information of each tracked object to obtain an attention probability mask; The similarity prediction model fuses the first feature information and the second feature information between each target object and each tracked object, and performs a mask operation on the fusion result and the attention probability mask to obtain the similarity between each target object and each tracked object; The similarity prediction model determines and outputs whether each target object and each tracked object are the same target based on a trained similarity threshold and the similarity between each target object and each tracked object.

3. The method according to claim 2, characterized in that The number of target objects is m, the number of tracked objects is n, and the number of clusters is k; The splicing result is represented by a tensor of (m*n)*k*2dims; where m*n represents the number of matching pairs between the target object and the tracked object, k represents the number of clusters, and 2dims represents the number of dimensions of the feature information; The determining, based on the target feature information of each target object and the historical feature information of each tracked object, second feature information for indicating feature similarity between each target object and each tracked object includes: Perform a convolution operation on the concatenated result in a dimension corresponding to 2dims to obtain a first tensor, where the first tensor is represented by (m*n)*k*p, where p represents the number of dimensions of the convolution operation result; Swap the elements of the dimension corresponding to k and the dimension corresponding to p in the first tensor to obtain a second tensor, which is represented by (m*n)*p*k; A convolution operation is performed on the second tensor in the dimension corresponding to k, and the obtained result is determined as second feature information for representing the feature similarity between each target object and each tracked object.

4. The method according to claim 2, characterized in that The similarity prediction model represents the similarity between each target object and each tracked object through a similarity matrix, and each element in the similarity matrix represents the similarity between a target object and a tracked object; The determining and outputting whether each target object and each tracked object are the same target based on the trained similarity threshold and the similarity between each target object and each tracked object includes: Expanding the similarity matrix using the similarity threshold so that the expanded similarity matrix includes the similarity threshold; Based on each row of the expanded similarity matrix, a weight is set for the elements in the row based on the values ​​of the elements in the row, and based on each column of the expanded similarity matrix, a weight is set for the elements in the column based on the values ​​of the elements in the column; Perform minimum flow calculation on the expanded similarity matrix based on the weights set for each row element and each column element to obtain the matching value between each target object and each tracked object; Based on the matching value between each target object and each tracked object, it is determined whether each target object and each tracked object are the same target and outputted.

5. The method according to claim 4, characterized in that The number of target objects is m, the number of tracked objects is n; the size of the similarity matrix is ​​m*n; The step of expanding the similarity matrix by using the similarity threshold comprises: The m*n similarity matrix is ​​expanded to obtain an M*N similarity matrix; where M represents the maximum number of detectable target objects and N represents the maximum number of trackable target objects; The similarity threshold is used to expand the M*N similarity matrix to obtain a 2M*2N similarity matrix.

6. The method according to claim 4, characterized in that The step of determining whether each target object and each tracked object are the same target based on the matching value between each target object and each tracked object includes: For each target object, if the matching value between the target object and any tracked object is a first preset value, then the target object is determined to be the same target as the tracked object; If the target object and all the tracked objects match to the second preset value, it means that the target object and all the tracked objects are not the same target.

7. The method according to claim 6, characterized in that The method further comprises: For each target object, if the target object is not the same as any of the tracked objects, the target object is determined to be a newly appeared object; For each tracked object, if the tracked object is not the same as all target objects, it is determined that the tracked object disappears in the first image.

8. The method according to claim 2, characterized in that The similarity threshold is a model parameter of the similarity prediction model, and the similarity threshold is obtained during the training of the similarity prediction model.

9. An object association device, characterized in that: The device comprises: an acquisition unit, configured to acquire an image frame sequence comprising a first image and a second image, wherein the acquisition time of the second image is earlier than the acquisition time of the first image; acquire target position information and target feature information of a target object identified from the first image; and acquire historical position information, historical feature information, and historical tracking information of a tracked object identified from the second image; a prediction unit, configured to predict predicted position information of each tracked object in the first image based on historical position information of the tracked object; an output unit, configured to input the target position information of the target object, the predicted position information of the tracked object, the historical tracking information, the target feature information of the target object, and the historical feature information of the tracked object into a trained similarity prediction model to determine whether the target object and the tracked object are the same target; the historical tracking information at least includes the track length, the number of lost frames, and the overlap rate of the tracked object; The output unit is used to cluster the historical feature information of the tracked object to obtain a clustering result when the target feature information of the target object and the historical feature information of the tracked object are input into a trained similarity prediction model. The clustering result includes: at least one feature category and at least one cluster cluster corresponding to each feature category; the historical feature information in each cluster cluster is matched with the feature category corresponding to the cluster cluster; the target feature information of the target object is spliced ​​with the cluster center of each cluster cluster, and the splicing result is input into the trained similarity prediction model.

10. An electronic device, characterized in that: The electronic device includes a readable storage medium and a processor; Wherein, the readable storage medium is used to store machine-executable instructions; The processor is configured to read the machine-executable instructions on the readable storage medium and execute the instructions to implement the steps of any one of the methods of claims 1-8.

Images