A video-based multi-detection box processing method and related apparatus

Abstract

The application discloses a video-based multi-detection frame processing method and a related device. The method comprises the following steps: obtaining N current detection frame feature vectors corresponding to N current detection frames in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories, wherein M and N are positive integers; performing difference processing on the N current detection frame feature vectors and the M trajectory feature vectors to obtain N*M difference feature vectors to form a difference feature matrix; performing splicing processing on M historical trajectory feature matrices corresponding to the M historical trajectories to obtain a splicing feature matrix, wherein the historical trajectory feature matrix comprises a plurality of historical detection frame feature vectors corresponding to a plurality of historical detection frames in the historical trajectory; performing binary classification processing on the difference feature matrix and the splicing feature matrix based on an attention mechanism through a preset decoder to obtain a matching matrix; and determining the association relationship between the N current detection frames and the M historical trajectories according to the matching matrix, thereby improving the video multi-target tracking accuracy.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and particularly relates to a multi-detection frame processing method based on a video and a related device. BACKGROUND

[0002] Video multi-target tracking refers to positioning and labeling multiple targets in a video, one-to-one correspondence between targets in different video frames, the same target maintaining the same label, and then forming a target trajectory. At present, with the rapid development of detection algorithms, multi-target tracking methods based on detection are widely used in video multi-target tracking.

[0003] In related technologies, the multi-target tracking method based on detection refers to obtaining appearance information and spatio-temporal information of multiple current detection frames in a video through a detector, constructing feature vectors of the multiple current detection frames, performing multi-classification processing on feature vectors of multiple historical trajectories corresponding to the multiple targets tracked in the video, obtaining a multi-classification processing result, and realizing video multi-target tracking of the multiple current detection frames corresponding to the multiple historical trajectories.

[0004] However, the above multi-target tracking method based on detection only focuses on the local relationship between the current detection frame and the historical trajectory, while the feature vectors of the multiple current detection frames have similarities, and the feature vectors of the multiple historical trajectories also have similarities, resulting in an inaccurate classification processing result, thereby reducing the accuracy of video multi-target tracking of the multiple current detection frames corresponding to the multiple historical trajectories. SUMMARY

[0005] Therefore, the embodiments of the present application provide a multi-detection frame processing method based on a video and a related device, so that the classification processing result is more accurate, the association relationship between N current detection frames and M historical trajectories is determined, and the accuracy of video multi-target tracking of the multiple current detection frames corresponding to the multiple historical trajectories is improved.

[0006] In a first aspect, the embodiments of the present application provide a multi-detection frame processing method based on a video, and the method comprises the following steps:

[0007] obtaining N current detection frame feature vectors corresponding to N current detection frames in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories; M and N are positive integers;

[0008] performing difference processing on the N current detection frame feature vectors and the M trajectory feature vectors to obtain N*M difference feature vectors to form a difference feature matrix;

[0009] The M historical trajectory feature matrices corresponding to the M historical trajectories are spliced to obtain a spliced feature matrix; the historical trajectory feature matrix includes a plurality of historical bounding box feature vectors corresponding to a plurality of historical bounding boxes in the historical trajectory;

[0010] The difference feature matrix and the spliced feature matrix are subjected to binary classification processing based on an attention mechanism through a preset decoder to obtain a matching matrix;

[0011] According to the matching matrix, an association relationship between the N current bounding boxes and the M historical trajectories is determined.

[0012] Optionally, the preset decoder includes a preset edge decoding layer, and the preset edge decoding layer includes a first self-attention module, a cross-attention module, and a first feedforward network module; the binary classification processing of the difference feature matrix and the spliced feature matrix based on the attention mechanism through the preset decoder to obtain the matching matrix includes:

[0013] The difference feature matrix is subjected to self-attention calculation processing through the first self-attention module to obtain a processed difference feature matrix;

[0014] The processed difference feature matrix and the spliced feature matrix are subjected to cross-attention calculation processing through the cross-attention module to obtain a cross feature matrix;

[0015] The cross feature matrix is subjected to binary classification processing through the first feedforward network module to obtain the matching matrix.

[0016] Optionally, the current bounding box feature vector is obtained by:

[0017] The appearance information and the space-time information of the current bounding box are subjected to encoding processing to obtain an appearance feature vector and a space-time feature vector of the current bounding box;

[0018] The appearance feature vector and the space-time feature vector of the current bounding box are subjected to summation processing to obtain the current bounding box feature vector.

[0019] Optionally, the plurality of historical bounding boxes in the historical trajectory are sorted in time sequence, and the trajectory feature vector is obtained by:

[0020] The appearance information of the plurality of historical bounding boxes and the space-time information of the last historical bounding box sorted are subjected to encoding processing to obtain an appearance feature vector of the plurality of historical bounding boxes and a space-time feature vector of the last historical bounding box;

[0021] The appearance feature vectors of the plurality of historical bounding boxes are subjected to mean value processing to obtain a mean value feature vector.

[0022] summing the mean feature vector and a spatiotemporal feature vector of a last historical bounding box to obtain the trajectory feature vector.

[0023] Optionally, the method further comprises:

[0024] optimizing, by a preset encoder, M historical trajectory feature matrices corresponding to the M historical trajectories based on an attention mechanism to obtain M optimized trajectory feature matrices;

[0025] the splicing processing of the M historical trajectory feature matrices to obtain the spliced feature matrix comprises:

[0026] splicing the M optimized trajectory feature matrices to obtain the spliced feature matrix.

[0027] Optionally, the preset encoder comprises a preset edge encoding layer, and the preset edge encoding layer comprises a second self-attention module and a second feedforward network module; the optimizing, by the preset encoder, of the M historical trajectory feature matrices based on the attention mechanism to obtain the M optimized trajectory feature matrices comprises:

[0028] for each historical trajectory feature matrix, performing self-attention calculation processing on the historical trajectory feature matrix by the second self-attention module to obtain a processed historical trajectory feature matrix;

[0029] optimizing, by the second feedforward network module, the processed historical trajectory feature matrix to obtain the optimized trajectory feature matrix.

[0030] In a second aspect, an embodiment of the present application provides a video-based multi-bounding box processing device, which comprises an obtaining unit, a difference unit, a splicing unit, a classification unit and a determining unit.

[0031] The obtaining unit is configured to obtain N current bounding box feature vectors corresponding to N current bounding boxes in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories; M and N are positive integers.

[0032] The difference unit is configured to perform difference processing on the N current bounding box feature vectors and the M trajectory feature vectors to obtain N×M difference feature vectors to form a difference feature matrix.

[0033] The splicing unit is configured to perform splicing processing on M historical trajectory feature matrices corresponding to the M historical trajectories to obtain a spliced feature matrix; the historical trajectory feature matrix comprises a plurality of historical bounding box feature vectors corresponding to a plurality of historical bounding boxes in the historical trajectory.

[0034] The classification unit is configured to perform binary classification processing on the difference feature matrix and the spliced feature matrix based on an attention mechanism through a preset decoder to obtain a matching matrix.

[0035] The determination unit is configured to determine an association relationship between the N current detection boxes and the M historical trajectories according to the matching matrix.

[0036] Optionally, the preset decoder comprises a preset edge decoding layer, and the preset edge decoding layer comprises a first self-attention module, a cross-attention module and a first feedforward network module.

[0037] The first self-attention module is configured to perform self-attention calculation processing on the difference feature matrix to obtain a processed difference feature matrix.

[0038] The cross-attention module is configured to perform cross-attention calculation processing on the processed difference feature matrix and the spliced feature matrix to obtain a cross feature matrix.

[0039] The first feedforward network module is configured to perform binary classification processing on the cross feature matrix to obtain the matching matrix.

[0040] Optionally, the obtaining unit is further configured to:

[0041] encode the appearance information and the space-time information of the current detection box to obtain an appearance feature vector and a space-time feature vector of the current detection box;

[0042] perform sum processing on the appearance feature vector and the space-time feature vector of the current detection box to obtain a current detection box feature vector.

[0043] Optionally, the historical trajectories are sorted in time sequence, and the obtaining unit is further configured to:

[0044] encode the appearance information of the plurality of historical detection boxes and the space-time information of the last historical detection box sorted to obtain an appearance feature vector of the plurality of historical detection boxes and a space-time feature vector of the last historical detection box;

[0045] perform mean processing on the appearance feature vectors of the plurality of historical detection boxes to obtain a mean feature vector;

[0046] perform sum processing on the mean feature vector and the space-time feature vector of the last historical detection box to obtain a trajectory feature vector.

[0047] Optionally, the device further comprises an optimization unit.

[0048] An optimization unit is configured to perform attention mechanism-based optimization processing on the M historical trajectory feature matrices by using a preset encoder to obtain M optimized trajectory feature matrices;

[0049] A splicing unit is configured to:

[0050] splicing the M optimized trajectory feature matrices to obtain a spliced feature matrix.

[0051] Optionally, the preset encoder comprises a preset edge encoding layer, and the preset edge encoding layer comprises a second self-attention module and a second feedforward network module; the optimization unit is configured to:

[0052] for each historical trajectory feature matrix, perform self-attention calculation processing on the historical trajectory feature matrix by using the second self-attention module to obtain a processed historical trajectory feature matrix;

[0053] perform optimization processing on the processed historical trajectory feature matrix by using the second feedforward network module to obtain an optimized trajectory feature matrix.

[0054] In a third aspect, an embodiment of the present application provides a computer device, which comprises a processor and a memory:

[0055] The memory is configured to store program code and transmit the program code to the processor;

[0056] The processor is configured to execute the video-based multi-detection box processing method according to the instructions in the program code.

[0057] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, which is configured to store program code, and the program code is configured to be executed by a processor to execute the video-based multi-detection box processing method.

[0058] Compared with the prior art, the present application has at least the following advantages:

[0059] By adopting the technical solutions of the embodiments of the present application, firstly, N current detection box feature vectors corresponding to N current detection boxes in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories are obtained, M and N are positive integers; secondly, the N current detection box feature vectors and the M trajectory feature vectors are subjected to difference processing to obtain N*M difference feature vectors to form a difference feature matrix; the M historical trajectory feature matrices corresponding to the M historical trajectories are subjected to splicing processing to obtain a splicing feature matrix, the historical trajectory feature matrix comprises a plurality of historical detection box feature vectors corresponding to a plurality of historical detection boxes in the historical trajectory; then, the difference feature matrix and the splicing feature matrix are subjected to binary classification processing based on an attention mechanism through a preset decoder to obtain a matching matrix; finally, the matching matrix is used to determine the association relationship between the N current detection boxes and the M historical trajectories.

[0060] It can be seen that by constructing the difference between the N current detection box feature vectors and the M trajectory feature vectors to form the difference feature matrix, and performing binary classification processing on the splicing feature matrix obtained by splicing the M historical trajectory feature matrices, the matching matrix representing the overall relationship between the N current detection boxes and the M historical trajectories is obtained, so that the classification processing result is more accurate, and the association relationship between the N current detection boxes and the M historical trajectories is facilitated to be determined, thereby improving the accuracy of video multi-target tracking of a plurality of current detection boxes corresponding to a plurality of historical trajectories. BRIEF DESCRIPTION OF DRAWINGS

[0061] In order to more clearly illustrate the technical solutions of the embodiments of the present application, the drawings required to be used in the description of the embodiments of the present application will be briefly introduced. Obviously, the drawings in the following description are only some embodiments described in the present application, and other drawings can be obtained by those skilled in the art without creative labor.

[0062] Figure 1 The system framework schematic diagram involved in an application scenario in the embodiments of the present application;

[0063] Figure 2 The flowchart of a video-based multi-detection box processing method provided by the embodiments of the present application;

[0064] Figure 3 The schematic diagram of a preset decoder provided by the embodiments of the present application;

[0065] Figure 4 The schematic diagram of a preset edge decoding layer provided by the embodiments of the present application;

[0066] Figure 5 The schematic diagram of a preset encoder provided by the embodiments of the present application;

[0067] Figure 6 A preset edge coding layer diagram provided by an embodiment of the present application;

[0068] Figure 7 An edge coding diagram provided by an embodiment of the present application;

[0069] Figure 8 A structure diagram of a video-based multi-detection frame processing device provided by an embodiment of the present application. DETAILED DESCRIPTION

[0070] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only some of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work fall within the scope of protection of the present application.

[0071] In the related art, the multi-target tracking method based on detection refers to: obtaining appearance information and space-time information of multiple current detection frames in a video through a detector, constructing feature vectors of the multiple current detection frames, performing multi-classification processing on feature vectors of multiple historical trajectories corresponding to the multiple targets tracked in the video, obtaining a multi-classification processing result, and realizing video multi-target tracking of the multiple current detection frames corresponding to the multiple historical trajectories.

[0072] However, the inventors have found through research that the multi-target tracking method based on detection only focuses on the local relationship between the current detection frame and the historical trajectory, while there is similarity between the feature vectors of the multiple current detection frames and there is also similarity between the feature vectors of the multiple historical trajectories, resulting in an inaccurate classification processing result, thereby reducing the accuracy of the video multi-target tracking of the multiple current detection frames corresponding to the multiple historical trajectories.

[0073] To solve this problem, in the embodiments of the present application, a difference feature matrix is constructed by constructing the difference between each two of N current detection frame feature vectors and M trajectory feature vectors, and a matching matrix representing the overall relationship between the N current detection frames and the M historical trajectories is obtained by performing two-classification processing on a spliced feature matrix obtained by splicing M historical trajectory feature matrices, so that the classification processing result is more accurate, the association relationship between the N current detection frames and the M historical trajectories is determined, and the accuracy of the video multi-target tracking of the multiple current detection frames corresponding to the multiple historical trajectories is improved.

[0074] For example, one of the scenarios of the embodiments of the present application can be applied to, for example, a video-based multi-target tracking method based on detection. Figure 1The scenario shown. The scenario includes terminal device 101 and server 102, and terminal device 101 interacts with server 102. Terminal device 101 determines to send a to-be-processed video to server 102 for video multi-target tracking. After server 102 has tracked M historical trajectories for the to-be-processed video and detected N current detection boxes for the to-be-processed video, the embodiment provided in the present application is executed to determine the association relationship between the N current detection boxes and the M historical trajectories, and video multi-target tracking of the N current detection boxes corresponding to the M historical trajectories is realized.

[0075] First, in the above application scenario, although the action description of the embodiment provided in the present application is executed by server 102; however, the present application is not limited in the execution subject, as long as the action disclosed in the embodiment provided in the present application is executed.

[0076] Secondly, the above scenario is only one scenario example provided in the present application, and the present application is not limited to this scenario.

[0077] The specific implementation of the multi-detection box processing method based on video and the related device in the present application will be described in detail in the embodiments below in combination with the drawings.

[0078] Exemplary method

[0079] Referring to Figure 2 , a flowchart of a multi-detection box processing method based on video in the present application is shown. In the present embodiment, the method may, for example, include the following steps:

[0080] Step 201: obtaining N current detection box feature vectors corresponding to N current detection boxes in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories; M and N are positive integers.

[0081] In the present application, after N current detection boxes are obtained by detecting a current video frame of the to-be-processed video by a detector, N current detection box feature vectors corresponding to the N current detection boxes need to be obtained. Since the to-be-processed video multi-target tracking refers to determining the association relationship between the N current detection boxes and the M historical trajectories corresponding to the multiple targets tracked by the to-be-processed video; therefore, on the basis of the N current detection box feature vectors, M trajectory feature vectors corresponding to the M historical trajectories also need to be obtained.

[0082] In the current detection frame feature vector, the appearance information of the current detection frame includes appearance and size information, and the spatio-temporal information of the current detection frame includes position and speed information. Therefore, the current detection frame feature vector needs to represent the appearance information and the spatio-temporal information of the current detection frame. Specifically, in the process of obtaining the current detection frame feature vector of the current detection frame, the appearance information of the current detection frame is first encoded to obtain the appearance feature vector of the current detection frame, and the spatio-temporal information of the current detection frame is encoded to obtain the spatio-temporal feature vector of the current detection frame. Then, the appearance feature vector of the current detection frame is added to the spatio-temporal feature vector of the current detection frame, so as to obtain the current detection frame feature vector of the current detection frame. That is, in an optional embodiment of the present application, the step of obtaining the current detection frame feature vector in step 201 may, for example, include the following steps A-B:

[0083] Step A: encode the appearance information and the spatio-temporal information of the current detection frame to obtain the appearance feature vector and the spatio-temporal feature vector of the current detection frame.

[0084] Step B: sum the appearance feature vector and the spatio-temporal feature vector of the current detection frame to obtain the current detection frame feature vector.

[0085] As an example, the calculation formula of the current detection frame feature vector of the current detection frame is as follows:

[0086] f j =a j +p j

[0087] wherein f j represents the current detection frame feature vector of the jth current detection frame, a j represents the appearance feature vector of the jth current detection frame, and p j represents the spatio-temporal feature vector of the jth current detection frame.

[0088] In the historical trajectory, since the historical trajectories include multiple historical bounding boxes, it can be similarly concluded that the historical bounding boxes have two types of attribute information, i.e., appearance information and spatio-temporal information; therefore, the trajectory feature vector of the historical trajectory is determined by the appearance information and the spatio-temporal information of the multiple historical bounding boxes. Considering that the spatio-temporal information is not additive and the spatio-temporal information has continuity, i.e., the spatial information of two bounding boxes in a relatively short time is more likely to be consistent; based on this, in the process of obtaining the trajectory feature vector of the historical trajectory under the condition that the multiple historical bounding boxes in the historical trajectory are sorted in time sequence, the appearance information of the multiple historical bounding boxes can be encoded first to obtain the appearance feature vectors of the multiple historical bounding boxes, and the spatio-temporal information of the last historical bounding box after sorting can be encoded to obtain the spatio-temporal feature vector of the last historical bounding box; then the average value of the appearance feature vectors of the multiple historical bounding boxes is calculated to obtain the mean feature vector, and the mean feature vector is added to the spatio-temporal feature vector of the last historical bounding box, so as to obtain the trajectory feature vector of the historical trajectory. That is, in an optional implementation of the embodiment of the present application, the multiple historical bounding boxes in the historical trajectory are sorted in time sequence, and the obtaining step of the trajectory feature vector in step 201 may, for example, include the following steps C-E:

[0089] Step C: The appearance information of the multiple historical bounding boxes and the spatio-temporal information of the last historical bounding box after sorting are encoded to obtain the appearance feature vectors of the multiple historical bounding boxes and the spatio-temporal feature vector of the last historical bounding box.

[0090] Step D: The appearance feature vectors of the multiple historical bounding boxes are averaged to obtain the mean feature vector.

[0091] Step E: The mean feature vector and the spatio-temporal feature vector of the last historical bounding box are summed to obtain the trajectory feature vector.

[0092] As an example, the calculation formula of the trajectory feature vector of the historical trajectory is as follows:

[0093]

[0094] wherein f i represents the trajectory feature vector of the i-th historical trajectory, a t-k represents the appearance feature vector of the historical bounding box k frames away from the current frame (the T-th frame) in the i-th historical trajectory, p t-1 represents the spatio-temporal feature vector of the historical bounding box 1 frame away from the current frame (the T-th frame) in the i-th historical trajectory, i.e., the spatio-temporal feature vector of the last historical bounding box in the i-th historical trajectory.

[0095] Step 202: Difference processing is performed on the N current bounding box feature vectors and the M trajectory feature vectors to obtain NXM difference feature vectors to form a difference feature matrix.

[0096] In the related art, the N current detection box feature vectors and the M trajectory feature vectors are directly subjected to multi-classification processing. Since the N current detection box feature vectors have similarities between them, the M trajectory feature vectors also have similarities between them, and only the local relationship between the current detection box and the historical trajectory is concerned, the classification processing result is not accurate enough, thereby reducing the accuracy of video multi-target tracking of multiple current detection boxes corresponding to multiple historical trajectories.

[0097] Therefore, in the embodiments of the present application, in order to reduce the influence of the similarities between the N current detection box feature vectors and the similarities between the M trajectory feature vectors on the classification processing result, and avoid only focusing on the local relationship between the current detection box and the historical trajectory, a difference feature matrix can be constructed between the current detection box feature vector and the trajectory feature vector, so that the difference feature matrix is used instead of the N current detection box feature vectors in subsequent classification processing.

[0098] Specifically, for the N current detection box feature vectors and the M trajectory feature vectors, the difference between each current detection box feature vector and each trajectory feature vector is calculated to obtain N*M difference feature vectors, so that the N*M difference feature vectors form a difference feature matrix.

[0099] As an example, on the basis of the above example, the calculation formula of the difference feature vector is as follows:

[0100] q ij =f j -f i

[0101] Wherein, q ij represents the difference feature vector of the current detection box feature vector of the jth current detection box and the trajectory feature vector of the ith historical trajectory.

[0102] Step 203: performing splicing processing on the M historical trajectory feature matrices corresponding to the M historical trajectories to obtain a spliced feature matrix; the historical trajectory feature matrix includes a plurality of historical detection box feature vectors corresponding to a plurality of historical detection boxes in the historical trajectory.

[0103] Corresponding to the difference feature matrix of step 202, M historical trajectory feature matrices corresponding to M historical trajectories are also needed to be spliced to obtain a spliced feature matrix, wherein the historical trajectory feature matrix is formed by a plurality of historical detection box feature vectors corresponding to a plurality of historical detection boxes in the historical trajectory.

[0104] It should be noted that in the embodiments of the present application, the execution order of steps 202 and 203 is not limited, that is, step 202 can be executed first and then step 203 can be executed, or step 203 can be executed first and then step 202 can be executed, or steps 202 and 203 can be executed simultaneously.

[0105] Step 204: performing attention mechanism-based binary classification processing on the difference feature matrix and the spliced feature matrix by a preset decoder to obtain a matching matrix.

[0106] In the embodiments of the present application, after obtaining the difference feature matrix and the spliced feature matrix in steps 201-203, it is necessary to perform attention mechanism-based binary classification processing on the difference feature matrix and the spliced feature matrix by a preset decoder to obtain a matching matrix representing the matching confidence between each current detection box and each historical trajectory in N current detection boxes and M historical trajectories.

[0107] In the implementation of step 204, the preset decoder may, for example, include a preset edge decoding layer, and the preset edge decoding layer may, for example, include a first self-attention module, a cross-attention module, and a first feedforward network module. The first self-attention module is used to perform self-attention calculation processing on the difference feature matrix to obtain a processed difference feature matrix, thereby optimizing the difference feature matrix; the cross-attention module is used to perform cross-attention calculation processing on the processed difference feature matrix and the spliced feature matrix to obtain a cross feature matrix, thereby realizing the interaction of the processed difference feature matrix and the spliced feature matrix; and the first feedforward network module is used to perform binary classification processing on the cross feature matrix to obtain the matching confidence between each current detection box and each historical trajectory to form a matching matrix. Therefore, in an optional implementation manner of the embodiments of the present application, step 204 may, for example, include steps 2041-2043 as follows:

[0108] Step 2041: performing self-attention calculation processing on the difference feature matrix by the first self-attention module to obtain a processed difference feature matrix.

[0109] Step 2042: performing cross-attention calculation processing on the processed difference feature matrix and the spliced feature matrix by the cross-attention module to obtain a cross feature matrix.

[0110] In the self-attention calculation processing and the cross-attention calculation processing, the following calculation formula is used:

[0111]

[0112] E=spt(Q,K)

[0113] Attn(H)=softmax(A+E)V

[0114] In the self-attention calculation process, Q, K and V are all difference feature matrices; in the cross-attention calculation process, Q is a difference feature matrix, and K and V are both concatenation feature matrices.

[0115] Step 2043: performing binary classification processing on the cross feature matrix through the first feedforward network module to obtain a matching matrix.

[0116] As an example, a schematic diagram of a preset decoder is shown in Figure 3 The preset decoder includes three preset edge decoding layers. The difference feature matrix and the concatenation feature matrix are input into the preset decoder for binary classification processing based on the attention mechanism, and the matching matrix is output.

[0117] As an example, a schematic diagram of a preset edge decoding layer is shown in Figure 4 The preset edge decoding layer includes a first self-attention module, a cross-attention module and a first feedforward network module. The first self-attention module includes a self-attention layer, an exit layer and a normalization layer. The cross-attention module includes a multi-head attention layer, an exit layer and a normalization layer. The first feedforward network module includes a full connection layer, a function layer, a full connection layer, an exit layer and a normalization layer.

[0118] In the implementation of step 2041, the difference feature matrix is input into the self-attention layer for self-attention calculation processing, and the first feature matrix is output. The first feature matrix is input into the exit layer to randomly lose part of the features, and the second feature matrix is output. The difference feature matrix and the second feature matrix are summed to obtain the third feature matrix. The third feature matrix is input into the normalization layer for normalization processing, and the processed difference feature matrix is output.

[0119] In the implementation of step 2042, the processed difference feature matrix and the concatenation feature matrix are input into the multi-head attention layer for cross-attention calculation processing, and the fourth feature matrix is output. The fourth feature matrix is input into the exit layer to randomly lose part of the features, and the fifth feature matrix is output. The processed difference feature matrix and the fifth feature matrix are summed to obtain the sixth feature matrix. The sixth feature matrix is input into the normalization layer for normalization processing, and the seventh feature matrix is output.

[0120] In the implementation of step 2043, the seventh feature matrix is input into the full connection layer, the function layer, the full connection layer and the exit layer, and the eighth feature matrix is output. The seventh feature matrix and the eighth feature matrix are summed to obtain the ninth feature matrix. The ninth feature matrix is input into the normalization layer for normalization processing, and the tenth feature matrix is output. The tenth feature matrix is input into the two preset edge decoding layers again, and the matching matrix is output.

[0121] Step 205: According to the matching matrix, determine the association relationship between the N current detection boxes and the M historical trajectories.

[0122] In the embodiment of the present application, after obtaining the matching matrix in step 204, since the matching matrix represents the matching confidence between each current detection box and each historical trajectory in the N current detection boxes and the M historical trajectories, the association relationship between the N current detection boxes and the M historical trajectories can be determined through the matching matrix, realizing the video multi-target tracking of the N current detection boxes corresponding to the M historical trajectories.

[0123] In addition, in the embodiment of the present application, since there are factors such as detection box occlusion, target conversion, target motion, and light condition in the process of video multi-target tracking, the appearance information and the space-time information of the multiple historical detection boxes in the historical trajectory are not stable, and therefore the M historical trajectory feature matrices corresponding to the M historical trajectories are not stable. Therefore, the M optimized trajectory feature matrices can also be obtained by performing optimization processing on the M historical trajectory feature matrices corresponding to the M historical trajectories based on the attention mechanism through the preset encoder, so as to reduce the influence of the appearance information and the space-time information of the unreasonable or low-quality historical detection box on the subsequent processing. That is, in an optional implementation manner of the embodiment of the present application, the method can further include the following step F: obtaining the M optimized trajectory feature matrices by performing optimization processing on the M historical trajectory feature matrices based on the attention mechanism through the preset encoder; correspondingly, step 203 can include: performing splicing processing on the M optimized trajectory feature matrices to obtain the spliced feature matrix.

[0124] In the implementation of step F, the preset decoder can include a preset edge coding layer, and the preset edge coding layer can include a second self-attention module and a second feedforward network module. The second self-attention module is used for performing self-attention calculation processing on the historical trajectory feature matrix to obtain a processed historical trajectory feature matrix, and the second feedforward network module is used for processing the optimized historical trajectory feature matrix to obtain the optimized trajectory feature matrix, thereby realizing the optimization of the historical trajectory feature matrix. Therefore, in an optional implementation manner of the embodiment of the present application, step F can include the following steps F1-F2:

[0125] Step F1: For each historical trajectory feature matrix, perform self-attention calculation processing on the historical trajectory feature matrix through the second self-attention module to obtain a processed historical trajectory feature matrix.

[0126] Step F2: Perform optimization processing on the processed historical trajectory feature matrix through the second feedforward network module to obtain an optimized trajectory feature matrix.

[0127] As an example, as shown in FIG. 6, the second self-attention module can include a first multi-head self-attention layer and a second multi-head self-attention layer. Figure 5A schematic diagram of a preset encoder is shown. The preset encoder includes three preset edge encoding layers, and M historical trajectory feature matrices are input into the preset encoder for attention mechanism-based optimization processing, and M optimized trajectory feature matrices are output.

[0128] As an example, as shown in Figure 6 A schematic diagram of a preset edge encoding layer is shown. The preset edge encoding layer includes a second self-attention module and a second feedforward network module. The second self-attention module includes a self-attention layer, an exit layer, and a normalization layer. The second feedforward network module includes a fully connected layer, a function layer, a fully connected layer, an exit layer, and a normalization layer.

[0129] In the implementation of step F1, the historical trajectory feature matrix is input into the self-attention layer for self-attention calculation processing, and a first trajectory feature matrix is output. The first trajectory feature matrix is input into the exit layer to randomly lose part of the features, and a second trajectory feature matrix is output. The historical trajectory feature matrix and the second trajectory feature matrix are summed to obtain a third trajectory feature matrix. The third trajectory feature matrix is input into the normalization layer for normalization processing, and the processed historical trajectory feature matrix is output.

[0130] In the implementation of step F2, the processed historical trajectory feature matrix is input into the fully connected layer, the function layer, the fully connected layer, and the exit layer, and a fourth trajectory feature matrix is output. The third trajectory feature matrix and the fourth trajectory feature matrix are summed to obtain a fifth trajectory feature matrix. The fifth trajectory feature matrix is input into the normalization layer for normalization processing, and a sixth trajectory feature matrix is output. The sixth trajectory feature matrix is input into the two preset edge encoding layers again, and an optimized trajectory feature matrix is output.

[0131] In the self-attention calculation processing, the following calculation formula is also used:

[0132]

[0133] E=spt(Q,K)

[0134] Attn(H)=softmax(A+E)V

[0135] In the self-attention calculation process, Q, K and V are all historical trajectory feature matrices; in order to describe the relative position relationship between the detection boxes, two methods of relative position encoding and absolute position encoding are used. Among them, the absolute position encoding converts the absolute position information into a feature vector and adds it to the features of the basic unit, while the relative position encoding obtains the corresponding weight according to the distance between the two detection boxes and adds it to the attention weight obtained after the dot product. However, in the process of video multi-target tracking, the spatio-temporal information of the detection box has continuity, and it is difficult to use the absolute position encoding and the relative position encoding scheme which serves the discrete method; in order to better describe the relative relationship between the detection boxes, a scheme of encoding the edges is proposed, and the process is as shown in a kind of edge encoding schematic diagram Figure 7 Fc is a full connection layer with an input of 5 dimensions and an output of 1 dimension, t represents the frame where the detection box is located, (x, y) represents the pixel coordinates of the top left corner of the detection box, and (w, h) represents the width and height of the detection box. The matrix of the relative relationship between the two detection boxes in time, space, appearance and size obtains a number used to describe the relative weight of the two detection boxes after the Fc layer. The weight is added to the attention weight obtained by the dot product for measuring the appearance information of the two detection boxes, to complete the fusion of the spatio-temporal information and the appearance information. The i-th row of A+E represents the relative weight of the i-th detection box in Q with respect to all detection boxes in K, and after softmax, the weight of each row is equal to 1. Dot product operation is performed with V, that is, the features of all detection boxes in V are multiplied by the corresponding relative weight, and finally summed.

[0136] Through various embodiments provided by the embodiment, first, N current detection box feature vectors corresponding to N current detection boxes in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories are obtained, M and N are positive integers; second, the N current detection box feature vectors and the M trajectory feature vectors are processed to obtain N*M difference feature vectors to form a difference feature matrix; the M historical trajectory feature matrices corresponding to the M historical trajectories are spliced to obtain a spliced feature matrix, and the historical trajectory feature matrix includes a plurality of historical detection box feature vectors corresponding to a plurality of historical detection boxes in the historical trajectory; then, the difference feature matrix and the spliced feature matrix are processed by a preset decoder based on an attention mechanism to obtain a matching matrix; finally, according to the matching matrix, the association relationship between the N current detection boxes and the M historical trajectories is determined.

[0137] It can be seen that by constructing the difference feature matrix between each two of the N current detection box feature vectors and the M trajectory feature vectors, and performing two-class processing on the spliced feature matrix obtained by splicing the M historical trajectory feature matrices, a matching matrix representing the overall relationship between the N current detection boxes and the M historical trajectories is obtained, so that the classification processing result is more accurate, the association relationship between the N current detection boxes and the M historical trajectories is facilitated to be determined, and thus the accuracy of multi-target tracking of multiple current detection boxes corresponding to multiple historical trajectories is improved.

[0138] Exemplary apparatus

[0139] Referring to Figure 8 , a structure schematic diagram of a multi-detection box processing device based on video in an embodiment of the present application is shown. In the embodiment, the device may, for example, specifically include: an obtaining unit 801, a difference unit 802, a splicing unit 803, a classification unit 804, and a determination unit 805;

[0140] The obtaining unit 801 is configured to obtain N current detection box feature vectors corresponding to N current detection boxes in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories.

[0141] The difference unit 802 is configured to perform difference processing on the N current detection box feature vectors and the M trajectory feature vectors to obtain N*M difference feature vectors to form a difference feature matrix.

[0142] The splicing unit 803 is configured to perform splicing processing on M historical trajectory feature matrices corresponding to the M historical trajectories to obtain a spliced feature matrix; the historical trajectory feature matrix includes a plurality of historical detection box feature vectors corresponding to a plurality of historical detection boxes in the historical trajectory.

[0143] The classification unit 804 is configured to perform two-class processing based on an attention mechanism on the difference feature matrix and the spliced feature matrix by using a preset decoder to obtain a matching matrix.

[0144] The determination unit 805 is configured to determine an association relationship between the N current detection boxes and the M historical trajectories according to the matching matrix.

[0145] In an optional implementation of an embodiment of the present application, the preset decoder includes a preset edge decoding layer, and the preset edge decoding layer includes a first self-attention module, a cross-attention module, and a first feedforward network module; the classification unit 804 is configured to:

[0146] perform self-attention calculation processing on the difference feature matrix by using the first self-attention module to obtain a processed difference feature matrix;

[0147] The cross-attention module is used for cross-attention calculation and processing on the processed difference feature matrix and the spliced feature matrix, to obtain a cross feature matrix.

[0148] The first feedforward network module is used for binary classification processing on the cross feature matrix, to obtain a matching matrix.

[0149] In an optional implementation of the embodiment of the present application, the obtaining unit 801 is further configured to:

[0150] The appearance information and the space-time information of the current detection frame are encoded to obtain an appearance feature vector and a space-time feature vector of the current detection frame.

[0151] The appearance feature vector and the space-time feature vector of the current detection frame are summed to obtain a current detection frame feature vector.

[0152] In an optional implementation of the embodiment of the present application, the plurality of historical detection frames in the historical trajectory are sorted in time sequence, and the obtaining unit 801 is further configured to:

[0153] The appearance information of the plurality of historical detection frames and the space-time information of the last historical detection frame sorted are encoded to obtain an appearance feature vector of the plurality of historical detection frames and a space-time feature vector of the last historical detection frame.

[0154] The appearance feature vectors of the plurality of historical detection frames are averaged to obtain a mean feature vector.

[0155] The mean feature vector and the space-time feature vector of the last historical detection frame are summed to obtain a trajectory feature vector.

[0156] In an optional implementation of the embodiment of the present application, the device further comprises an optimization unit.

[0157] The optimization unit is configured to perform attention mechanism-based optimization processing on the M historical trajectory feature matrices by using a preset encoder to obtain M optimized trajectory feature matrices.

[0158] The splicing unit 803 is configured to:

[0159] The M optimized trajectory feature matrices are spliced to obtain a spliced feature matrix.

[0160] In an optional implementation of the embodiment of the present application, the preset encoder comprises a preset edge encoding layer, and the preset edge encoding layer comprises a second self-attention module and a second feedforward network module; the optimization unit is configured to:

[0161] For each historical trajectory feature matrix, the second self-attention module is used for self-attention calculation and processing on the historical trajectory feature matrix to obtain a processed historical trajectory feature matrix.

[0162] The processed historical trajectory feature matrix is optimized by the second feedforward network module to obtain an optimized trajectory feature matrix.

[0163] According to the various embodiments provided in the embodiment, firstly, N current detection box feature vectors corresponding to N current detection boxes in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories are obtained, M and N are positive integers, M is greater than or equal to 2, and N is greater than or equal to 1; secondly, the N current detection box feature vectors and the M trajectory feature vectors are processed to obtain N*M difference feature vectors to form a difference feature matrix; the M historical trajectory feature matrices corresponding to the M historical trajectories are spliced to obtain a spliced feature matrix, and the historical trajectory feature matrix includes a plurality of historical detection box feature vectors corresponding to a plurality of historical detection boxes in the historical trajectory; then, the difference feature matrix and the spliced feature matrix are processed by a preset decoder based on an attention mechanism to obtain a matching matrix; finally, the matching matrix is used to determine the association relationship between the N current detection boxes and the M historical trajectories.

[0164] It can be seen that the difference feature matrix is formed by constructing the difference between each two of the N current detection box feature vectors and the M trajectory feature vectors, the spliced feature matrix is obtained by splicing the M historical trajectory feature matrices, and the matching matrix representing the overall relationship between the N current detection boxes and the M historical trajectories is obtained by performing binary classification processing, so that the classification processing result is more accurate, the association relationship between the N current detection boxes and the M historical trajectories is determined more easily, and the accuracy of video multi-target tracking of the plurality of current detection boxes corresponding to the plurality of historical trajectories is improved.

[0165] In addition, the embodiment of the application further provides a computer device, which comprises a processor and a memory:

[0166] The memory is used to store program code and transmit the program code to the processor.

[0167] The processor is used to execute the video-based multi-detection box processing method of the above-mentioned embodiments according to the instructions in the program code.

[0168] The embodiment of the application further provides a computer readable storage medium, which is used to store program code, and the program code is executed by the processor to execute the video-based multi-detection box processing method of the above-mentioned embodiments.

[0169] The various embodiments described in this specification are intended to be exemplary only. To save the reader from having to read through a long specification of every possible embodiment, the various embodiments are described in a progressive manner, with each embodiment highlighting a different aspect of the embodiments. The same or similar parts between the various embodiments are cross-referenced to each other. For the apparatus embodiments, the description is relatively brief because the apparatus embodiments are intended to correspond to the method embodiments.

[0170] Those skilled in the art will further appreciate that the units and algorithm steps of the various examples described in connection with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or combinations of both. To clearly illustrate this interchangeability of hardware and software, various examples have been described generally in terms of their functionality, without necessarily describing the specific programming or hardware of the examples. The particular implementation of the described functionality can depend on the particular application and design constraints imposed on the overall system. Skilled persons can implement the described functionality in varying ways for each particular application, but such implementation should not be construed to limit the scope of the application.

[0171] It should be noted that, in the present document, relational terms such as first and second and the like can be used solely to distinguish one entity or action from another entity or action without necessarily implying any actual relationship or order between such entities or actions. The terms "comprises", "comprising", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. The terms "includes", "including", or any other variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that includes a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus.

[0172] The above description is only preferred embodiments of the present application and is not intended in any way to limit the present application. Although the present application has been disclosed with reference to the preferred embodiments, the present application is not limited to the preferred embodiments. Any person skilled in the art, without departing from the scope of the present application, can make many possible changes and modifications to the disclosed methods and technical contents of the present application, or modify them into equivalent embodiments. Therefore, any simple modification, equivalent change and modification made to the above embodiments without departing from the scope of the present application, in accordance with the technical essence of the present application, are still within the scope of protection of the present application.

Claims

1. A video-based method for processing multiple detection boxes, characterized in that, The method comprises the following steps: obtaining N current detection box feature vectors corresponding to N current detection boxes in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories; M and N are positive integers; performing difference processing on the N current detection box feature vectors and the M trajectory feature vectors to obtain N*M difference feature vectors to form a difference feature matrix; performing splicing processing on M historical trajectory feature matrices corresponding to the M historical trajectories to obtain a splicing feature matrix; the historical trajectory feature matrix comprises a plurality of historical detection box feature vectors corresponding to a plurality of historical detection boxes in the historical trajectory; performing attention mechanism-based binary classification processing on the difference feature matrix and the splicing feature matrix by using a preset decoder to obtain a matching matrix; determining the association relationship between the N current detection boxes and the M historical trajectories according to the matching matrix.

2. The method of claim 1, wherein, The preset decoder comprises a preset edge decoding layer, and the preset edge decoding layer comprises a first self-attention module, a cross-attention module and a first feedforward network module. The method comprises the following steps: performing self-attention calculation processing on the difference feature matrix by using the first self-attention module to obtain a processed difference feature matrix; performing cross-attention calculation processing on the processed difference feature matrix and the splicing feature matrix by using the cross-attention module to obtain a cross feature matrix; performing binary classification processing on the cross feature matrix by using the first feedforward network module to obtain the matching matrix.

3. The method of claim 1, wherein, The method comprises the following steps: performing encoding processing on the appearance information and the space-time information of the current detection box to obtain an appearance feature vector and a space-time feature vector of the current detection box; performing sum processing on the appearance feature vector and the space-time feature vector of the current detection box to obtain the current detection box feature vector.

4. The method of claim 1, wherein, The plurality of historical detection boxes in the historical trajectory are sorted in chronological order, and the method comprises the following steps: performing encoding processing on the appearance information of the plurality of historical detection boxes and the space-time information of the last historical detection box sorted to obtain an appearance feature vector of the plurality of historical detection boxes and a space-time feature vector of the last historical detection box; performing mean processing on the appearance feature vectors of the plurality of historical detection boxes to obtain a mean feature vector; performing sum processing on the mean feature vector and the space-time feature vector of the last historical detection box to obtain the trajectory feature vector.

5. The method according to any one of claims 1 to 4, characterized in that, The method further comprises the following steps: performing attention mechanism-based optimization processing on the M historical trajectory feature matrices by using a preset encoder to obtain M optimized trajectory feature matrices; The method comprises the following steps: performing splicing processing on the M optimized trajectory feature matrices to obtain the splicing feature matrix.

6. The method of claim 5, wherein, The preset encoder comprises a preset edge encoding layer, and the preset edge encoding layer comprises a second self-attention module and a second feedforward network module; The preset encoder comprises a preset edge encoding layer, and the preset edge encoding layer comprises a second self-attention module and a second feedforward network module; The second self-attention module is used for performing self-attention calculation on the historical trajectory feature matrix to obtain a processed historical trajectory feature matrix. The second feedforward network module is used for performing optimization processing on the processed historical trajectory feature matrix to obtain the optimized trajectory feature matrix.

7. A video-based multi-detection box processing apparatus, characterized in that, Comprise: The obtaining unit, the difference unit, the splicing unit, the classification unit and the determination unit; The obtaining unit is configured to obtain N current detection box feature vectors corresponding to N current detection boxes in a to-be-processed video and M trajectory feature vectors corresponding to M historical trajectories; M and N are positive integers; The difference unit is configured to perform difference processing on the N current detection box feature vectors and the M trajectory feature vectors to obtain N*M difference feature vectors to form a difference feature matrix; The splicing unit is configured to perform splicing processing on M historical trajectory feature matrices corresponding to the M historical trajectories to obtain a spliced feature matrix; the historical trajectory feature matrix comprises a plurality of historical detection box feature vectors corresponding to a plurality of historical detection boxes in the historical trajectory; The classification unit is configured to perform two-class processing on the difference feature matrix and the spliced feature matrix based on an attention mechanism by using a preset decoder to obtain a matching matrix; The determination unit is configured to determine an association relationship between the N current detection boxes and the M historical trajectories according to the matching matrix.

8. The apparatus of claim 7, wherein, The preset decoder comprises a preset edge decoding layer, and the preset edge decoding layer comprises a first self-attention module, a cross-attention module and a first feedforward network module; and the classification unit is configured to: The first self-attention module is used for performing self-attention calculation on the difference feature matrix to obtain a processed difference feature matrix; The cross-attention module is used for performing cross-attention calculation on the processed difference feature matrix and the spliced feature matrix to obtain a cross feature matrix; The first feedforward network module is used for performing two-class processing on the cross feature matrix to obtain the matching matrix.

9. A computer device, comprising: The computer device comprises a processor and a memory: The memory is configured to store program code and transmit the program code to the processor; The processor is configured to execute instructions in the program code to perform the video-based multi-detection box processing method according to any one of claims 1-6.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium is configured to store program code, and the program code is executed by a processor to perform the video-based multi-detection box processing method according to any one of claims 1-6.

Images