Method, device, system and storage medium for determining multiple target trajectories in a video

Abstract

Embodiments of the present invention provide a method, device, system and storage medium for determining the trajectories of multiple targets in a video. The method includes: performing target detection on a current video frame to obtain a detection frame of all targets in the current video frame. Based on the trajectory of the target in the previous video frame, predict the prediction frame of the corresponding target in the current video frame. Calculate the similarity between the detection frame and the prediction frame respectively to obtain the original matching data. Input the original matching data into a matching neural network so that the matching neural network outputs the final matching data between the detection frame and the trajectory. According to the final matching data, the detection frame is stored in the matching trajectory. The end-to-end determination of the trajectories of multiple targets in the video is achieved. There is no iterative calculation process, which improves the calculation efficiency. In addition, a GPU can be used for calculation acceleration, and all targets in the video frame can be calculated in parallel, which further greatly improves the calculation speed of determining the trajectories of multiple targets in the video.

Description

Technical Field

[0001] The present invention relates to the field of computer vision technology, and more specifically to a method, device, system and storage medium for determining multi-target trajectories in a video. Background Art

[0002] Determining the trajectories of multiple targets in a video is an important problem in the field of computer vision. It is used to determine the trajectories of multiple targets in a continuous video. Its essence is to associate the same target in the previous and next frames of the video and assign a unique trajectory.

[0003] Since the trajectories of each target are constantly changing during the process of determining the trajectories of multiple targets in a video, the number and types of targets in the video are not fixed. Specifically, in the detection results for the targets, in addition to the targets that have already appeared in the previous video frame, new targets may also appear. For targets that no longer appear in the video frame, their trajectories need to be terminated. This results in the number of targets and the number of existing trajectories not being fixed. As a result, accurately determining the trajectories of multiple targets in a video requires a huge amount of calculation and is extremely time-consuming.

[0004] In the prior art, deep learning algorithms can generally be used to determine the trajectories of multiple targets in a video. First, all targets in each frame of the image are detected to obtain target detection results. Then, the corresponding target prediction results are obtained based on the trajectory prediction of the detected targets. Finally, the target detection results and the trajectories based on the target prediction results are matched to obtain the final trajectory matching results. Among them, both target detection and target prediction can be achieved through neural networks. However, the step of matching targets with trajectories is mostly achieved using the Hungarian algorithm.

[0005] This leads to the following technical problems: the Hungarian algorithm for matching targets and trajectories requires multiple iterations and cannot be accelerated by a graphics processing unit (GPU). Based on the large amount of calculation, this leads to a low speed of the algorithm. Summary of the invention

[0006] The present invention is proposed in view of the above problems. According to one aspect of the present invention, a method for determining the trajectories of multiple targets in a video is provided. The method comprises: performing target detection on a current video frame to obtain detection frames of all targets in the current video frame. Based on the trajectories of targets in previous video frames, predicting prediction frames of corresponding targets in the current video frame. Calculating the similarities between the detection frames and the prediction frames respectively to obtain original matching data. Inputting the original matching data into a matching neural network so that the matching neural network outputs final matching data between the detection frames and the trajectories. Based on the final matching data, storing the detection frames into the matching trajectories.

[0007] Exemplarily, target detection is performed on the current video frame to obtain detection frames of all targets in the current video frame, prediction frames of corresponding targets in the current video frame are predicted based on the trajectory of the targets in the previous video frame, and similarities between the detection frames and the prediction frames are respectively calculated to obtain raw matching data using a first neural network. The first neural network and the matching neural network are interconnected. The first neural network outputs raw matching data to the matching neural network.

[0008] Exemplarily, the method further includes: simultaneously training the first neural network and the matching neural network using the training video and the corresponding annotated data, wherein the annotated data includes the position information of the real trajectory of the target in the training video.

[0009] Exemplarily, the matching neural network is a fully convolutional neural network.

[0010] Exemplarily, respectively calculating the similarity between the detection frame and the prediction frame to obtain the original matching data further includes: respectively calculating the similarity between each of the detection frames and each of the prediction frames. Constructing a similarity matrix from the similarities. The elements in the i-th row and j-th column of the similarity matrix are the similarities between the detection frame of the i-th target in the current video frame and the prediction frame predicted based on the trajectory of the j-th target in the previous video frame. Constructing the original matching data according to the similarity matrix.

[0011] Exemplarily, before constructing the original matching data based on the similarity matrix, respectively calculating the similarities between the detection box and the prediction box to obtain the original matching data also includes: for a detection box whose similarity with all prediction boxes is 0, adding a column to the similarity matrix, and setting the elements of the row corresponding to the detection box in the added column to 1 to update the similarity matrix.

[0012] Exemplarily, before constructing the original matching data based on the similarity matrix, respectively calculating the similarities between the detection box and the prediction box to obtain the original matching data also includes: for the trajectory based on the prediction box whose similarity with all detection boxes is 0, adding a row to the similarity matrix, and setting the elements of the column corresponding to the trajectory in the added row to 1 to update the similarity matrix.

[0013] Exemplarily, the similarity includes a plurality of similarity representations. Constructing a similarity matrix from the similarities includes: forming a corresponding similarity matrix from the corresponding similarity representations of the similarities. Constructing original matching data from the similarity matrix includes: determining each similarity matrix as data of a channel in a three-dimensional tensor, so as to use the three-dimensional tensor as original matching data.

[0014] Exemplarily, the similarity includes one or more of the following similarity representation quantities: intersection over union ratio, position difference, and similarity of visual features of images within a frame.

[0015] According to another aspect of the present invention, there is provided an apparatus for determining the trajectories of multiple targets in a video, comprising: a target detection module for performing target detection on a current video frame to obtain detection frames of all targets in the current video frame. A target prediction module for predicting the prediction frame of the corresponding target in the current video frame based on the trajectory of the target in the previous video frame. A similarity calculation module for respectively calculating the similarity between the detection frame and the prediction frame to obtain original matching data. A matching module for inputting the original matching data into a matching neural network so that the matching neural network outputs the final matching data between the detection frame and the trajectory. A storage module for storing the detection frame to the matching trajectory according to the final matching data.

[0016] According to another aspect of the present invention, a system for determining multiple target trajectories in a video is provided, comprising a processor and a memory. The memory stores computer program instructions. The computer program instructions are used to execute the method for determining multiple target trajectories in a video as described above when the processor is executed.

[0017] According to another aspect of the present invention, a storage medium is provided, on which program instructions are stored, and when the program instructions are run, they are used to execute the method for determining multiple target trajectories in a video as described above.

[0018] By using a neural network to match the detection frame with the trajectory, the matching neural network is added to the entire calculation process of determining the trajectory of multiple targets in the video, and the end-to-end determination of the trajectory of multiple targets in the video is achieved. The so-called end-to-end means that the calculation is performed from the starting point to the end point in sequence, and the subsequent calculation will not affect the previous calculation. Therefore, there is no iterative calculation process, which improves the calculation efficiency. In addition, the GPU can be used for calculation acceleration, and all targets in the video frame can be calculated in parallel, which further greatly improves the calculation speed of determining the trajectory of multiple targets in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] The above and other purposes, features and advantages of the present invention will become more apparent by describing the embodiments of the present invention in more detail in conjunction with the accompanying drawings. The accompanying drawings are used to provide a further understanding of the embodiments of the present invention and constitute a part of the specification. Together with the embodiments of the present invention, they are used to explain the present invention and do not constitute a limitation of the present invention. In the accompanying drawings, the same reference numerals generally represent the same components or steps.

[0020] Figure 1 A schematic diagram of determining the trajectories of multiple objects in a video is shown;

[0021] Figure 2A schematic block diagram of an example electronic device for implementing a method and apparatus for determining multiple target trajectories in a video according to an embodiment of the present invention is shown;

[0022] Figure 3 A schematic flow chart of a method for determining multiple target trajectories in a video according to an embodiment of the present invention is shown;

[0023] Figure 4 A schematic flow chart of a method for obtaining original matching data according to an embodiment of the present invention is shown;

[0024] Figure 5 A schematic block diagram of an apparatus for determining trajectories of multiple objects in a video according to an embodiment of the present invention is shown; and

[0025] Figure 6 A schematic block diagram of a system for determining trajectories of multiple objects in a video according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0026] In recent years, research on computer vision, deep learning, machine learning, image processing, image recognition and other technologies based on artificial intelligence has made important progress. Artificial Intelligence (AI) is an emerging science and technology that studies and develops theories, methods, technologies and application systems for simulating and extending human intelligence. Artificial intelligence is a comprehensive discipline involving many types of technologies such as chips, big data, cloud computing, the Internet of Things, distributed storage, deep learning, machine learning, neural networks, etc. Computer vision, as an important branch of artificial intelligence, specifically allows machines to recognize the world. Computer vision technology usually includes face recognition, liveness detection, fingerprint recognition and anti-counterfeiting verification, biometric recognition, face detection, pedestrian detection, target detection, pedestrian recognition, image processing, image recognition, image semantic understanding, image retrieval, text recognition, video processing, video content recognition, behavior recognition, 3D reconstruction, virtual reality, augmented reality, simultaneous localization and mapping (SLAM), computational photography, robot navigation and positioning and other technologies. With the research and advancement of artificial intelligence technology, this technology has been applied in many fields, such as security, urban management, traffic management, building management, park management, facial access, facial attendance, logistics management, warehouse management, robots, intelligent marketing, computational photography, mobile phone imaging, cloud services, smart homes, wearable devices, unmanned driving, automatic driving, smart medical care, facial payment, facial unlocking, fingerprint unlocking, identity verification, smart screens, smart TVs, cameras, mobile Internet, live streaming, beauty, makeup, medical beauty, and smart temperature measurement.

[0027] In order to make the purpose, technical scheme and advantages of the present invention more obvious, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described herein. Based on the embodiments described in the present invention, all other embodiments obtained by those skilled in the art without creative work should fall within the protection scope of the present invention.

[0028] Figure 1 Schematic diagram of determining the trajectories of multiple objects in a video. Figure 1 , target detection is performed on each video frame to obtain one or more detection frames. The detection frames of the same target in multiple video frames constitute the trajectory of the target. In order to solve the above-mentioned problems in the existing technology for determining the trajectories of multiple targets in a video and improve its calculation speed and efficiency, an embodiment of the present invention provides a method for determining the trajectories of multiple targets in a video. The method uses a neural network to achieve matching between target detection results and target prediction results. In the entire matching calculation of the neural network, the subsequent calculation results will not affect the previous calculation results, and there is no iterative calculation. The calculation steps in the neural network can be processed in parallel. As a result, the calculation speed is effectively improved.

[0029] First, refer to Figure 2 An example electronic device 200 for implementing the method and apparatus for determining multiple target trajectories in a video according to an embodiment of the present invention is described.

[0030] like Figure 2 As shown, the electronic device 200 includes one or more processors 202 and one or more storage devices 204. Optionally, the electronic device 200 may also include an input device 206, an output device 208 and an image acquisition device 210, and these components are interconnected via a bus system 212 and / or other forms of connection mechanisms (not shown). It should be noted that Figure 2 The components and structures of the electronic device 200 shown are merely exemplary and non-limiting. The electronic device may also have other components and structures as required.

[0031] The processor 202 may be implemented in at least one hardware form of a microprocessor, a digital signal processor (DSP), a field programmable gate array (FPGA), or a programmable logic array (PLA). The processor 202 may also be a central processing unit (CPU), a graphics processing unit (GPU), an application specific integrated circuit (ASIC), or a combination of one or more of other processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 200 to perform desired functions.

[0032] The storage device 204 may include one or more computer program products. The computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on a computer-readable storage medium, and the processor 202 may run the program instructions to implement the function of determining multiple target trajectories in a video in an embodiment of the present invention described below (implemented by the processor). Various applications and various data, such as various data used and / or generated by the application, may also be stored in the computer-readable storage medium.

[0033] The input device 206 may be a device used by a user to input instructions or images, and may include one or more of a keyboard, a mouse, a microphone, a touch screen, and the like.

[0034] The output device 208 can output various information (such as images and / or sounds) to the outside (such as a user), and can include one or more of a display, a speaker, etc. Optionally, the input device 206 and the output device 208 can be integrated together and implemented using the same interactive device (such as a touch screen).

[0035] The image acquisition device 210 can acquire images (including static images and video frames) and store the acquired images in the storage device 204 for use by other components. The image acquisition device 210 can be a separate camera, a camera in a mobile terminal, or an image sensor in a capture machine. It should be understood that the image acquisition device 210 is only an example, and the electronic device 200 may not include the image acquisition device 210. In this case, other image acquisition devices can be used to acquire images and send the acquired images to the electronic device 200.

[0036] Exemplarily, the example electronic device 200 for implementing the method and apparatus for determining multiple target trajectories in a video according to an embodiment of the present invention may be implemented on a device such as a personal computer or a remote server.

[0037] The following will refer to Figure 3 A method for determining trajectories of multiple objects in a video according to an embodiment of the present invention is described. Figure 3 FIG. 3 is a schematic flow chart of a method 300 for determining multiple target trajectories in a video according to an embodiment of the present invention. Figure 3 As shown, the method 300 includes the following steps.

[0038] Step S110, performing object detection on the current video frame to obtain detection frames of all objects in the current video frame.

[0039] The video is composed of continuous video frames. The video can be the original video captured by the image acquisition device 210 such as a camera, or it can be a video obtained after preprocessing the original video. The preprocessing operation can include all operations for more clear target recognition. For example, the preprocessing operation can include denoising operations such as filtering.

[0040] The video frames may be captured by the image acquisition device 210 (e.g., a camera) included in the electronic device 200 and transmitted to the processor 202 for processing. The video frames may also be captured by a client device (e.g., an image acquisition device including a camera) and finally transmitted to the electronic device 200 for processing by the processor 202 of the electronic device 200.

[0041] In addition, the original video can be captured by the image acquisition device 210 (such as a camera) included in the electronic device 200 or by a client device (such as an image acquisition device including a camera) and transmitted to the processor 202 for preprocessing to obtain a video to be processed, and then the processor 202 still performs subsequent processing.

[0042] The target may be any object that can be tracked, such as a human body, various animals, or a vehicle.

[0043] The specific scheme of target detection in step S110 is not limited in this application, and any existing or future technical scheme that can realize target detection is within the protection scope of this application. After performing target detection on the current video frame, the detection frame of all targets in the video frame can be obtained. The detection frame can be a minimum external graphic that contains only one target and is for the target, such as a rectangular frame, an elliptical frame, etc.

[0044] Step S120 , predicting a prediction box of a corresponding target in a current video frame according to the trajectory of the target in the previous video frame.

[0045] It can be understood that a video is composed of continuous video frames. There may be one or more video frames before the current video frame appears, which can be collectively referred to as previous video frames. The trajectory of the target can represent the position movement change process of each target in each video frame. Because the movement of the target is usually continuous, when the trajectory of the target in the previous video frame is known, the position of the target in the current video frame can be predicted, and the position can be represented by a prediction box. Similar to the detection box, the prediction box can also be a minimum external graphic that contains only one target and is for the target, such as a rectangular box, an elliptical box, etc.

[0046] Exemplarily, step S120 can be implemented using any existing or future technology that can achieve target prediction in a video. For example, the position of a target in a video frame can be predicted based on linear movement calculation. In adjacent video frames, assuming that the target moves linearly, the prediction box of the target in the current video frame can be calculated based on the trajectory of the target in the previous video frame.

[0047] Step S130, respectively calculating the similarity between the detection box and the prediction box of the target to obtain original matching data.

[0048] Exemplarily, the similarity between the detection frame and the prediction frame can be used to indicate the similarity between the detection frame in the current video frame and the target in the prediction frame in the current video frame. The similarity can be indicated by the area of ​​the overlapping area of ​​the detection frame and the prediction frame, the intersection-over-union ratio of the two, the position difference between the two, the similarity of the visual features of the image in the frame, etc. The similarity can be any value between 0 and 1. Exemplarily, the similarity can be calculated for all the detection frames obtained in step S110 and all the prediction frames obtained in step S120. Thus, for each detection frame, its similarity with all the prediction frames can be obtained respectively; in other words, for each prediction frame, its similarity with all the detection frames is also obtained respectively. Alternatively, the similarity between only some detection frames and some prediction frames can be calculated according to preset rules. When the value of the similarity is larger, the probability that the target in the detection frame and the prediction frame is the same target may be greater. The original matching data can be obtained according to the calculated similarity between all the detection frames and the prediction frames in the video frame. The original matching data can represent the relationship between the detection frames of all targets in the current video frame and the trajectory of the target.

[0049] Step S140: input the original matching data into the matching neural network, so that the matching neural network outputs the final matching data between the detection box and the trajectory.

[0050] The matching neural network is a neural network used to match the detection box in the current video frame with the target trajectory in the previous video frame based on the original matching data. The final matching data output by the matching neural network indicates which trajectory the detection box in the current video frame matches, that is, both belong to the same target. The final matching data includes data indicating the degree of matching between the detection box and the trajectory, which can be 0 or 1. Exemplarily, when a detection box matches a trajectory, the corresponding degree of matching between the two is 1, and when a detection box does not match a trajectory, the corresponding degree of matching between the two is 0. It can be understood that only one matching trajectory can be obtained for a detection box. In other words, for a detection box, only the trajectory that matches it outputs 1, and the remaining trajectories all output 0.

[0051] Exemplarily, the matching neural network can be any deep learning neural network, such as a fully convolutional neural network (FCN). The fully convolutional neural network has good deep learning capabilities and can use GPUs to implement parallel computing. Using a fully convolutional neural network to implement the matching neural network effectively improves the matching speed between the detection frame and the prediction frame, thereby improving the target trajectory determination speed.

[0052] Step S150, based on the final matching data, the detection frame is stored in the matching trajectory. For example, when the matching degree is 1, it means that the corresponding detection frame and the trajectory match, and the detection frame can be stored in the trajectory. Thus, the trajectory of the target in the current video frame is determined.

[0053] By using a neural network to match the detection frame with the trajectory, the matching neural network is added to the entire calculation process of determining the trajectory of multiple targets in the video, and the end-to-end determination of the trajectory of multiple targets in the video is achieved. The so-called end-to-end means that the calculation is performed from the starting point to the end point in sequence, and the subsequent calculation will not affect the previous calculation. Therefore, there is no iterative calculation process, which improves the calculation efficiency. In addition, the GPU can be used for calculation acceleration, and all targets in the video frame can be calculated in parallel, which further greatly improves the calculation speed of determining the trajectory of multiple targets in the video.

[0054] Exemplarily, the above-mentioned steps S110, S120 and S130 can be implemented using a first neural network. The first neural network can also be a convolutional neural network, etc., which is not limited. According to the above, the video can be input into the first neural network, and the first neural network can output the original matching data. It can be understood that the first neural network can also be divided into multiple sub-networks, such as a neural network for performing target detection in step S110, a neural network for performing target prediction in step S120, etc. Since the specific process has been described in detail above, it will not be repeated here. The first neural network and the matching neural network are interconnected. The first neural network can output the original matching data to the matching neural network, so that the matching neural network calculates the final matching data between the detection box and the trajectory based on the original matching data.

[0055] Here, the first neural network and the matching neural network form an end-to-end, larger-scale neural network. The problem of determining the multi-target trajectories in the video is implemented end-to-end using the neural network as a whole, giving full play to the powerful computing power of the neural network, and quickly obtaining more accurate target trajectories.

[0056] There may be interactions between the background and the target, and between different targets in the video, and the detection results of the target may be inaccurate, resulting in the situation that the detection results are not aligned with the video. For example, multiple detection frames correspond to the same target, or a detection frame covers multiple targets, etc. The inaccuracy of the detection results of the target will further affect the accuracy of the trajectory. Exemplarily, the method also includes: using the training video and the corresponding annotation data to simultaneously train the first neural network and the matching neural network. The training video may include multiple video frames, and there is annotation data corresponding to each video frame, and the annotation data includes the location information of the real trajectory of all targets. The real trajectory of the target can be obtained by manual or machine annotation. Here, the first neural network and the matching neural network are trained as a whole. The loss function can be used to train the entire neural network. The function value of the loss function can represent the deviation of the trajectory obtained by matching the matching neural network relative to the real annotated trajectory. During the training process, the function value based on the loss function can not only correct the matching neural network, but also correct the first neural network at the same time through the forward transmission of the loss function. Through the above process, the first neural network can supplement the missed detection frame, remove the redundant detection frame, etc., to achieve the training of the first neural network.

[0057] Through the above training process, the overall training of the end-to-end neural network for determining the trajectories of multiple targets in the video is achieved, and at the same time, the inaccurate target detection of the first neural network, such as the re-identification problem, and the matching accuracy problem of the matching neural network are corrected. As a result, the error that may be generated by the neural network model is reduced, thereby improving the accuracy of the trajectories of multiple targets in the video.

[0058] Figure 4 FIG. 4 is a schematic flow chart of a method for obtaining original matching data according to an embodiment of the present invention. Figure 4 As shown, step S130 further includes the following steps.

[0059] Step S131, respectively calculate the similarity between each of the detection boxes and each of the prediction boxes.

[0060] According to the above, the similarity between all detection boxes and all prediction boxes in the current video frame can be calculated pairwise. The similarity can be any value between 0 and 1. When the similarity value is larger, the possibility that the objects in the detection box and the prediction box are the same object is greater.

[0061] Exemplarily, the similarity may include one or more of the following similarity characterization quantities: intersection-over-union ratio, position difference, similarity of visual features of images in a frame, etc. The intersection-over-union ratio may represent the area overlap ratio of the detection frame and the prediction frame in the current video frame. The larger the area overlap ratio, the closer the distance between the objects in the two frames is, and the greater the probability that the objects in the two frames are the same object. The position difference may represent the position coordinate deviation of the detection frame and the prediction frame in the current video frame. The smaller the position coordinate deviation, the closer the distance between the objects in the two frames is, and the greater the probability that the objects in the two frames are the same object. The similarity of visual features of images in a frame may represent the similarity of visual features of the objects in the detection frame and the prediction frame in the current video frame. The greater the similarity of visual features, the more similar the appearance of the objects in the two frames is, and the greater the probability that the two objects are the same object. The visual features may be, for example, color features, texture features, and the like.

[0062] According to the above three similarity representation quantities, a more accurate similarity can be obtained at a lower computational cost, thereby providing more accurate input data for the matching neural network, thereby improving the accuracy of the determined target trajectory.

[0063] Step S132, constructing a similarity matrix from the similarities. Exemplarily, the number of rows of the similarity matrix may be equal to the number of all detection frames in the current video frame. The number of columns of the similarity matrix may be equal to the number of all prediction frames in the current video frame. The elements of the i-th row and j-th column in the similarity matrix may be the similarity between the detection frame of the i-th target in the current video frame and the prediction frame predicted based on the trajectory of the j-th target in the previous video frame.

[0064] In an example, the current video frame contains 3 detection boxes and 3 prediction boxes. Therefore, a 3-row 3-column similarity matrix can be constructed, which contains 9 elements to represent the similarity obtained by calculating the detection boxes and prediction boxes pairwise. For example, the matrix can be As mentioned above, the similarity can be expressed by the area of ​​the overlapping area of ​​the detection frame and the prediction frame, the intersection-and-union ratio of the two, the position difference of the two, the similarity of the visual features of the image in the frame, etc. Take the similarity as the intersection-and-union ratio of the detection frame and the prediction frame as an example to illustrate the above matrix. The intersection-and-union ratio can represent the area overlap rate of the detection frame and the prediction frame in the current video frame. When the area overlap rate is larger, it can be said that the possibility that the target in the detection frame and the prediction frame is the same target is greater. Referring to the first two rows of the above matrix, 0.8 and 0.6 in the first row and 0.6 and 0.9 in the second row are not much different in value. This phenomenon may be caused by the close distance between the targets in the detection frames corresponding to the first and second rows, resulting in overlap. In the first row, the overlap rate between the prediction frame corresponding to the first column and the detection frame is greater than the overlap rate between the prediction frame corresponding to the second column and the detection frame. Therefore, the target in the prediction frame corresponding to the first column is more likely to be the same target as the target in the detection frame. Similarly, in the second row, the target in the prediction frame corresponding to the second column is more likely to be the same target as the target in the detection frame. In the matrix, 0 means that there is no overlap between the detection box and the prediction box, which means that the probability that the objects in the two are the same object is almost 0.

[0065] Step S133, constructing original matching data based on the similarity matrix. The original matching data is constructed based on the similarity matrix to represent the relationship between the detection box in the current video frame and the trajectory of the target. Exemplarily, the original matching data can be directly constructed from the similarity matrix. In the above example, the similarity matrix That is, it can be used as the original matching data. Alternatively, the similarity matrix can be further processed to obtain the original matching data.

[0066] The obtained original matching data can be input into the matching neural network, so that the matching neural network can match the detection box and the trajectory, thereby outputting the final matching data. In the example, the final matching data may be That is, the detection frame of the first target, the detection frame of the second target, and the detection frame of the third target are matched with the trajectory of the first target, the trajectory of the second target, and the trajectory of the third target, respectively.

[0067] The similarity matrix is ​​used to construct the original matching data, which facilitates data management and reduces the amount of calculation for subsequent calculation processes.

[0068] Exemplarily, the similarity includes multiple similarity representation quantities, for example, it may include the area of ​​the overlapping area in the detection box and the prediction box as described above, the intersection and union ratio of the two, the position difference between the two, the visual feature similarity of the images in the frame, etc. Different similarity matrices can be constructed according to different similarity representation quantities, that is, the corresponding similarity representation quantities of the similarity constitute the corresponding similarity matrix. For example, for the intersection and union ratio, the similarity matrix about the intersection and union ratio can be constructed by the intersection and union ratios between all the detection boxes and the prediction boxes. For the position difference, the similarity matrix about the position difference can be constructed by the position difference between all the detection boxes and the prediction boxes. The dimensions of these similarity matrices are the same, that is, the number of rows and columns are the same.

[0069] Constructing the original matching data according to the similarity matrix may include: determining that each similarity matrix is ​​the data of a channel in the three-dimensional tensor, so as to use the three-dimensional tensor as the original matching data. Since the similarity matrix can be one or more, the number of channels of the three-dimensional tensor changes accordingly. Each similarity matrix is ​​the data of one of the channels. All similarity matrices can jointly construct the original matching data.

[0070] Therefore, the original matching data can include similarity matrices of different similarity representation quantities, so that when matching the detection box and the trajectory, there is more detailed information reference, which ensures the accuracy of determining the trajectories of multiple targets.

[0071] Exemplarily, before constructing the original matching data based on the similarity matrix, step S130 also includes: step S134, for the detection box whose similarity with all prediction boxes is 0, adding a column to the similarity matrix, and setting the elements of the row corresponding to the detection box in the added column to 1 to update the similarity matrix.

[0072] Specifically, when the similarity between a detection box and all prediction boxes is 0, it can be said that the target in the detection box has never appeared in the previous video frame and may be a new target in the current video frame. Therefore, its track does not exist in the previous video frame, and no prediction box predicted based on the track will be generated. For example, the similarity matrix is In the similarity matrix, the similarity between the detection box of the third target and any prediction box is 0. You can add a column to the similarity matrix and set the elements of the row corresponding to the detection box in the column to 1. This will update the similarity matrix. For the above similarity matrix, the updated similarity matrix can be The updated similarity matrix is ​​used as input, and a new track can be established for the detection frame through the matching neural network. It can be understood that according to the final matching result obtained by the similarity matrix, there may be a track that disappears or there are missed detection frames, that is, there is no matching detection frame in the current video frame.

[0073] In this way, the trajectory of the newly appeared target can be determined, that is, all targets in the current video frame have their corresponding trajectories, which ensures the accuracy of the task of determining the trajectories of multiple targets in the video.

[0074] Exemplarily, before constructing the original matching data based on the similarity matrix, step S130 also includes: step S135, for the trajectory based on the predicted box whose similarity with all detection boxes is 0, adding a row to the similarity matrix, and setting the elements of the column corresponding to the trajectory in the added row to 1 to update the similarity matrix.

[0075] Similar to step S134, when the similarity between the prediction frame and all the detection frames is 0, it can be said that the trajectory based on the prediction frame does not match any detection frame. The target matching this trajectory has left the area corresponding to the current video frame or the detection frame has been missed. For example, the similarity matrix is A row can be added to the similarity matrix, and the elements of the column corresponding to the track in the row are set to 1 to match a detection box to the track. The similarity matrix is ​​updated accordingly. For the above similarity matrix, the updated similarity matrix can be The updated similarity matrix is ​​used as input and calculated by the matching neural network to obtain a detection box that matches the trajectory. This trajectory can be considered as an extinction trajectory, and the target corresponding to the trajectory has left the area corresponding to the current video frame. It can be understood that according to the final matching result obtained by the similarity matrix, there may be a detection box that currently has no trajectory matching it.

[0076] In this way, the trajectory of the target that has left the area corresponding to the current video frame can be determined, and the problem of missed detection of the target can be corrected, thereby ensuring the accuracy of the trajectory of the target.

[0077] It can be understood that step S134 and step S135 are two possible results in the process of performing the task of determining the multi-target trajectories in the video, and there is no order of precedence. These two results may appear at the same time, neither of them appears, or only one of them appears.

[0078] According to another aspect of the present invention, a device for determining trajectories of multiple objects in a video is provided. Figure 5 A schematic block diagram of an apparatus 500 for determining trajectories of multiple objects in a video according to an embodiment of the present invention is shown.

[0079] like Figure 5 As shown, the device 500 includes a target detection module 510, a target prediction module 520, a similarity calculation module 530, a matching module 540 and a storage module 550. The modules can respectively perform the steps / functions of the method for determining the trajectories of multiple targets in a video described above. Only the main functions of the components of the device 500 are described below, and the details described above are omitted.

[0080] The object detection module 510 is used to perform object detection on the current video frame to obtain the detection frames of all objects in the current video frame. Figure 2 The processor 202 in the electronic device shown executes program instructions stored in the storage device 204 to implement.

[0081] The target prediction module 520 is used to predict the prediction box of the corresponding target in the current video frame based on the trajectory of the target in the previous video frame. The target prediction module 520 can be composed of Figure 2 The processor 202 in the electronic device shown executes program instructions stored in the storage device 204 to implement.

[0082] The similarity calculation module 530 is used to calculate the similarity between the detection frame and the prediction frame to obtain the original matching data. Figure 2 The processor 202 in the electronic device shown executes program instructions stored in the storage device 204 to implement.

[0083] The matching module 540 is used to input the original matching data into the matching neural network so that the matching neural network outputs the final matching data between the detection box and the trajectory. Figure 2 The processor 202 in the electronic device shown executes program instructions stored in the storage device 204 to implement.

[0084] The storage module 550 is used to store the detection frame to the matching trajectory according to the final matching data. The storage module 550 can be composed of Figure 2 The processor 202 in the electronic device shown executes program instructions stored in the storage device 204 to implement.

[0085] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware to implement the device 500. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0086] According to yet another aspect of the present invention, a system for determining trajectories of multiple objects in a video is provided. Figure 6 FIG. 6 is a schematic block diagram of a system 600 for determining multiple target trajectories in a video according to an embodiment of the present invention. Figure 6 As shown, the system 600 includes an input device 610 , a memory 620 , and a processor 630 .

[0087] The input device 610 is used to receive instructions input by the user and collect or acquire images. The input device 610 may include one or more of a keyboard, a mouse, a microphone, a touch screen, and an image acquisition device.

[0088] The memory 620 stores computer program instructions for implementing corresponding steps in the method for determining multiple target trajectories in a video according to an embodiment of the present invention.

[0089] The processor 630 is used to run the computer program instructions stored in the memory 620 to execute the corresponding steps of the method for determining the trajectories of multiple targets in a video according to an embodiment of the present invention, and to implement the target detection module 510, target prediction module 520, similarity calculation module 530, matching module 540 and storage module 550 in the device for determining the trajectories of multiple targets in a video according to an embodiment of the present invention.

[0090] According to another aspect of the present invention, a storage medium is also provided. Program instructions are stored on the storage medium, and when the program instructions are run by a computer or a processor, the computer or the processor performs the corresponding steps of the method for determining the multi-target trajectory in the video according to the embodiment of the present invention, and is used to implement the corresponding modules in the device for determining the multi-target trajectory in the video according to the embodiment of the present invention. The storage medium may include, for example, a memory card of a smart phone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disk read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media.

[0091] Although example embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above example embodiments are merely exemplary and are not intended to limit the scope of the present invention thereto. Various changes and modifications may be made therein by one of ordinary skill in the art without departing from the scope and spirit of the present invention. All such changes and modifications are intended to be included within the scope of the present invention as required by the appended claims.

[0092] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of the present invention.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of the units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed.

[0094] In the description provided herein, a large number of specific details are described. However, it is understood that embodiments of the present invention can be practiced without these specific details. In some instances, well-known methods, structures and techniques are not shown in detail so as not to obscure the understanding of this description.

[0095] Similarly, it should be understood that in order to streamline the present invention and help understand one or more of the various inventive aspects, in the description of the exemplary embodiments of the present invention, the various features of the present invention are sometimes grouped together into a single embodiment, figure, or description thereof. However, the method of the present invention should not be interpreted as reflecting the following intention: the claimed invention requires more features than the features explicitly stated in each claim. More specifically, as reflected in the corresponding claims, the inventive point is that the corresponding technical problem can be solved with less than all the features of a single disclosed embodiment. Therefore, the claims following the specific embodiment are hereby expressly incorporated into the specific embodiment, wherein each claim itself serves as a separate embodiment of the present invention.

[0096] It will be understood by those skilled in the art that, except for mutually exclusive features, all features disclosed in this specification (including the accompanying claims, abstracts and drawings) and all processes or units of any method or device disclosed in this specification may be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstracts and drawings) may be replaced by an alternative feature that provides the same, equivalent or similar purpose.

[0097] In addition, those skilled in the art will appreciate that, although some embodiments described herein include certain features included in other embodiments but not other features, the combination of features of different embodiments is meant to be within the scope of the present invention and form different embodiments. For example, in the claims, any one of the claimed embodiments may be used in any combination.

[0098] The various component embodiments of the present invention can be implemented in hardware, or in software modules running on one or more processors, or in a combination thereof. It should be understood by those skilled in the art that a microprocessor or a digital signal processor (DSP) can be used in practice to implement some or all of the functions of some modules in the device for determining multi-target trajectories in a video according to an embodiment of the present invention. The present invention can also be implemented as a device program (e.g., a computer program and a computer program product) for executing part or all of the methods described herein. Such a program implementing the present invention can be stored on a computer-readable medium, or can have the form of one or more signals. Such a signal can be downloaded from an Internet website, or provided on a carrier signal, or provided in any other form.

[0099] It should be noted that the above embodiments illustrate the present invention rather than limit it, and that those skilled in the art may devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference symbol between brackets shall not be construed as a limitation on the claims. The word "comprising" does not exclude the presence of elements or steps not listed in the claims. The word "one" or "an" preceding an element does not exclude the presence of a plurality of such elements. The present invention may be implemented by means of hardware comprising a number of different elements and by means of a suitably programmed computer. In a unit claim enumerating a number of devices, several of these devices may be embodied by the same hardware item. The use of the words first, second, and third, etc., does not indicate any order. These words may be interpreted as names.

[0100] The above is only a specific embodiment of the present invention or an explanation of a specific embodiment, and the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. The protection scope of the present invention shall be based on the protection scope of the claims.

Claims

1. A method for determining multiple target trajectories in a video, wherein: The method comprises: Performing target detection on the current video frame to obtain detection frames of all targets in the current video frame; Predicting a prediction box of a corresponding target in the current video frame according to the trajectory of the target in the previous video frame; Calculating the similarity between the detection frame and the prediction frame respectively to obtain original matching data; Inputting the original matching data into a matching neural network, so that the matching neural network outputs final matching data between the detection box and the track, wherein the final matching data includes data indicating a matching degree between the detection box and the track; and According to the final matching data, storing the detection frame into a matching trajectory; The performing of target detection on the current video frame to obtain detection frames of all targets in the current video frame, the predicting of prediction frames of corresponding targets in the current video frame based on the trajectories of targets in previous video frames, and the respectively calculating of the similarities between the detection frames and the prediction frames to obtain the original matching data are all implemented using a first neural network, the first neural network and the matching neural network are interconnected, and the first neural network outputs the original matching data to the matching neural network; The method further comprises: The first neural network and the matching neural network are trained simultaneously using a training video and corresponding annotated data, wherein the annotated data includes position information of a real trajectory of a target in the training video.

2. The method of claim 1, wherein: The matching neural network is a fully convolutional neural network.

3. The method of claim 1, wherein: The respectively calculating the similarity between the detection frame and the prediction frame to obtain the original matching data further comprises: Calculate the similarity between each of the detection frames and each of the prediction frames respectively; Constructing a similarity matrix from the similarities, wherein the elements in the i-th row and the j-th column in the similarity matrix are the similarities between the detection frame of the i-th target in the current video frame and the prediction frame predicted based on the trajectory of the j-th target in the previous video frame; The original matching data is constructed according to the similarity matrix.

4. The method of claim 3, wherein: Before constructing the original matching data according to the similarity matrix, respectively calculating the similarity between the detection box and the prediction box to obtain the original matching data further includes: For a detection frame whose similarity with all prediction frames is 0, a column is added to the similarity matrix, and the elements of the row corresponding to the detection frame in the added column are set to 1 to update the similarity matrix.

5. The method of claim 3, wherein: Before constructing the original matching data according to the similarity matrix, respectively calculating the similarity between the detection box and the prediction box to obtain the original matching data further includes: For the trajectory based on the predicted box whose similarity with all detection boxes is 0, a row is added to the similarity matrix, and the elements of the column corresponding to the trajectory in the added row are set to 1 to update the similarity matrix.

6. The method of claim 3, wherein: The similarity includes multiple similarity representation quantities. The constructing a similarity matrix from the similarities comprises: The similarity representation quantities corresponding to the similarities form a corresponding similarity matrix; The constructing the original matching data according to the similarity matrix comprises: Each similarity matrix is ​​determined to be data of a channel in a three-dimensional tensor, so as to use the three-dimensional tensor as the original matching data.

7. The method of claim 3, wherein: The similarity includes one or more of the following similarity representation quantities: intersection over union ratio, position difference, and similarity of visual features of images within a frame.

8. A device for determining multiple target trajectories in a video, comprising: The target detection module is used to perform target detection on the current video frame to obtain detection frames of all targets in the current video frame; A target prediction module, used to predict a prediction box of a corresponding target in the current video frame based on the trajectory of the target in the previous video frame; A similarity calculation module, used to calculate the similarity between the detection frame and the prediction frame respectively to obtain original matching data; a matching module, configured to input the original matching data into a matching neural network, so that the matching neural network outputs final matching data between the detection box and the trajectory, wherein the final matching data includes data indicating a matching degree between the detection box and the trajectory; as well as A storage module, used for storing the detection frame into a matching trajectory according to the final matching data; The performing of target detection on the current video frame to obtain detection frames of all targets in the current video frame, the predicting of prediction frames of corresponding targets in the current video frame based on the trajectories of targets in previous video frames, and the respectively calculating of the similarities between the detection frames and the prediction frames to obtain the original matching data are all implemented using a first neural network, the first neural network and the matching neural network are interconnected, and the first neural network outputs the original matching data to the matching neural network; The first neural network and the matching neural network are trained simultaneously using training videos and corresponding annotated data, wherein the annotated data includes position information of a real trajectory of a target in the training video.

9. A system for determining trajectories of multiple objects in a video, comprising a processor and a memory, wherein: The memory stores computer program instructions, which are used by the processor to execute the method for determining multiple target trajectories in a video as claimed in any one of claims 1 to 7 when the computer program instructions are executed.

10. A storage medium having program instructions stored thereon, wherein the program instructions are used to execute the method for determining trajectories of multiple objects in a video according to any one of claims 1 to 7 when run.

Images