Motion direction prediction method, electronic device and storage medium

Abstract

The present application discloses a motion direction prediction method, an electronic device and a storage medium. In the present application, the electronic device pre-trains a motion direction prediction model. For a target image, after determining the detection frame sub-image of the target object to be tracked, the sub-image is input into the motion direction prediction model. Based on the motion direction prediction model, the posture key points of the target object in the sub-image are first determined, and then the motion direction probability value of the target object is determined based on the sub-image and the posture key points. Finally, based on the motion direction probability value, the target motion direction of the target object is predicted, and the target object can be tracked based on the target motion direction. Compared with the solution of realizing target tracking based on the target detection results of a video collected over a period of time, the present application improves the tracking efficiency and has better real-time tracking. The solution included in the present application has the characteristics of high accuracy and fast inference speed, and meets the trustworthy characteristics.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a motion direction prediction method, electronic device, and storage medium. Background Art

[0002] Due to their long history, the old urban areas of major cities often have complex alleys and lanes with intersecting paths and forks. These complex road networks are interconnected and make target tracking difficult. In target detection and tracking scenarios, the following solutions are currently commonly used:

[0003] The first method involves rationally positioning detection frames, detecting pixel changes within multiple detection frames, and using inter-frame differencing to analyze the significant differences between the head and tail of a target. This allows for secondary analysis of pixels in the moving regions of the frame-differenced image. The second method involves joint tracking and classification of irregular multi-extended targets based on multi-Bernoulli algorithms. The third method uses computer vision technology based on deep learning models to perceive traffic flow parameters such as the speed, size, and volume of moving targets in a traffic system. This allows for real-time traffic monitoring and automated video detection and analysis of target motion and abnormal behavior.

[0004] The above three solutions all achieve target tracking based on target detection results of videos collected over a period of time. The tracking efficiency is low and the real-time performance is poor. Summary of the Invention

[0005] The present application provides a motion direction prediction method, electronic device and storage medium to solve the problems of low efficiency and poor real-time performance of target tracking in the prior art.

[0006] In a first aspect, the present application provides a motion direction prediction method, the method comprising:

[0007] Determine a detection frame sub-image of a target object to be tracked in a target image;

[0008] Inputting the sub-image into a pre-trained motion direction prediction model for processing to obtain a motion direction probability value of the target object; wherein the motion direction prediction model is used to determine the posture key points of the target object in the sub-image, and determining the motion direction probability value based on the sub-image and the posture key points;

[0009] The target motion direction of the target object is predicted according to the motion direction probability value.

[0010] In a second aspect, the present application provides a motion direction prediction device, the device comprising:

[0011] A determination module, configured to determine a detection frame sub-image of a target object to be tracked in a target image;

[0012] an input module, configured to input the sub-image into a pre-trained motion direction prediction model for processing to obtain a motion direction probability value of the target object; wherein the motion direction prediction model is configured to determine posture key points of the target object in the sub-image, and determine the motion direction probability value based on the sub-image and the posture key points;

[0013] A prediction module is used to predict the target movement direction of the target object according to the movement direction probability value.

[0014] In a third aspect, the present application provides an electronic device, comprising a processor, a communication interface, a memory, and a communication bus, wherein the processor, the communication interface, and the memory communicate with each other via the communication bus;

[0015] Memory for storing computer programs;

[0016] The processor is used to implement the method steps when executing the program stored in the memory.

[0017] In a fourth aspect, the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described are implemented.

[0018] The present application provides a motion direction prediction method, electronic device and storage medium, the method comprising: determining a detection frame sub-image of a target object to be tracked in a target image; inputting the sub-image into a pre-trained motion direction prediction model for processing to obtain a motion direction probability value of the target object; wherein the motion direction prediction model is used to determine the posture key points of the target object in the sub-image, and determine the motion direction probability value based on the sub-image and the posture key points; and predicting the target motion direction of the target object according to the motion direction probability value.

[0019] The above technical solution has the following advantages or beneficial effects:

[0020] In the present application, the electronic device pre-trains a motion direction prediction model, and for the target image, after determining the detection frame sub-image of the target object to be tracked, the sub-image is input into the motion direction prediction model. Based on the motion direction prediction model, the posture key points of the target object in the sub-image are first determined, and then the motion direction probability value of the target object is determined based on the sub-image and the posture key points. Finally, based on the motion direction probability value, the target motion direction of the target object is predicted, and the target object can be tracked based on the target motion direction. Compared with the solution of realizing target tracking based on the target detection results of the video collected over a period of time, the present application improves the tracking efficiency and has better real-time tracking. The solution included in the present application has the characteristics of high accuracy and fast inference speed, and meets the trustworthy characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0022] Figure 1 Schematic diagram of the first motion direction prediction process provided by this application;

[0023] Figure 2 A schematic diagram of the second motion direction prediction process provided by this application;

[0024] Figure 3 Schematic diagram of the training process of the motion direction prediction model provided in this application;

[0025] Figure 4 Schematic diagram of the process of obtaining sample continuous frame videos in the training set provided by this application;

[0026] Figure 5 Schematic diagram of the third motion direction prediction process provided by this application;

[0027] Figure 6 Schematic diagram of the fourth motion direction prediction process provided by this application;

[0028] Figure 7 Schematic diagram of the fifth motion direction prediction process provided by this application;

[0029] Figure 8 Schematic diagram of the sixth motion direction prediction process provided by this application;

[0030] Figure 9 The motion direction prediction framework diagram provided for this application;

[0031] Figure 10 Schematic diagram of the motion direction prediction model training module framework provided in this application;

[0032] Figure 11 This is the target detection interface diagram provided for this application;

[0033] Figure 12 Schematic diagram of target object selection provided for this application;

[0034] Figure 13 This is a diagram of the motion direction prediction application module framework provided by this application;

[0035] Figure 14 This is the framework diagram of the motion direction prediction and deployment module provided by this application;

[0036] Figure 15 A schematic diagram of the tripwire configuration for the algorithm provided in this application;

[0037] Figure 16 Schematic diagram of the tripwire area configured for the algorithm provided in this application;

[0038] Figure 17 A schematic diagram of the preliminary results of the algorithm for estimating the possible motion directions of each detected target provided in this application;

[0039] Figure 18 This is a schematic diagram of the algorithm estimation results of the possible motion directions of each detected target provided by this application;

[0040] Figure 19 This is a schematic diagram of the results of moving target estimation in the tripwire area provided by this application;

[0041] Figure 20 This is a schematic diagram of the structure of the motion direction prediction device provided by this application;

[0042] Figure 21 This is a schematic diagram of the electronic device structure provided in this application. DETAILED DESCRIPTION

[0043] In order to make the purpose and implementation of this application clearer, the exemplary implementation of this application will be clearly and completely described below in conjunction with the drawings in the exemplary embodiments of this application. Obviously, the described exemplary embodiments are only part of the embodiments of this application, not all of the embodiments.

[0044] It should be noted that the brief descriptions of terms in this application are only for the purpose of facilitating the understanding of the embodiments described below, and are not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their ordinary and usual meanings.

[0045] In the specification and claims of this application and the accompanying drawings, the terms "first," "second," "third," etc. are used to distinguish similar or similar objects or entities, and are not necessarily intended to limit a particular order or sequence, unless otherwise noted. It should be understood that the terms used in this manner are interchangeable under appropriate circumstances.

[0046] The terms "comprise," "include," and "have," and any variations thereof, are intended to cover but not exclude inclusion; for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0047] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functionality associated with that element.

[0048] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some or all of the technical features therein. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present application.

[0049] For ease of explanation, the above description has been made with reference to specific embodiments. However, the above exemplary discussion is not intended to be exhaustive or to limit the embodiments to the specific forms disclosed above. Based on the above teachings, various modifications and variations are possible. The above embodiments are selected and described to better explain the principles and practical applications, so that those skilled in the art can better utilize the embodiments and various different variations of the embodiments suitable for specific use considerations.

[0050] Figure 1 This is a schematic diagram of the motion direction prediction process provided by this application. The process includes the following steps:

[0051] S101: Determine a detection frame sub-image of a target object to be tracked in a target image.

[0052] S102: Input the sub-image into a pre-trained motion direction prediction model for processing to obtain a motion direction probability value of the target object; wherein the motion direction prediction model is used to determine the posture key points of the target object in the sub-image, and determine the motion direction probability value based on the sub-image and the posture key points.

[0053] S103: Predicting the target movement direction of the target object according to the movement direction probability value.

[0054] The motion direction prediction method provided in the present application is applied to an electronic device, which may be a PC, a tablet computer, or a server.

[0055] The electronic device determines a detection frame sub-image of a target object to be tracked in a target image. A user may select a detection frame of the target object to be tracked in the target image, and the electronic device determines the sub-image contained in the selected detection frame. Alternatively, the electronic device may detect the target object to be tracked in the target image using a target object detection algorithm or a pre-trained target object detection model to obtain a detection frame of the target object, and determine the sub-image contained in the selected detection frame.

[0056] The electronic device stores a pre-trained motion direction prediction model. Optionally, the motion direction prediction model can be trained based on sample images in a training set and information about the motion directions of sample objects annotated in the sample images. During training, the motion direction prediction model first extracts pose keypoints from the sample objects in the sample images. Then, based on the pose keypoints of the sample images and the sample objects, the motion direction of the sample objects is estimated. A model loss value is calculated based on the estimated motion direction and the annotated motion direction. Model parameters of the motion direction prediction model are adjusted based on the model loss value. When the model loss value meets the required value, the motion direction prediction model training is complete. The electronic device inputs a sub-image of a target object's detection frame into the pre-trained motion direction prediction model. The motion direction prediction model first determines the pose keypoints of the target object in the sub-image. Then, based on the sub-image and the pose keypoints, a probability value for the target object's motion direction is determined. Optionally, the probability values ​​of the target object moving in multiple surrounding directions can be determined. Based on the motion direction probabilities, the target motion direction of the target object is predicted. Optionally, the motion direction with the highest probability value is used as the predicted target motion direction of the target object.

[0057] For example, based on the sub-image and the pose keypoints, the probability values ​​for the target object's motion direction are 80% for the target object moving forward, 30% for the target object moving backward, 50% for the target object moving left, and 40% for the target object moving right. The probability value for forward motion is the highest, so the predicted target object motion direction is forward motion.

[0058] In the present application, the electronic device pre-trains a motion direction prediction model, and for the target image, after determining the detection frame sub-image of the target object to be tracked, the sub-image is input into the motion direction prediction model. Based on the motion direction prediction model, the posture key points of the target object in the sub-image are first determined, and then the motion direction probability value of the target object is determined based on the sub-image and the posture key points. Finally, based on the motion direction probability value, the target motion direction of the target object is predicted, and the target object can be tracked based on the target motion direction. Compared with the solution of realizing target tracking based on the target detection results of the video collected over a period of time, the present application improves the tracking efficiency and has better real-time tracking. The solution included in the present application has the characteristics of high accuracy and fast inference speed, and meets the trustworthy characteristics.

[0059] Trustworthy features:

[0060] (1) Real-time performance: This solution can provide real-time feedback on the inference results of the acquired frames while extracting one frame every five frames of the video, which meets the real-time feature of the trustworthy characteristic.

[0061] (2) Controllability: Whether it is the selection of the model prediction direction or the setting of the threshold, or the drawing of the on-site trip wire area, the user's preferences can be added for adjustment. The method and device for predicting the direction of the moving target can set the time period for opening and closing, which meets the characteristics of controllability.

[0062] Considering that the movement of the target object is continuous, in order to more accurately predict the movement direction of the target object, in this application, the method includes:

[0063] Acquire multiple images including the target image acquired within a preset period, and determine a detection frame sub-image of the target object in the multiple images;

[0064] Multiple sub-images are input into the motion direction prediction model for processing to obtain the motion direction probability value of the target object; wherein the motion direction prediction model is used to determine the posture key points of the target object corresponding to each of the multiple sub-images, and determine the motion direction probability value based on the multiple sub-images and the corresponding posture key points.

[0065] In the present application, the electronic device acquires a plurality of images including a target image captured within a preset period. Preferably, the last frame image captured within the preset period can be used as the target image. The preset period is, for example, 2 seconds, 3 seconds, etc. After acquiring a plurality of images, the detection frame sub-images of the target object in the plurality of images are respectively determined, and then the plurality of sub-images are respectively input into the motion direction prediction model. The motion direction prediction model first determines the posture key points of the target object corresponding to each of the plurality of sub-images, and determines the correspondence between the sub-images and the posture key points. That is, a plurality of sub-images and the posture key points corresponding to each of the plurality of sub-images are determined. Finally, based on the plurality of sub-images and the posture key points corresponding to each of the sub-images, the motion direction probability value of the target object in the target image is determined.

[0066] Optionally, during training of the motion direction prediction model, a first sub-model within the motion direction prediction model can be trained using sample continuous frame images from a training set. The first sub-model is used to learn the correspondence between sub-images and pose key points. During training of the first sub-model, the first sub-model is trained using the sample continuous frame images from the training set and the correspondence between pre-labeled sample images and pose key points of sample objects. A second sub-model within the motion direction prediction model is trained using the motion directions of sample objects corresponding to the pre-labeled sample continuous frame images. The second sub-model is used to learn the motion direction information of the sample objects. Multiple images, including a target image, acquired within a preset period are obtained, and detection frame sub-images of the target object in the multiple images are determined. The multiple sub-images are then input into the motion direction prediction model. Based on the first sub-model within the motion direction prediction model, the pose key points of the target object corresponding to each of the multiple sub-images can be determined. Based on the second sub-model within the motion direction prediction model, a motion direction probability value of the target object in the target image can be determined based on the multiple sub-images and their corresponding pose key points.

[0067] Figure 2 The motion direction prediction process provided in this application includes the following steps:

[0068] S201: Acquire multiple images including the target image collected within a preset period, and determine a detection frame sub-image of the target object in the multiple images.

[0069] S202: Input multiple sub-images into the motion direction prediction model for processing to obtain the motion direction probability value of the target object; wherein the motion direction prediction model is used to determine the posture key points of the target object corresponding to each of the multiple sub-images, and determine the motion direction probability value based on the multiple sub-images and the corresponding posture key points.

[0070] S203: Predicting the target movement direction of the target object according to the movement direction probability value.

[0071] In order to make the prediction results of the motion direction prediction model more accurate, in this application, the training process of the motion direction prediction model includes:

[0072] Obtaining a sample continuous frame video in a training set, inputting sample images in the sample continuous frame video, posture key points of sample objects in the sample images, and the annotated correspondences between the sample images and the posture key points of the sample objects into an initial motion direction prediction model, and training the initial motion direction prediction model;

[0073] The sample image in the sample continuous frame video, the posture key points of the sample object in the sample image, and the semantic description information corresponding to the sample continuous frame video are input into the trained initial motion direction prediction model, and the trained initial motion direction prediction model is trained to obtain the motion direction prediction model.

[0074] In the present application, an electronic device obtains a large number of sample continuous frame videos in a training set, and for each frame of sample image, annotates the correspondence between the frame of sample image and the posture key points of the sample object in the frame of sample image. For each sample continuous frame video, the semantic description information of the sample continuous frame video is annotated. The semantic description information carries the motion direction information of the sample object. For example, the semantic description information is "the man in the yellow-green jersey ran from the middle of the square to the upper right of the screen, then the man ran back to the middle of the screen, then the man ran to the upper left of the screen, then the man turned away from the screen along the center line of the screen and disappeared above the screen." The sample images in the sample continuous frame video, the posture key points of the sample objects in the sample images, and the correspondence between the annotated sample images and the posture key points of the sample objects are input into the initial motion direction prediction model to train the initial motion direction prediction model. This enables the motion direction prediction model to learn the correspondence between the sample images and the posture key points of the sample objects.

[0075] Furthermore, the sample images in the sample continuous frame video, the posture key points of the sample objects in the sample images, and the semantic description information corresponding to the sample continuous frame video are input into the trained initial motion direction prediction model, and the trained initial motion direction prediction model is trained to obtain a motion direction prediction model. This enables the motion direction prediction model to learn the motion direction information of the sample objects.

[0076] Figure 3 The training process of the motion direction prediction model provided in this application includes the following steps:

[0077] S301: Obtain a sample continuous frame video in a training set, input the sample image in the sample continuous frame video, the posture key points of the sample object in the sample image, and the marked correspondence between the sample image and the posture key points of the sample object into an initial motion direction prediction model, and train the initial motion direction prediction model.

[0078] S302: Input the sample image in the sample continuous frame video, the posture key points of the sample object in the sample image, and the semantic description information corresponding to the sample continuous frame video into the trained initial motion direction prediction model, train the trained initial motion direction prediction model, and obtain the motion direction prediction model.

[0079] Considering that there are few sample continuous frame videos in the database, in order to enrich the training set and thus obtain a motion direction prediction model with higher accuracy, in this application, each sample continuous frame video stored in the database is referred to as each first sample continuous frame video. Each first sample continuous frame video has corresponding semantic description information. The electronic device obtains the first sample continuous frame video stored in the database, and the semantic description information corresponding to the first sample continuous frame video. Then, the human body posture transformation algorithm is used to transform the human body posture in the first sample image in the first sample continuous frame video to obtain a second sample continuous frame video. For example, the sample object in the first sample image is facing the stage, and the human body posture transformation algorithm is used to transform the posture of the sample object in the first sample image, transforming it from facing the stage to facing away from the stage. The sample object in the first sample continuous frame video is continuously transformed in posture, so that a second sample continuous frame video can be obtained, and the semantic description information corresponding to the second sample continuous frame video is generated based on the posture information of the sample object in the second sample continuous frame video.

[0080] Finally, the first sample continuous frame video and the second sample continuous frame video are used as sample continuous frame videos in the training set, thereby expanding the number of sample continuous frame videos in the training set and improving the accuracy of the motion direction prediction model.

[0081] Figure 4 The process diagram for obtaining sample continuous frame videos in the training set provided by this application includes the following steps:

[0082] S401: Acquire a first sample continuous frame video stored in a database, and semantic description information corresponding to the first sample continuous frame video.

[0083] S402: using a human body posture transformation algorithm, transforming the human body posture in the first sample image in the first sample continuous frame video to obtain a second sample continuous frame video, and generating semantic description information corresponding to the second sample continuous frame video.

[0084] S403: Use the first sample continuous frame video and the second sample continuous frame video as sample continuous frame videos in a training set.

[0085] In the present application, a stable diffusion model (SD) can be used to transform the human body posture in the first sample image in the first sample continuous frame video to obtain a second sample continuous frame video. Preferably, considering that SD is better at generating scene pictures from single-frame images, but due to the randomness and uncertainty of the SD model scene picture generation, SD has limited ability in generating event continuous frame videos, so the present application re-pre-trains the original stable diffusion model SD. During the model pre-training process, the training data set is supplemented with a large number of continuous frames of event videos, and the optical flow information and semantic information of each continuous frame are added to obtain a pre-training model of the continuous-frame diffusion model (CFDM-PC). CFDM-PC is used to transform the human body posture in the first sample image in the first sample continuous frame video to obtain a second sample continuous frame video.

[0086] In order to determine the probability value of the moving direction of the target object more accurately, in this application, determining the probability value of the moving direction of the target object includes:

[0087] Determine the movement direction probability values ​​of the target object toward multiple directions, and enhance the movement direction probability value of the tripwire direction toward a preset tripwire area according to a preset probability value.

[0088] In the present application, a tripwire area is pre-set, and the tripwire area is an area of ​​focus. After determining the probability values ​​of the target object's movement direction toward multiple directions, for example, if the tripwire area is on the left side of the target object, then the left direction of the target object is determined to be the tripwire direction toward the tripwire area; for another example, if the tripwire area is on the right side of the target object, then the right direction of the target object is determined to be the tripwire direction toward the tripwire area. According to the preset probability value, the movement direction probability value of the tripwire direction is enhanced. Optionally, the movement direction probability value of the tripwire direction is added to the preset probability value to obtain the final movement direction probability value of the tripwire direction. For example, the right direction of the target object is the tripwire direction toward the tripwire area, and the probability value of the right direction of the target object determined based on the movement direction prediction model is 50%, and the preset probability value is 20%. Then, according to the preset probability value, the probability value of the movement direction probability value of the tripwire direction after enhancement is 50%+20%=70%.

[0089] Figure 5 The motion direction prediction process provided in this application includes the following steps:

[0090] S501: Determine a detection frame sub-image of a target object to be tracked in a target image.

[0091] S502: Input the sub-image into a pre-trained motion direction prediction model for processing to obtain motion direction probability values ​​of the target object in multiple directions, and enhance the motion direction probability value of the trip wire direction in the trip wire direction toward a preset trip wire area according to the preset probability value.

[0092] S503: Predicting the target movement direction of the target object according to the movement direction probability value.

[0093] In the present application, the step of enhancing the movement direction probability value of the tripwire direction according to a preset probability value in the tripwire direction toward the preset tripwire area includes:

[0094] Determine the first position information of the target object in the target image. If the trigger probability enhancement condition is determined based on the first position information and the second position information of the preset tripwire area, the movement direction probability value of the tripwire direction is enhanced according to a preset probability value in the tripwire direction toward the preset tripwire area.

[0095] In the present application, when the target object is far away from the tripwire area, the probability value of the motion direction in the tripwire direction may not be enhanced. When the target object is close to the tripwire area, the probability value of the motion direction in the tripwire direction is enhanced. Specifically, the first position information of the target object in the target image is determined, and the second position information of the tripwire area is pre-saved. According to the first position information and the second position information, it is determined whether the probability enhancement condition is triggered. The probability enhancement condition is, for example, when it is determined that the distance between the target object and the tripwire area is less than a set distance threshold according to the first position information and the second position information, the probability enhancement condition is triggered; or when it is determined that the target object and the tripwire area intersect according to the first position information and the second position information, the probability enhancement condition is triggered. When the probability enhancement condition is determined to be triggered, the probability value of the motion direction in the tripwire direction toward the preset tripwire area is enhanced according to the preset probability value.

[0096] Figure 6 The motion direction prediction process provided in this application includes the following steps:

[0097] S601: Determine a detection frame sub-image of a target object to be tracked in a target image.

[0098] S602: Input the sub-image into a pre-trained motion direction prediction model for processing to obtain motion direction probability values ​​of the target object in multiple directions, determine the first position information of the target object in the target image, and if the trigger probability enhancement condition is determined based on the first position information and the second position information of the preset trip wire area, the motion direction probability value of the trip wire direction is enhanced according to the preset probability value in the trip wire direction toward the preset trip wire area.

[0099] S603: Predicting the target movement direction of the target object according to the movement direction probability value.

[0100] Figure 7 The motion direction prediction process provided in this application includes the following steps:

[0101] S701: Determine a detection frame sub-image of a target object to be tracked in a target image.

[0102] S702: Input the sub-image into a pre-trained motion direction prediction model for processing to obtain motion direction probability values ​​of the target object in multiple directions, determine the first position information of the target object in the target image, and if the trigger probability enhancement condition is determined based on the first position information and the second position information of the preset trip wire area, the motion direction probability value of the trip wire direction is enhanced according to the preset probability value in the trip wire direction toward the preset trip wire area.

[0103] S703: Compare the motion direction probability values ​​of multiple motion directions, select a target motion direction probability value according to the comparison result; and predict the target motion direction of the target object according to the target motion direction probability value.

[0104] In the present application, optionally, the direction of motion probability values ​​of multiple directions of motion are compared, and the maximum direction of motion probability value is selected as the target direction of motion probability value based on the comparison result. Alternatively, based on the comparison result, multiple direction of motion probability values ​​are selected in a descending order as the target direction of motion probability value. Among them, the direction of motion probability values ​​of multiple directions of motion can be sorted in a descending order, and then the direction of motion probability value at the top of the sorting order is selected as the target direction of motion probability value, and the direction of motion corresponding to the target direction of motion probability value is used as the target direction of motion of the predicted target object. Alternatively, the direction of motion probability values ​​of multiple directions of motion can be sorted in a descending order, and then the direction of motion probability value at the bottom of the sorting order is selected as the target direction of motion probability value, and the direction of motion corresponding to the target direction of motion probability value is used as the target direction of motion of the predicted target object. In some cases, after sorting the direction of motion probability values ​​of multiple directions of motion, at least two direction of motion probability values ​​can be selected in a descending order as the target direction of motion probability value, so that the target direction of motion of the predicted target object outputted is at least two. Based on the at least two target directions of motion outputted, the management personnel can carry out tracking management or early warning on the target object.

[0105] In order to remind the administrator that the target object may enter the tripwire area, in this application, the method further includes:

[0106] If the target movement direction is the tripwire direction, and the movement direction probability value corresponding to the target movement direction is greater than a preset alarm threshold, a high probability warning prompt information indicating the tripwire direction is output.

[0107] The preset alarm threshold is, for example, 60%, 70%, etc. Optionally, a warning message indicating a high probability of tripping the wire direction may be outputted by an audible and visual alarm; or a warning message indicating a high probability of tripping the wire direction may be sent to a terminal device of a manager.

[0108] Figure 8 The motion direction prediction process provided in this application includes the following steps:

[0109] S801: Determine a detection frame sub-image of a target object to be tracked in a target image.

[0110] S802: Input the sub-image into a pre-trained motion direction prediction model for processing to obtain a motion direction probability value of the target object; wherein the motion direction prediction model is used to determine the posture key points of the target object in the sub-image, and determine the motion direction probability value based on the sub-image and the posture key points.

[0111] S803: Predicting the target movement direction of the target object according to the movement direction probability value.

[0112] S804: If the target movement direction is the tripwire direction, and the movement direction probability value corresponding to the target movement direction is greater than a preset alarm threshold, outputting a high probability warning prompt information indicating the tripwire direction.

[0113] The motion direction prediction process provided by this application is described in detail below with reference to the accompanying drawings.

[0114] Combining a database of target object motion direction videos and images, a semantic description is formed for the target's motion frames, as well as the side and back views within each frame. This semantic description is then combined with a continuous frame diffusion model with added posture conditions (CFDM-PC) and a self-attention mechanism (or GAN) to generate a continuous frame generation dataset of 6 to 8 positions of the target object's future motion direction. A self-supervised learning model is trained, leveraging continuous frame video data collected on-site and a training set generated by the self-attention mechanism (or GAN). The trained model can predict the probability of a moving target's forward or backward motion direction, providing tracking information on the possible motion directions and improving tracking efficiency.

[0115] Figure 9 The motion direction prediction framework diagram provided by this application is as follows: Figure 9 As shown, it includes a motion direction prediction model training module, a motion direction prediction deployment module, and a motion direction prediction application module. The motion direction prediction model training module includes semantic data annotation description, data collected on-site, a continuous frame diffusion model with added posture conditions (CFDM-PC), a continuous frame generation data set of 6 to 8 directions of the moving target's future motion direction, a motion direction prediction model obtained through self-supervised learning training, and predicts the probability values ​​of the moving target moving in multiple directions. The motion direction prediction deployment module includes drawing trip wires for each complex intersection, drawing trip wire areas (no-entry areas) for each complex intersection, providing reinforced incremental learning information, and forward guiding the moving target motion direction prediction model. The motion direction prediction application module includes a rectangular search box for each target in the picture, circling the target object, tracking the target, and outputting a visual list of the target object's motion direction probability values ​​based on the motion direction prediction model.

[0116] When the motion direction prediction model is obtained through self-supervised learning training, 15% of the sample continuous frame videos come from the real video library and image library (the first sample continuous frame video), and 85% of the data comes from the data generated by the CFDM-PC self-attention mechanism (or GAN) containing richer motion direction possibility information (the second sample continuous frame video).

[0117] In the square rally scene, the original video footage recorded by the surveillance camera shows that the running routes of the people are extremely irregular. The number of surveillance video footage that can be collected worldwide is relatively small, less than 20,000 continuous frame images. Based on the above considerations, in order to enrich the training set, in this application, first, the original video recorded by the above-mentioned square rally surveillance is used to extract the skeleton key points of each frame image in the personnel video, and the extracted skeleton points are matched one-to-one with each frame in the video to produce "frame image-posture key point" data. At the same time, the "frame image-posture key point" video of the continuous frames is semantically annotated, such as: "The man in the yellow-green jersey ran from the middle of the square to the upper right corner of the screen, then the man ran back to the middle of the screen, then the man ran to the upper left corner of the screen, then the man left the screen along the center line of the screen and disappeared above the screen", thereby forming a "frame image-posture key point (semantic annotation)" video pair, which is used as one of the video data in the original video training set. According to the above steps, all the original videos of the square assembly surveillance records that can be collected are matched with the "frame image-gesture key point (semantic annotation)" video pairs.

[0118] Then, a large number of "irregular motion direction video pairs" datasets of moving targets newly generated by CFDM-PC are used. Combined with the above-mentioned "frame image-pose key point (semantic annotation)" video pair matching dataset, human posture transformation algorithms are used, such as "Continuous-Frame Diffusion Models with PoseConditions", ControlNet and its Pose function, "pose generation adversarial network (PoseGAN)," etc., by modifying the existing posture key points in the original dataset "frame image-pose key point" data, such as changing the continuous posture of the departure direction to a certain angle of departure, such as a posture of 20° or 15° to the side, or changing the continuous posture of running to the upper right of the screen to a continuous posture of running to the upper left. In the above posture generation process, ensure that the generated image of each frame is paired with the corresponding posture to form a newly generated "frame image-pose key point" data, and in the newly generated "frame image-pose key point" While collecting the "gesture key point" data, the corresponding semantic annotation description is added, such as: "The woman in the black coat ran from the center of the square to the left side of the screen, then ran to the upper left side of the screen, then ran to the upper center of the screen, and then ran out of the screen from the upper right side of the screen.", thereby generating a new "frame image-gesture key point (semantic annotation)" video pair. In this way, a series of "irregular motion direction video pairs" of different targets are newly generated, and the possible motion routes and directions of the square in the corresponding original surveillance video are covered as much as possible, thereby obtaining a large number of "irregular motion direction video pairs" datasets of different targets in the square. According to the above steps, the "frame image-gesture key point (semantic annotation)" video pairs of the original video are modified in large quantities, and a large number of new "irregular motion direction video pairs" datasets of moving targets are generated.

[0119] Figure 10 The framework diagram of the motion direction prediction model training module provided in this application includes a video and image database of the motion direction of the moving target, continuous frames of the target object's motion picture and semantic descriptions such as the side and back of the moving target in each frame, data collected on-site, and a continuous frame diffusion model (CFDM-PC) with added posture conditions. The function of CFDM-PC is to amplify the continuous frame videos in the video and image database to obtain a continuous frame data set of the motion direction of the moving target in 6-8 directions. CFDM-PC amplifies the continuous frame videos in the video and image database to strengthen the incremental learning information. Combined with the data collected on-site and the continuous frame data set of the motion direction in 6-8 directions, the motion direction prediction model is obtained by self-supervised learning. The motion direction prediction model predicts the probability value of the moving direction of the moving target forward or backward and outputs it.

[0120] This application pre-trains the original stable diffusion model (SD, Stable Diffusion, which works well when generating scene pictures from single-frame images, but due to the randomness and uncertainty of the SD model scene picture generation, SD has limited ability in generating event continuous videos) based on the original stable diffusion model. During the model pre-training process, the training data set is supplemented with a large number of continuous frames of event videos, and the optical flow information of each continuous frame is added to supplement the training data set, thereby obtaining a pre-training model of the continuous-frame diffusion model (CFDM-PC). CFDM-PC is an improvement based on SD. On the basis of SD, prompts such as continuous frame images, posture information and optical flow information are added to train CFDM-PC, so that the effect of generating event continuous videos based on CFDM-PC is better.

[0121] Then, when fine-tuning the pre-trained CFDM-PC model for the downstream task of "pose pair generation," the pose keypoints corresponding to each frame of the training set are introduced as input conditions. A prompt dialog box is displayed to provide a semantic description of the target pose motion in each frame, such as "the man in red is walking at a 45-degree angle to the side." This trains a continuous-frame CFDM model capable of understanding "image-pose pairs." This continuous-frame CFDM model can serve as the initial motion direction prediction model. Furthermore, a prompt is used to provide a semantic description of the entire continuous-frame video pair, such as "the man in the yellow-green jersey runs from the center of the square to the upper right of the frame, then runs back to the center, then to the upper left, then away from the frame along the centerline and disappears at the top of the frame." This ultimately trains and fine-tunes the "Continual-Frame Diffusion Model with Pose Conditions," which has superior performance for the downstream task of "pose pair generation." In this application, CFDM-PC and the motion direction prediction model can be arranged separately, that is, the training set is expanded using CFDM-PC, and then the independently arranged motion direction prediction model is trained using the expanded training set. When training the motion direction prediction model, the initial motion direction prediction model is first trained to obtain the initial motion direction prediction model, which can learn the correspondence between the image frame and the posture key points. The initial motion direction prediction model is then retrained to obtain the final motion direction prediction model, so that the motion direction prediction model can learn the probability values ​​of the moving target in the image moving in various directions. The initial motion direction prediction model can also be used as the first sub-model in the motion direction prediction model, and the probability values ​​of the moving target in the predicted image moving in various directions can be used as the second sub-model.

[0122] The self-supervised training of the probability prediction model for each "irregular motion direction" of the moving target is as follows:

[0123] After obtaining the original video "frame image-pose key point (semantic annotation)" video pairs and a large number of moving target "irregular motion direction video pair" data sets, these data sets contain "frame image-pose key point" matching pairs, semantic descriptions of video pairs, and "irregular motion directions" of different targets. Therefore, using this data set for self-supervised training, the obtained probability prediction model can learn what is the real-time motion position or direction of an object, and has the ability to identify the probability of each "irregular motion direction" of a moving target from the continuous frames of video captured by the surveillance camera.

[0124] At the same time, in the motion direction prediction application module, a rectangular search box is provided for each target in the picture. The search box of the monitoring picture is configurable. On the frame image of the real-time picture, you can manually circle a rectangular box for a specific single target or multiple targets by quickly clicking through the operation interface.

[0125] Figure 11 The target detection interface diagram provided for this application includes pedestrian detection, human body detection, pedestrian and cyclist detection, escalator head detection, tripwire area intrusion detection, driver detection, helmet detection, large luggage detection, smoking and phone behavior detection, etc.

[0126] Figure 12 Schematic diagram of target object selection provided for this application, Figure 12 The person within the rectangular box is the target object. After the box is selected, the target object is tracked and its direction of movement is predicted. As the target rectangular box is provided to the frame image, the target tracking algorithm running in the background immediately circles the target and initiates target tracking. Simultaneously, the target direction estimation algorithm obtains the frame image and estimates the probability of the future movement direction of the target within the specific rectangular box in the image.

[0127] The motion direction prediction algorithm is implemented by combining the motion direction prediction model of the moving target's "irregular motion direction" with the actual algorithm business logic.

[0128] Input: The algorithm application platform interface provides moving target selection through quick clicks or frame selection. One or more targets can be selected. After the target object is selected, each frame of the real-time video will frame the selected target and track it in real time.

[0129] Algorithm startup: When a target is selected, the algorithm dynamically calls and runs the "Moving Target Irregular Motion Direction" Motion Direction Prediction Model, begins to estimate the direction probability of the moving target in real time, and periodically reports and outputs the estimated motion direction probability level (for example, every 3 seconds);

[0130] Reinforcement Learning: When a selected target reaches the restricted zone or tripwire detection trigger conditions, the estimated probability value of the "Moving Target 'Irregular Motion Direction' Motion Direction Prediction Model" in that direction is strengthened, significantly increasing the probability of the target moving in that direction.

[0131] Output: When the estimated movement direction of the selected target significantly increases to exceed a certain threshold, such as a 55% probability threshold, an alarm indicating that the estimated direction has reached a high probability is issued, and the backend notifies the front-line personnel to take action.

[0132] Figure 13 The framework diagram of the motion direction prediction application module provided in this application includes a rectangular search box for each target in the picture, quick click to quickly circle the rectangular box, circle the target object, target tracking, and output a visual list of the motion direction probability value of the target object based on the motion direction prediction model.

[0133] Figure 14 The framework diagram of the motion direction prediction deployment module provided in this application, while the tracking algorithm and the target motion estimation algorithm are deployed, the algorithm configures the trip wires and no-entry areas of each complex intersection, and these configured trip wire areas provide enhanced incremental learning information for the target motion direction estimation algorithm. The motion direction prediction deployment module includes drawing the trip wires of each complex intersection, drawing the trip wire areas (no-entry areas) of each complex intersection, providing enhanced incremental learning information and target motion direction estimation algorithm training and probability estimation. The enhanced incremental learning information includes the business practice of the detection and estimation algorithm, the increase in the probability value of the direction with a large number of successful detections, and the positive guidance of the moving target motion direction prediction model.

[0134] Figure 15 A schematic diagram of the tripwire configuration for the algorithm provided in this application, Figure 16 Schematic diagram of the tripwire area configured for the algorithm provided in this application. After multiple implementations of the detection and estimation algorithms, the probability of target motion directions with a high number of tripwires or intrusions will increase, thereby positively guiding the moving target direction estimation algorithm to continuously improve its direction prediction success rate.

[0135] Figure 17 The schematic diagram of the preliminary results of the algorithm estimation of the possible movement directions of each detection target provided in this application. After quickly clicking and selecting these five moving targets in sequence, the initial possible movement direction of each target is shown as the circle in the figure. During actual application, each circle will display the predicted probability value of the direction.

[0136] Figure 18The schematic diagram of the algorithm estimation results of the possible movement direction of each detected target provided in this application shows that after the target has been moving continuously for more than 1 second, the "moving target "irregular movement direction" probability prediction model" can give several higher direction probability values, and some of these probability values ​​will exceed the threshold (such as a 55% probability), thereby triggering an early warning and prompting the person in front to take action. At this time, the algorithm has not configured a tripwire area. Taking the possible movement direction of each moving target as an example, the light-colored box in the figure shows an intuitive description of the possible movement direction of the moving target. In other words, the possible posture information after the movement.

[0137] Figure 19 The schematic diagram of the moving target estimation results in the configured tripwire area provided for this application, when the moving target crosses the restricted area or tripwire, the detection probability of the algorithm is auxiliary enhanced. At this time, the target direction estimation algorithm will give a clearer estimation of the moving target direction and trigger an early warning to prompt the personnel in front to take action.

[0138] Figure 20 This is a schematic diagram of the structure of the motion direction prediction device provided by this application, which includes:

[0139] A determination module 21 is used to determine a detection frame sub-image of a target object to be tracked in a target image;

[0140] An input module 22 is configured to input the sub-image into a pre-trained motion direction prediction model for processing to obtain a motion direction probability value of the target object; wherein the motion direction prediction model is configured to determine the posture key points of the target object in the sub-image, and determine the motion direction probability value based on the sub-image and the posture key points;

[0141] The prediction module 23 is configured to predict the target motion direction of the target object according to the motion direction probability value.

[0142] The determination module 21 is specifically configured to obtain a plurality of images including the target image acquired within a preset period, and determine a detection frame sub-image of the target object in the plurality of images;

[0143] The input module 22 is specifically used to input multiple sub-images into the motion direction prediction model for processing to obtain the motion direction probability value of the target object; wherein, the motion direction prediction model is used to determine the posture key points of the target object corresponding to each of the multiple sub-images, and determine the motion direction probability value based on the multiple sub-images and the corresponding posture key points.

[0144] The device further comprises:

[0145] The training module 24 is used to obtain the sample continuous frame video in the training set, input the sample image in the sample continuous frame video, the posture key points of the sample object in the sample image, and the marked correspondence between the sample image and the posture key points of the sample object, into the initial motion direction prediction model, and train the initial motion direction prediction model; input the sample image in the sample continuous frame video, the posture key points of the sample object in the sample image, and the semantic description information corresponding to the sample continuous frame video into the trained initial motion direction prediction model, train the trained initial motion direction prediction model, and obtain the motion direction prediction model.

[0146] The training module 24 is also used to obtain a first sample continuous frame video stored in a database, and semantic description information corresponding to the first sample continuous frame video; use a human posture transformation algorithm to transform the human posture in the first sample image in the first sample continuous frame video to obtain a second sample continuous frame video, and generate semantic description information corresponding to the second sample continuous frame video; use the first sample continuous frame video and the second sample continuous frame video as sample continuous frame videos in a training set.

[0147] The input module 22 is specifically configured to determine the movement direction probability values ​​of the target object toward multiple directions, and enhance the movement direction probability value of the tripwire direction toward a preset tripwire area according to a preset probability value.

[0148] The input module 22 is specifically used to determine the first position information of the target object in the target image. If the trigger probability enhancement condition is determined based on the first position information and the second position information of the preset trip wire area, the movement direction probability value of the trip wire direction is enhanced according to the preset probability value in the trip wire direction toward the preset trip wire area.

[0149] The prediction module 23 is specifically configured to predict the target motion direction of the target object according to at least one motion direction probability value that is greater than a preset direction probability threshold.

[0150] The device further comprises:

[0151] The alarm module 25 is configured to output a high probability warning prompt message indicating a tripwire direction if the target movement direction is a tripwire direction and the movement direction probability value corresponding to the target movement direction is greater than a preset alarm threshold.

[0152] The present application also provides an electronic device, such as Figure 21As shown, it includes: a processor 31, a communication interface 32, a memory 33 and a communication bus 34, wherein the processor 31, the communication interface 32, and the memory 33 communicate with each other through the communication bus 34;

[0153] The memory 33 stores a computer program, and when the program is executed by the processor 31 , the processor 31 performs any of the above method steps.

[0154] The communication bus mentioned in the electronic device mentioned above may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. This communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in the figure, but this does not mean that there is only one bus or only one type of bus.

[0155] The communication interface 32 is used for communication between the electronic device and other devices.

[0156] The memory may include random access memory (RAM) or non-volatile memory (NVM), such as at least one disk memory. Alternatively, the memory may be at least one storage device located away from the processor.

[0157] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0158] The present application also provides a computer storage readable storage medium, which stores a computer program that can be executed by an electronic device. When the program runs on the electronic device, the electronic device implements any of the above method steps when executing.

[0159] Although the preferred embodiments of the present application have been described, those skilled in the art may make additional changes and modifications to these embodiments once they have learned the basic creative concept. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments and all changes and modifications that fall within the scope of the present application.

[0160] Obviously, those skilled in the art may make various changes and modifications to this application without departing from the spirit and scope of this application. Thus, if these modifications and variations of this application fall within the scope of the claims of this application and their equivalents, this application is intended to include these modifications and variations.

Claims

1. A motion direction prediction method, characterized in that: The method comprises: Determine a detection frame sub-image of a target object to be tracked in a target image; Inputting the sub-image into a pre-trained motion direction prediction model for processing to obtain a motion direction probability value of the target object; wherein the motion direction prediction model is used to determine the posture key points of the target object in the sub-image, and determining the motion direction probability value based on the sub-image and the posture key points; predicting a target movement direction of the target object according to the movement direction probability value; The method comprises: Acquire multiple images including the target image acquired within a preset period, and determine a detection frame sub-image of the target object in the multiple images; Inputting a plurality of sub-images into the motion direction prediction model for processing to obtain a motion direction probability value of the target object; wherein the motion direction prediction model is used to determine the posture key points of the target object corresponding to each of the plurality of sub-images, and determining the motion direction probability value based on the plurality of sub-images and the corresponding posture key points; The training process of the motion direction prediction model includes: Obtaining a sample continuous frame video in a training set, inputting sample images in the sample continuous frame video, posture key points of sample objects in the sample images, and the annotated correspondences between the sample images and the posture key points of the sample objects into an initial motion direction prediction model, and training the initial motion direction prediction model; The sample image in the sample continuous frame video, the posture key points of the sample object in the sample image, and the semantic description information corresponding to the sample continuous frame video are input into the trained initial motion direction prediction model, and the trained initial motion direction prediction model is trained to obtain the motion direction prediction model.

2. The method according to claim 1, wherein The obtaining of sample continuous frame videos in the training set includes: Obtaining a first sample continuous frame video stored in a database, and semantic description information corresponding to the first sample continuous frame video; Using a human posture transformation algorithm, transform the human posture in the first sample image in the first sample continuous frame video to obtain a second sample continuous frame video, and generate semantic description information corresponding to the second sample continuous frame video; The first sample continuous frame video and the second sample continuous frame video are used as sample continuous frame videos in a training set.

3. The method according to claim 1, wherein Determining the probability value of the moving direction of the target object includes: Determine the movement direction probability values ​​of the target object toward multiple directions, and enhance the movement direction probability value of the tripwire direction toward a preset tripwire area according to a preset probability value.

4. The method according to claim 3, wherein The step of enhancing the movement direction probability value of the tripwire direction according to a preset probability value in the tripwire direction toward the preset tripwire area includes: Determine the first position information of the target object in the target image. If the trigger probability enhancement condition is determined based on the first position information and the second position information of the preset tripwire area, the movement direction probability value of the tripwire direction is enhanced according to a preset probability value in the tripwire direction toward the preset tripwire area.

5. The method according to claim 1, wherein Predicting the target movement direction of the target object according to the movement direction probability value includes: The motion direction probability values ​​of the plurality of motion directions are compared, and a target motion direction probability value is selected according to the comparison result; and the target motion direction of the target object is predicted according to the target motion direction probability value.

6. The method according to claim 3 or 4, wherein: The method further comprises: If the target movement direction is the tripwire direction, and the movement direction probability value corresponding to the target movement direction is greater than a preset alarm threshold, a high probability warning prompt information indicating the tripwire direction is output.

7. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory for storing computer programs; A processor, configured to implement the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps according to any one of claims 1 to 6 are implemented.

Images