Motion state recognition method and device, terminal and computer readable storage medium

Abstract

The invention provides a motion state recognition method and device, a terminal and a computer readable storage medium, and the method comprises the steps: carrying out the key point detection and graph structure construction of a to-be-detected image containing a target object, and obtaining the detection key point information and an initial graph structure corresponding to the target object; performing diffusion updating on each piece of detection key point information in the initial graph structure of the target object to obtain an updated graph structure of the target object; the number of the detection key points contained in the initial graph structure does not exceed the number of the preset key points contained in the updated graph structure; and determining the motion state of the target object based on the updated graph structure of the target object. Under the condition that the target is partially shielded or the target image is blurred, the complete image structure of the target object is obtained by performing diffusion updating on the initial image structure of the target object and complementing the shielded and undetected key points, so that the recognition accuracy of the motion state of the target object is improved.

Description

Technical Field

[0001] The present invention relates to the field of image recognition technology, and in particular to a motion state recognition method, device, terminal and computer-readable storage medium. Background Art

[0002] With the advancement of technology and the strengthening of safety awareness, social governance and park management are becoming more and more intelligent, and everyone's requirements for intelligent management systems are also increasing. In order to better improve the efficiency of social governance, ensure the safety of targets, improve the efficiency of park management and user experience, it is essential to better detect and identify targets and analyze the status of targets. This helps to quickly discover unreasonable behaviors and take corresponding measures in a timely manner, thereby improving governance efficiency; for scenes such as coal and construction sites, it can more effectively manage dangerous areas and standardize operating behaviors, thereby providing real-time safety warnings; and in scenes such as parks, the status of the park can be judged based on the behavior of the target, such as whether there is congestion and whether security measures need to be adjusted. This can help administrators manage the park more effectively. For targets in the park, the intelligent detection system can provide more intelligent services, such as dredging guidance, safety prompts, etc., and can also improve user experience. Summary of the invention

[0003] The main technical problem solved by the present invention is to provide a motion state recognition method, device, terminal and computer-readable storage medium to solve the problem in the prior art of identifying the state of a target when the target is partially blocked or the target image is blurred.

[0004] In order to solve the above technical problems, the first technical solution adopted by the present invention is to provide a motion state recognition method, which comprises: Perform key point detection and graph structure construction on the image to be detected containing the target object to obtain the detection key point information and the initial graph structure corresponding to the target object; the initial graph structure includes the detection key points and the connecting edges between the detection key points; The information of each detection key point in the initial graph structure of the target object is diffusely updated to obtain an updated graph structure of the target object; the updated graph structure includes all preset key points and connecting edges between the preset key points, and the number of detection key points included in the initial graph structure does not exceed the number of preset key points included in the updated graph structure; Based on the updated graph structure of the target object, the motion state of the target object is determined.

[0005] Wherein, determining the motion state of the target object based on the updated graph structure of the target object includes: Extract features from the updated graph structure of the target object to obtain structural features of the target object; Perform feature extraction on the image containing the target object to obtain the image features of the target object; The structural features and image features corresponding to the target object are fused to obtain the fused features of the target object; Based on the fused features of the target object, the motion state of the target object is determined.

[0006] The information of each detection key point in the initial graph structure of the target object is diffused and updated to obtain an updated graph structure of the target object, including: The diffusion model is used to perform diffusion denoising and reverse denoising on the initial graph structure in sequence to generate an updated graph structure of the target object.

[0007] Among them, the training method of the diffusion model includes: Acquire a first training data set, the first training data set including a first sample image having a first training target, the first training target being associated with a labeled graph structure and a motion state label; The diffusion model is used to perform diffusion noise addition processing and reverse denoising processing on the annotation graph structure of the first sample image in sequence to generate a prediction graph structure of the first training target; the prediction graph structure includes all preset key points of the first training target and connecting edges between the preset key points; Based on the prediction graph structure of the first training target, a predicted motion state of the first training target is obtained; The diffusion model is trained based on the error value between the predicted motion state and the motion state label corresponding to the first training target.

[0008] The structural features and image features corresponding to the target object are fused to obtain the fused features of the target object, including: The feature fusion model is used to fuse the structural features and image features corresponding to the target object to obtain the fusion features of the target object; Among them, the training method of the feature fusion model includes: Acquire a second training data set, the second training data set including a second sample image having a second training target, the second training target being associated with annotated image features, annotated structural features, and a motion state label; Using a feature fusion model to fuse the annotated image features and the annotated structural features corresponding to the second training target, to obtain a predicted fusion feature of the second training target; A state recognition module is used to identify the running state of the predicted fusion features to obtain the predicted motion state of the second training target; The feature fusion model is trained based on the error value between the predicted motion state and the motion state label corresponding to the second training target.

[0009] The feature extraction of the image containing the target object to obtain the image features of the target object includes: A feature extraction model is used to extract features from the image of the target object to obtain the image features of the target object; The training methods for feature extraction models include: Acquire a third training data set, the third training data set including a third sample image having a third training target, the third training target being associated with a motion state label; Using the initial feature extraction module in the initial neural network to extract features of the third sample image, to obtain a predicted feature map of the third sample image; Using the recognition module in the initial neural network to perform recognition according to the predicted feature map of the third sample image, to obtain the predicted motion state of the third training target in the third sample image; The initial neural network is trained based on the error value between the predicted motion state corresponding to the third training target in the third sample image and the motion state label, and the trained initial feature extraction module is used as the feature extraction model.

[0010] Among them, key point detection and graph structure construction are performed on the image to be detected containing the target object to obtain the detection key point information and the initial graph structure corresponding to the target object, including: Acquire a video stream, where the video stream includes multiple consecutive video frames containing a target object; Perform key point detection on the current video frame to obtain information about the key points of the target object; the information about the key points includes the position information of the key points and the category information of the key points; Generate an initial graph structure of the target object based on the information of the detected key points using preset rules; the initial graph structure includes multiple nodes and connecting edges between the nodes; the detected key points are used as nodes; Based on the timing information of video frames in the video stream, the initial graph structures of the target objects in each video frame are associated.

[0011] The method further comprises: performing diffusion update on the information of each detection key point in the initial graph structure of the target object to obtain an updated graph structure of the target object; Based on the detection key point information of the target object in the current video frame and the video frames adjacent to the current video frame, the initial graph structure of the target object in the current video frame is updated to obtain an updated graph structure of the target object.

[0012] Among them, after the step of fusing the structural features and image features corresponding to the target object to obtain the fused features of the target object, the method further includes: According to the time sequence information of the video frames, the fusion features corresponding to the target object in each video frame are sorted to obtain the fusion feature sequence of the target object; Based on the fusion features of the target object, the motion state of the target object is determined, including: Based on the fused feature sequence of the target object, the motion state of the target object is determined.

[0013] Among them, key point detection and graph structure construction are performed on the image to be detected containing the target object to obtain the detection key point information and the initial graph structure corresponding to the target object, including: Perform key point detection on an image to be detected containing a target object to obtain information on the key points of the target object; the information on the key points includes position information of the key points and category information of the key points; the category information of the key points includes a nose, a left eye, a right eye, a left ear, a right ear, a left shoulder, a right shoulder, a left elbow, a right elbow, a left wrist, a right wrist, a left hip, a right hip, a left knee, a right knee, a left ankle, and a right ankle; According to the spatial relationship between the category information of different detection key points, connecting edges are established between the detection key points to generate the initial graph structure of the target object.

[0014] In order to solve the above technical problems, the second technical solution adopted by the present invention is: to provide a motion state recognition device, the motion state recognition device comprising: A processing module is used to perform key point detection and graph structure construction on the image to be detected containing the target object, and obtain the detection key point information and the initial graph structure corresponding to the target object; the initial graph structure includes the detection key points and the connecting edges between the detection key points; A completion module is used to diffuse and update the information of each detection key point in the initial graph structure of the target object to obtain an updated graph structure of the target object; the updated graph structure includes all preset key points and connecting edges between the preset key points, and the number of detection key points included in the initial graph structure does not exceed the number of preset key points included in the updated graph structure; The recognition module is used to determine the motion state of the target object based on the updated graph structure of the target object.

[0015] To solve the above technical problems, the third technical solution adopted by the present invention is: to provide a terminal, the terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, the processor is used to execute program data to implement the steps in the motion state recognition method as mentioned above.

[0016] To solve the above technical problems, the fourth technical solution adopted by the present invention is: providing a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps in the motion state recognition method as described above are implemented.

[0017] The beneficial effects of the present invention are as follows: different from the prior art, a motion state recognition method, device, terminal and computer-readable storage medium are provided, and the motion state recognition method includes: performing key point detection and graph structure construction on an image to be detected containing a target object, obtaining detection key point information and an initial graph structure corresponding to the target object; the initial graph structure includes detection key points and connecting edges between detection key points; performing diffusion update on each detection key point information in the initial graph structure of the target object, obtaining an updated graph structure of the target object; the updated graph structure includes all preset key points and connecting edges between preset key points, and the number of detection key points contained in the initial graph structure does not exceed the number of preset key points contained in the updated graph structure; based on the updated graph structure of the target object, determining the motion state of the target object. In the case where the target is partially occluded or the target image is blurred, the present application completes the key points that are occluded and not detected by performing diffusion update on the initial graph structure of the target object, obtains a complete graph structure of the target object, and improves the recognition accuracy of the motion state of the target object. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without creative work.

[0019] Figure 1 It is a flow chart of the motion state recognition method provided by the present invention; Figure 2 It is a flowchart of a specific embodiment of a training method for a diffusion model provided by the present invention; Figure 3 It is a flowchart of a specific embodiment of a training method for a feature fusion model provided by the present invention; Figure 4 It is a flow chart of a specific embodiment of the motion state recognition method provided by the present invention; Figure 5 yes Figure 1 A schematic flow chart of a specific embodiment of step S3 in the motion state recognition method provided; Figure 6 yes Figure 5 A flowchart of a specific embodiment of step S31 in the motion state recognition method provided; Figure 7 It is a schematic diagram of the framework of an embodiment of a motion state recognition device provided by the present invention; Figure 8 It is a schematic diagram of the framework of a terminal embodiment provided by the present invention; Fig. 9 It is a schematic diagram of a framework of an embodiment of a computer-readable storage medium provided by the present invention. DETAILED DESCRIPTION

[0020] The scheme of the embodiment of the present application is described in detail below in conjunction with the drawings of the specification.

[0021] In the following description, for the purpose of explanation rather than limitation, specific details such as specific system structures, interfaces, and technologies are provided to facilitate a thorough understanding of the present application.

[0022] The term "and / or" in this article is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article generally indicates that the associated objects before and after are in an "or" relationship. In addition, "many" in this article means two or more than two.

[0023] In order to enable those skilled in the art to better understand the technical solution of the present invention, a motion state recognition method provided by the present invention is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0024] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0025] The diffusion model defines a Markov chain of diffusion steps, gradually adding random noise to the data, and then learning the inverse diffusion process to construct the desired data samples from the noise. The diffusion model is learned with a fixed procedure, and the latent variables have high dimensionality, the same as the original data.

[0026] Before further describing the embodiments of the present application in detail, the nouns and terms involved in the embodiments of the present application are explained. The nouns and terms involved in the embodiments of the present application are subject to the following interpretations.

[0027] The motion state recognition method provided in the embodiment of the present application can be implemented by the server or the terminal alone, or by the server and the terminal in collaboration. In some embodiments, the terminal or the server can implement the motion state recognition method provided in the embodiment of the present application by running a computer program. For example, the computer program can be a native program or software module in the operating system; it can be a native application (APP, Application), that is, a program that needs to be installed in the operating system to run, such as a client that supports virtual scenes, such as a game APP; it can also be a small program, that is, a program that only needs to be downloaded to a browser environment to run; it can also be a small program that can be embedded in any APP. In short, the above-mentioned computer program can be an application, module or plug-in in any form.

[0028] The following uses server implementation as an example to illustrate the motion state recognition method provided in the embodiment of the present application.

[0029] See also Figure 1 , Figure 1 It is a flow chart of the motion state recognition method provided by the present invention.

[0030] This embodiment provides a motion state recognition method, which includes the following steps.

[0031] S1: Perform key point detection and graph structure construction on the image to be detected containing the target object to obtain the detection key point information and the initial graph structure corresponding to the target object; the initial graph structure includes the detection key points and the connecting edges between the detection key points.

[0032] S2: Diffuse and update the information of each detection key point in the initial graph structure of the target object to obtain an updated graph structure of the target object; the updated graph structure includes all preset key points and connecting edges between the preset key points, and the number of detection key points included in the initial graph structure does not exceed the number of preset key points included in the updated graph structure.

[0033] S3: Determine the motion state of the target object based on the updated graph structure of the target object.

[0034] The motion state recognition method provided in this embodiment, when the target is partially occluded or the target image is blurred, completes the occluded and undetected key points by diffusion updating the initial graph structure of the target object, obtains the complete graph structure of the target object, and improves the recognition accuracy of the motion state of the target object.

[0035] See also Figure 2 , Figure 2 It is a flowchart of a specific embodiment of the training method of the diffusion model provided by the present invention.

[0036] In one embodiment, the training method of the diffusion model specifically includes the following steps.

[0037] S101: Acquire a first training data set, where the first training data set includes a first sample image having a first training target, where the first training target is associated with a labeled graph structure and a motion state label.

[0038] Specifically, a first sample image is obtained, and the first sample image specifically includes a first training target. The first training target in the first sample image may be partially blocked. The first training target may be a target that can move, such as an animal. The first sample image may include at least one first training target. The key point detection model is used to perform key point detection on the first training target to obtain the detection key point information of the first training target. The detection key point information includes the position information of the detection key point and the category of the detection key point. Among them, the position information of the detection key point is the two-dimensional coordinate of the detection key point in the first sample image. The category of the detection key point may be at least part of the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The key point detection model can detect the key points of the unblocked part of the first training target, and the key point detection model cannot detect the key points of the blocked part of the first training target.

[0039] Based on the preset rules and the categories of the detection key points corresponding to the first training target, connecting edges are established between the detection key points to form a labeled graph structure of the first training target. In one embodiment, the preset rules may be to establish a connecting edge between the nose and the left eye, to establish a connecting edge between the nose and the right eye, to establish a connecting edge between the nose and the left shoulder, to establish a connecting edge between the nose and the right shoulder, to establish a connecting edge between the left elbow and the left shoulder, to establish a connecting edge between the right elbow and the right shoulder, to establish a connecting edge between the left elbow and the left wrist, to establish a connecting edge between the right elbow and the right wrist, to establish a connecting edge between the left hip and the left shoulder, to establish a connecting edge between the right hip and the right shoulder, to establish a connecting edge between the left hip and the left knee, to establish a connecting edge between the right hip and the right knee, to establish a connecting edge between the left ankle and the left knee, and to establish a connecting edge between the right ankle and the right knee.

[0040] For example, according to the above preset rules, a connection edge is established between the detection key points of the right shoulder and the left shoulder; a connection edge is established between the detection key points of the right shoulder and the right elbow; a connection edge is established between the detection key points of the right elbow and the right wrist. The labeling graph structure of the first training target can be obtained through the above preset rules. The detection key points in the labeling graph structure of the first training target are nodes, and the connection edges between the detection key points represent the spatial relationship between the detection key points. The labeling graph structure of the first training target G=(V, E), where V represents a node set and E represents a connection edge set.

[0041] The motion state label of the first training target may be walking, standing, running, jumping, etc. In a specific embodiment, the preset motion states include walking, standing, running, squatting, jumping, etc. For example, if the motion state label of the first training target is walking, the label probability of walking is 100%, the label probability of standing is 0, the label probability of running is 0, the label probability of squatting is 0, and the label probability of jumping is 0.

[0042] S102: using a diffusion model to perform diffusion denoising and reverse denoising on the annotated graph structure of the first sample image in sequence to generate a predicted graph structure of the first training target; the predicted graph structure includes all preset key points of the first training target and connecting edges between the preset key points.

[0043] Specifically, a diffusion model is used to perform diffusion noise processing on the annotated graph structure of the first training target in the first sample image to generate a diffusion image of the first training target. The diffusion image of the first training target is subjected to reverse denoising processing to generate the material concentration of each preset key point. The material concentration of the preset key point represents the probability that the preset key point is in each preset motion state. The preset graph structure of the first training target is generated through all the preset key points. The categories of nodes contained in the preset graph structure include nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The connecting edges between the nodes in the preset graph structure are constructed according to preset rules.

[0044] The diffusion model in this embodiment is a generative model that is used to generate new data samples in the field of machine learning and deep learning. The diffusion model first gradually transforms the data into Gaussian noise, a process called the forward diffusion process. In each step, random noise is added to the data until the data is completely transformed into noise. The goal of the model is then to learn how to recover the original data from the noise, a process called the reverse diffusion process. A neural network is trained to estimate how much noise should be removed at each step, thereby gradually recovering the original data. The diffusion model is trained by minimizing the gap between the predicted noise and the actual added noise. This gap is usually measured using the mean square error (MSE) or other loss functions.

[0045] S103: Using a state recognition module to perform running state recognition on the prediction graph structure of the first training target to obtain a predicted motion state of the first training target.

[0046] Specifically, the predicted motion state of the first training target is predicted based on the substance concentration of each node in the preset graph structure of the first training target by the state recognition model, wherein the predicted motion state may be the probability of each preset motion state.

[0047] S104: Training the diffusion model based on the error value between the predicted motion state and the motion state label corresponding to the first training target.

[0048] Specifically, an error value is calculated by calculating the predicted motion state and the running state label corresponding to the first training target, and the parameters of the diffusion model are optimized according to the error value to realize the training of the diffusion model. When the predicted error value of the diffusion model is less than the preset error, the optimization of the diffusion model can be stopped. Or when the number of iterations reaches the preset number, the optimization of the diffusion model can be stopped.

[0049] In one embodiment, the diffusion model can also be trained based on the training video stream. Specifically, a training video stream containing a first training target is input into the diffusion model, the first training target has a labeled structure diagram in each training video frame, and the first training target has a motion state label. Among them, the first training target has a corresponding labeled structure diagram sequence in the training video frame. The diffusion model performs diffusion noise addition processing and reverse denoising processing on the structure diagrams in the labeled structure diagram sequence of the first training target, respectively, to obtain the predicted graph structure of each structure diagram. According to the timing information of the labeled structure diagram sequence, each predicted graph structure is sorted to generate a predicted graph structure sequence of the first training target. A state recognition module is used to identify the running state of the predicted graph structure sequence of the first training target to obtain the predicted motion state of the first training target. The diffusion model is trained based on the error value between the predicted motion state corresponding to the first training target and the motion state label.

[0050] Specifically, when the training sample for training the diffusion model is a training video stream, the diffusion model determines the predicted graph structure of the first training target in the current video frame based on the initial graph structure of the first training target in the current video frame and the initial graph structure of the first training target in the video frame adjacent to the current video stream to generate the predicted graph structure of the first training target in the current video frame.

[0051] See also Figure 3 , Figure 3 It is a flowchart of a specific embodiment of the training method of the feature fusion model provided by the present invention.

[0052] In one embodiment, the training method of the feature fusion model specifically includes the following steps.

[0053] S201: Acquire a second training data set, where the second training data set includes a second sample image having a second training target, where the second training target is associated with annotated image features, annotated structural features, and a motion state label.

[0054] Specifically, a second sample image is obtained, and the second sample image specifically includes a second training target. The second training target in the second sample image may be partially blocked. The second training target may be a movable target such as an animal. The second sample image may include at least one second training target. The second training target has annotated image features and annotated structural features. Among them, the annotated image features of the second training target are obtained by extracting features from an image containing the second training target using a target feature extraction model. The annotated structural features of the second training target are obtained by performing diffusion update and feature extraction on the initial graph structure of the second training target using the diffusion model trained in the above steps. The initial graph structure of the second training target is generated based on the detection key points of the second training target according to preset rules. Among them, the target feature extraction model may be a convolutional neural network (CNN for short). In this embodiment, the second sample image may be the same as or different from the first sample image; the second training target may be the same as or different from the first training target.

[0055] The motion state label of the second training target may be walking, standing, running, jumping, etc. In a specific embodiment, the preset motion states include walking, standing, running, squatting, jumping, etc. For example, if the motion state label of the second training target is walking, the label probability of walking is 100%, the label probability of standing is 0, the label probability of running is 0, the label probability of squatting is 0, and the label probability of jumping is 0.

[0056] S202: Using a feature fusion model to perform feature fusion on the annotated image features and the annotated structural features corresponding to the second training target, to obtain predicted fusion features of the second training target.

[0057] Specifically, the annotated image features and the annotated structural features corresponding to the second training target are simultaneously input into the feature fusion model, and the feature fusion model fuses the annotated image features and the annotated structural features to obtain the predicted fusion features of the second training target. The feature fusion model may be a fully connected layer.

[0058] In this embodiment, the feature fusion model has at least two input terminals to facilitate the fusion of the structural features and image features of the second training target. The structural features and image features of the second training target are fused through the feature fusion model to obtain richer semantic information, thereby facilitating the improvement of the accuracy of the running state recognition of the second training target.

[0059] S203: Using a state recognition module to identify the running state of the predicted fusion feature to obtain the predicted motion state of the second training target.

[0060] Specifically, the state recognition model is used to perform state recognition on the predicted fusion features of the second training target to generate the predicted motion state of the second training target, wherein the predicted motion state may be the probability of each preset motion state.

[0061] S204: Training the feature fusion model based on the error value between the predicted motion state and the motion state label corresponding to the second training target.

[0062] Specifically, the loss value between the predicted motion state and the motion state label of the second training target is calculated by the following formula: .

[0063] (Formula 1) Where n is the number of preset motion states; Represents the predicted motion state, Indicates the motion status label; Represents the weight of the i-th feature, which is used to adjust the impact of this feature on the loss function.

[0064] The parameters in the feature fusion model are optimized based on the loss value between the predicted motion state and the motion state label of the second training objective to achieve training of the feature fusion model.

[0065] When the loss value of the feature fusion model is less than the preset value, the optimization of the feature fusion model can be stopped. Or when the number of iterations reaches the preset number, the optimization of the feature fusion model can be stopped. Alternatively, the preset validation data set can be input into the feature fusion model, and the training effect of the feature fusion model can be evaluated according to the preset validation data set, and then the structure and hyperparameters of the feature fusion model can be adjusted to ensure that the feature fusion model can effectively fuse different types of features and improve the accuracy of target state classification.

[0066] In the above embodiment, the feature fusion model can also be trained based on the training video stream. Specifically, the annotated image feature sequence and the annotated structural feature sequence corresponding to the training video stream containing the second training target are input into the feature fusion model. Among them, the annotated image feature sequence is an image feature sequence composed of the image features corresponding to the second training target in each video frame according to the time sequence; the annotated structural feature sequence is a structural feature sequence composed of the structural features corresponding to the second training target in each video frame according to the time sequence, and the second training target has a motion state label. The feature fusion model performs feature fusion on the structural features and image features of the second training target in the same video frame to generate the predicted fusion features in the video frame. The predicted fusion features corresponding to each video frame can be obtained through the above steps. The predicted fusion feature sequence of the second training target is generated according to the time sequence of the video frame. The state recognition module is used to identify the running state of the predicted fusion feature sequence of the second training target to obtain the predicted motion state of the second training target. The feature fusion model is trained based on the error value between the predicted motion state corresponding to the second training target and the motion state label.

[0067] The above steps can realize the training of diffusion model and feature fusion model.

[0068] In one embodiment, the training method of the feature extraction model specifically includes the following steps.

[0069] Obtain a third training data set, the third training data set includes a third sample image with a third training target, and the third training target is associated with a motion state label; use the initial feature extraction module in the initial neural network to extract features from the third sample image to obtain a predicted feature map of the third sample image; use the recognition module in the initial neural network to perform recognition based on the predicted feature map of the third sample image to obtain a predicted motion state of the third training target in the third sample image; train the initial neural network based on the error value between the predicted motion state corresponding to the third training target in the third sample image and the motion state label, and use the trained initial feature extraction module as a feature extraction model. The feature extraction model in this embodiment can be the CNN model in the above embodiment.

[0070] See also Figure 4 , Figure 4 It is a flow chart of a specific embodiment of the motion state recognition method provided by the present invention.

[0071] Specifically, in step S1, key point detection and graph structure construction are performed on the image to be detected containing the target object to obtain the detection key point information corresponding to the target object and the specific implementation method of the initial graph structure is as follows.

[0072] In one embodiment, an image to be detected of a target area is acquired in real time by an image acquisition device, and the image to be detected includes a target object. The target object is a movable target. The target area may be an intersection, a street, a park, etc. Target detection is performed on the image to be detected to obtain a detection frame of each target object. The detection frame only includes one target object. Key point detection is performed on the detection frame including the target object to obtain detection key point information of the target object.

[0073] In one embodiment, key point detection is performed on an image to be detected containing a target object to obtain information on detection key points of the target object; the information on the detection key points includes position information of the detection key points and category information of the detection key points, and the category information of the detection key points includes parts of a nose, a left eye, a right eye, a left ear, a right ear, a left shoulder, a right shoulder, a left elbow, a right elbow, a left wrist, a right wrist, a left hip, a right hip, a left knee, a right knee, a left ankle, and a right ankle; according to the spatial relationship between the category information of different detection key points, connecting edges are established between the detection key points to generate an initial graph structure of the target object. For example, a connecting edge is established between the nose and the left eye, between the nose and the right eye, between the right ear and the right eye, between the left ear and the left eye, between the left shoulder and the nose, between the right shoulder and the nose, between the left shoulder and the left elbow, between the left wrist and the left elbow, between the right shoulder and the right elbow, between the right wrist and the right elbow, between the left shoulder and the left hip, between the right shoulder and the right hip, between the left knee and the left hip, between the right knee and the right hip, between the left knee and the left ankle, and between the right knee and the right ankle.

[0074] In one embodiment, an offline video stream is obtained, and the video stream includes multiple frames of video frames that continuously contain a target object. All video frames in the video stream are traversed, and each video frame is sequentially used as a current video frame. Target detection is performed on the current video frame to obtain a detection frame containing the target object, and the detection frame only contains one target object. Key point detection is performed on the detection frame containing the target object to obtain information on the detection key points of the target object. The information on the detection key points of the target object in each video frame is obtained through the above steps.

[0075] The key point information detected includes the key points and the position information and category information corresponding to the key points. The category information of the key points may include at least part of the nose, left eye, right eye, left ear, right ear, left shoulder, right shoulder, left elbow, right elbow, left wrist, right wrist, left hip, right hip, left knee, right knee, left ankle, and right ankle. The initial graph structure of the target object is generated based on the information of the key points detected using preset rules; the initial graph structure includes multiple nodes and connecting edges between the nodes, and the key points detected are used as nodes.

[0076] When the acquired information is a video stream, the initial graph structures of the target object in each video frame are associated based on the timing information of the video frames in the video stream to generate an initial graph structure sequence of the target object.

[0077] In one embodiment, an initial graph structure is constructed based on the position information of all detected key points of the target object in the video frame. For example, the initial graph structure can be represented as G t =(V t , E t ), where V t is the set of key points of the target object in the video frame collected at time t; E t It is the edge set between the key points of the target object in the video frame collected at time t. Specifically, the connecting edges between the key points are established according to the spatial relationship between the key points, such as the distance and angle between the key points.

[0078] The graph structure sequence {G1, G2, ..., G t}, G t =(V t , E t ).

[0079] Specifically, in step S2, the initial graph structure is updated based on the detection key point information of the target object to obtain the updated graph structure of the target object as shown below.

[0080] In one embodiment, the initial graph structure corresponding to the target object is input into the diffusion model trained in the above steps S101 to S104, and the diffusion model performs diffusion noise addition and reverse denoising on the initial graph structure in sequence to generate an updated graph structure of the target object.

[0081] Specifically, the following formula can be used to perform noise diffusion on each detection key point in the initial graph structure to generate a diffuse image of the target object.

[0082] (Formula 2) Where: C t Indicates the concentration of the substance at the critical detection point at time t, C t-1 represents the substance concentration of the detection key point at time t-1, N(i) represents the set of detection key points adjacent to the detection key point i, and f() is a function used to update the substance concentration of the current detection key point according to the substance concentration of the neighboring detection key points.

[0083] Specifically, the diffusion image corresponding to the initial graph structure can be subjected to reverse denoising processing through the following formula to generate an updated graph structure of the target object.

[0084] (Formula 3) Where: C t It represents the concentration of the substance at the key detection point at time t; P is a probability distribution used to determine the probability of the substance propagating from other nodes to the central node.

[0085] In one embodiment, a forward diffusion process is simulated on the initial graph structure according to the detection key point information of the target object, and the concentration of the substance is diffused from the central node to the external nodes.

[0086] Specifically, the central node v in the initial graph structure c The initial substance concentration is The central node in the initial graph structure may be the left hip, the right hip, the left shoulder or the right shoulder. The substance concentration of each detection key point is initialized based on the category information of the detection key point.

[0087] In one embodiment, if the category information of the detection key point is an external node, the substance concentration of the detection key point corresponding to the external node in the initial graph structure is used as the initial concentration. The external node may be a left ankle, a right ankle, a left wrist, a right wrist, a left ear, or a right ear.

[0088] In one embodiment, if the category information of the detected key point does not belong to an external node, the initial concentration of the detected key point is set to zero.

[0089] Initial substance concentration of each central node in the specific initial graph structure As shown below.

[0090] (Formula 4) Where C0 is the substance concentration of the detection key point in the initial graph structure.

[0091] The diffusion model simulates the diffusion of substances from the central node to the surrounding nodes. The diffusion process updates the substance concentration of each node according to the following formula.

[0092] (Formula 5) Where: Represents the substance concentration of the central node of the target object at the t+1th time; represents the material concentration of the central node of the target object at the tth time; N(v) represents the set of key points adjacent to the central node v; It represents the material concentration of each key point of the target object adjacent to the central node at the tth moment; f() is an update function, which is used to update the material concentration of the central node according to the material concentration of the adjacent nodes. Specifically, weighted average or numerical solution of the diffusion equation can be used.

[0093] Through multiple iterations, the central node gradually diffuses to all nodes of the initial graph structure until a balanced state is reached. The output of this step is the material concentration distribution of each key detection point at the tth moment { }, which reflects the diffusion characteristics of the target object’s posture in space.

[0094] In one embodiment, the reverse diffusion process simulation is continued on the initial graph structure according to the detection key point information of the target object, and the concentration of the substance is propagated from the external nodes back to the central node.

[0095] In one embodiment, each external node v in the initial graph structure c The initial substance concentration is In one embodiment, if the category information of the detection key point is an external node, the material concentration of the detection key point corresponding to the external node in the initial graph structure is used as the initial concentration. The external node may be a left ankle, a right ankle, a left wrist, a right wrist, a left ear, or a right ear.

[0096] In one embodiment, if the category information of the detected key point does not belong to an external node, the initial concentration of the detected key point is set to zero.

[0097] Initial substance concentration of each external node in the specific initial graph structure As shown below.

[0098] (Formula 6) In the formula, C T is the substance concentration at the starting point of back diffusion.

[0099] The diffusion model simulates the reverse diffusion of substances from external nodes to central nodes. The reverse diffusion process updates the substance concentration of each node according to the following formula.

[0100] (Formula 7) Where: Represents the substance concentration of the external node of the target object at the t-1th time; represents the material concentration of the external node of the target object at the tth time; N(v) represents the set of key points adjacent to the external node v; It represents the material concentration of each key point of the target object adjacent to the external node at the tth time; g() is the probability distribution function, which is used to determine the probability of the material concentration propagating from other nodes to the central node.

[0101] Through multiple iterations, the external nodes are gradually gathered back to the central node to form the final material concentration distribution. The output of this step is the material concentration distribution of each node after the reverse diffusion process at time t-1 { }, which reflects the diffusion characteristics of the target object’s posture in space.

[0102] After completing the forward diffusion process simulation and the reverse diffusion process simulation, the material concentration of each node is extracted as the structural feature of the target object posture.

[0103] In a specific embodiment, the concentration of the substance at each key detection point of the target object in the video frame is extracted. } and convert it into a numerical vector F t . . Among them, v1, v2, ..., v n is the detection key point in the initial graph structure.

[0104] The structural feature vector F of the target object in the video frame t As sequence features, it is convenient for the subsequent sequence analysis model to analyze. Through this step, the feature vector sequence [F1, F2, ..., F t ] to represent the posture structural characteristics of the target object at different time points.

[0105] In a specific embodiment, the initial graph structure of the target object in the current video frame is updated based on the detection key point information of the target object in the current video frame and video frames adjacent to the current video frame to obtain an updated graph structure of the target object.

[0106] Specifically, the substance concentration of each preset key point in the updated graph structure of the target object in the current video frame can be determined based on the following formula.

[0107] y t =f(x t ,y t-1 , W) (Formula 8) Where: y t is the concentration of the substance at the preset key point; x t Indicates the initial material concentration of the preset key point in the current video frame; y t-1 represents the initial substance concentration of the preset key point in the previous video frame adjacent to the current video frame; W is the model parameter of the diffusion model.

[0108] Specifically, the specific implementation method of determining the motion state of the target object based on the updated graph structure of the target object in step S3 is as follows.

[0109] See also Figure 5 , Figure 5 yes Figure 1 A flowchart of a specific embodiment of step S3 in the motion state recognition method is provided.

[0110] S31: Fusing the updated graph structure of the target object with the image to be detected containing the target object to obtain a fused feature of the target object.

[0111] See also Figure 6 , Figure 6 yes Figure 5 A flowchart of a specific embodiment of step S31 in the motion state recognition method is provided.

[0112] S311: Extract features from the updated graph structure of the target object to obtain structural features of the target object.

[0113] In one embodiment, each preset key point in the update graph structure of the target object has a corresponding material concentration. The material concentration of the preset key point reflects the probability that the target object is in each preset motion state. The structural features of the target object are generated by extracting the material concentrations of all preset key points in the update graph structure of the target object.

[0114] S312: Extract features from the image containing the target object to obtain image features of the target object.

[0115] In one embodiment, feature extraction is performed on a detection frame including a target object to obtain image features of the target object.

[0116] In a specific embodiment, a CNN model is used to extract features from a detection frame containing a target object to obtain image features of the target object. The CNN model includes multiple convolutional layers, pooling layers, and fully connected layers. The convolutional layer is used to extract local features in the image, such as edges, textures, and shapes. The pooling layer is used to reduce the size of the feature map, reduce the number of parameters, and retain important information. The fully connected layer is used to convert the feature map into final feature information.

[0117] S313: Fusing the structural features and image features corresponding to the target object to obtain fused features of the target object.

[0118] In one embodiment, the structural features and image features corresponding to the target object are input into the feature fusion model trained in the above steps S201 to S204 to obtain the fusion features of the target object.

[0119] S32: Determine the motion state of the target object based on the fusion features of the target object.

[0120] Specifically, a feature fusion model is used to fuse the structural features and image features corresponding to the target object to obtain the fusion features of the target object. A pre-trained motion state recognition model is used to determine the motion state of the target object based on the fusion features of the target object.

[0121] In a specific embodiment, according to the time sequence information of the video frames, the fusion features corresponding to the target object in each video frame are sorted to obtain the fusion feature sequence of the target object. The motion state of the target object is determined based on the fusion feature sequence of the target object using a pre-trained motion state recognition model.

[0122] The motion state recognition method provided in this embodiment, when the target is partially occluded or the target image is blurred, completes the occluded and undetected key points by diffusion updating the initial graph structure of the target object, obtains the complete graph structure of the target object, and improves the recognition accuracy of the motion state of the target object.

[0123] See also Figure 7 , Figure 7 Schematic diagram of a framework of an embodiment of a motion state recognition device provided by the present invention. This embodiment provides a motion state recognition device 60 , which includes a processing module 61 , a completion module 62 and a recognition module 63 .

[0124] The processing module 61 is used to perform key point detection and graph structure construction on the image to be detected containing the target object, and obtain the detection key point information and the initial graph structure corresponding to the target object; the initial graph structure includes the detection key points and the connecting edges between the detection key points.

[0125] The completion module 62 is used to diffuse and update the information of each detection key point in the initial graph structure of the target object to obtain an updated graph structure of the target object; the updated graph structure includes all preset key points and connecting edges between the preset key points, and the number of detection key points contained in the initial graph structure does not exceed the number of preset key points contained in the updated graph structure.

[0126] The identification module 63 is used to determine the motion state of the target object based on the updated graph structure of the target object.

[0127] When the target is partially occluded or the target image is blurred, the present application completes the occluded and undetected key points by diffusion updating the initial graph structure of the target object, obtains the complete graph structure of the target object, and improves the recognition accuracy of the target object's motion state.

[0128] See also Figure 8 , Figure 880 is a schematic diagram of a terminal embodiment provided by the present invention. The terminal 80 includes a memory 81 and a processor 82 coupled to each other, and the processor 82 is used to execute program instructions stored in the memory 81 to implement the steps of any of the above-mentioned motion state recognition method embodiments. In a specific implementation scenario, the terminal 80 may include but is not limited to: a microcomputer, a server, and in addition, the terminal 80 may also include a mobile device such as a laptop computer and a tablet computer, which is not limited here.

[0129] Specifically, the processor 82 is used to control itself and the memory 81 to implement the steps of any of the above-mentioned motion state recognition method embodiments. The processor 82 can also be called a CPU (Central Processing Unit). The processor 82 may be an integrated circuit chip with signal processing capabilities. The processor 82 can also be a general-purpose processor, a digital signal processor (Digital Signal Processor, DSP), an application-specific integrated circuit (Application Specific Integrated Circuit, ASIC), a field-programmable gate array (Field-Programmable Gate Array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. In addition, the processor 82 can be implemented by an integrated circuit chip.

[0130] See also Fig. 9 , Fig. 9 The computer-readable storage medium 90 stores program instructions 901 that can be executed by a processor, and the program instructions 901 are used to implement the steps of any of the above-mentioned motion state recognition method embodiments.

[0131] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0132] The above description of various embodiments tends to emphasize the differences between the various embodiments. The same or similar aspects can be referenced to each other, and for the sake of brevity, they will not be repeated herein.

[0133] In the several embodiments provided in the present application, it should be understood that the disclosed methods and devices can be implemented in other ways. For example, the device implementation described above is only schematic. For example, the division of modules or units is only a logical function division. There may be other division methods in actual implementation, such as units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, and the indirect coupling or communication connection of devices or units can be electrical, mechanical or other forms.

[0134] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0135] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) or a processor to execute all or part of the steps of each implementation method of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk and other media that can store program code.

[0136] The above are only implementation modes of the present invention, and are not intended to limit the patent protection scope of the present invention. Any equivalent structure or equivalent process transformation made using the contents of the present invention specification and drawings, or directly or indirectly applied in other related technical fields, are also included in the patent protection scope of the present invention.

Claims

1. A motion state recognition method, characterized in that: The motion state recognition method comprises: Performing key point detection and graph structure construction on an image to be detected containing a target object, and obtaining detection key point information and an initial graph structure corresponding to the target object; the initial graph structure includes detection key points and connecting edges between the detection key points; Perform diffusion update on the information of each of the detection key points in the initial graph structure of the target object to obtain an updated graph structure of the target object; the updated graph structure includes all preset key points and connecting edges between the preset key points, and the number of the detection key points included in the initial graph structure does not exceed the number of the preset key points included in the updated graph structure; Based on the updated graph structure of the target object, a motion state of the target object is determined.

2. The motion state recognition method according to claim 1, characterized in that: The determining the motion state of the target object based on the updated graph structure of the target object includes: Performing feature extraction on the updated graph structure of the target object to obtain structural features of the target object; Extracting features from an image containing the target object to obtain image features of the target object; Fusing the structural features and the image features corresponding to the target object to obtain fused features of the target object; Based on the fusion features of the target object, a motion state of the target object is determined.

3. The motion state recognition method according to claim 1, characterized in that: The step of performing diffusion updating on the information of each detection key point in the initial graph structure of the target object to obtain an updated graph structure of the target object includes: The diffusion model is used to perform diffusion noise addition processing and reverse denoising processing on the initial graph structure in sequence to generate an updated graph structure of the target object.

4. The motion state recognition method according to claim 3, characterized in that: The training method of the diffusion model includes: Acquire a first training data set, the first training data set comprising a first sample image having a first training target, the first training target being associated with a labeled graph structure and a motion state label; The diffusion model is used to sequentially perform diffusion noise addition processing and reverse denoising processing on the annotation graph structure of the first sample image to generate a prediction graph structure of the first training target; the prediction graph structure includes all preset key points of the first training target and connecting edges between the preset key points; Based on the prediction graph structure of the first training target, obtaining a predicted motion state of the first training target; The diffusion model is trained based on an error value between the predicted motion state and the motion state label corresponding to the first training target.

5. The motion state recognition method according to claim 2, characterized in that: The step of fusing the structural features and the image features corresponding to the target object to obtain the fused features of the target object includes: Using a feature fusion model to fuse the structural features and the image features corresponding to the target object to obtain a fusion feature of the target object; The training method of the feature fusion model includes: Acquire a second training data set, wherein the second training data set includes a second sample image having a second training target, wherein the second training target is associated with annotated image features, annotated structural features, and a motion state label; Using the feature fusion model to perform feature fusion on the annotated image features and the annotated structural features corresponding to the second training target, to obtain predicted fusion features of the second training target; Using a state recognition module to identify the running state of the predicted fusion feature to obtain the predicted motion state of the second training target; The feature fusion model is trained based on the error value between the predicted motion state and the motion state label corresponding to the second training target.

6. The motion state recognition method according to claim 2, characterized in that: The step of extracting features from an image containing the target object to obtain image features of the target object includes: Using a feature extraction model to extract features from the image of the target object to obtain image features of the target object; The training method of the feature extraction model includes: Acquire a third training data set, wherein the third training data set includes a third sample image having a third training target, and the third training target is associated with a motion state label; Using an initial feature extraction module in the initial neural network to extract features of the third sample image, to obtain a predicted feature map of the third sample image; Using the recognition module in the initial neural network to perform recognition according to the predicted feature map of the third sample image, to obtain the predicted motion state of the third training target in the third sample image; The initial neural network is trained based on the error value between the predicted motion state corresponding to the third training target in the third sample image and the motion state label, and the trained initial feature extraction module is used as the feature extraction model.

7. The motion state recognition method according to any one of claims 1 to 6, characterized in that: The step of performing key point detection and graph structure construction on the image to be detected containing the target object to obtain detection key point information and an initial graph structure corresponding to the target object includes: Acquire a video stream, wherein the video stream includes a plurality of video frames that continuously contain the target object; Perform key point detection on the current video frame to obtain information about the detected key points of the target object; the information about the detected key points includes position information of the detected key points and category information of the key points; Generate an initial graph structure of the target object based on the information of the detection key points using a preset rule; the initial graph structure includes a plurality of nodes and connecting edges between the nodes; the detection key points serve as the nodes; Based on the timing information of the video frames in the video stream, the initial graph structure of the target object in each of the video frames is associated.

8. The motion state recognition method according to claim 7, characterized in that: The step of performing diffusion update on the information of each of the detection key points in the initial graph structure of the target object to obtain an updated graph structure of the target object further includes: An initial graph structure of the target object in the current video frame is updated based on the detection key point information of the target object in the current video frame and the video frames adjacent to the current video frame to obtain an updated graph structure of the target object.

9. The motion state recognition method according to claim 8, characterized in that: After the step of fusing the structural features and the image features corresponding to the target object to obtain the fused features of the target object, the method further includes: According to the time sequence information of the video frames, the fusion features corresponding to the target object in each of the video frames are sorted to obtain a fusion feature sequence of the target object; The determining the motion state of the target object based on the fusion feature of the target object includes: Based on the fused feature sequence of the target object, a motion state of the target object is determined.

10. The motion state recognition method according to any one of claims 1 to 6, characterized in that: The step of performing key point detection and graph structure construction on the image to be detected containing the target object to obtain detection key point information and an initial graph structure corresponding to the target object includes: Perform key point detection on the image to be detected containing the target object to obtain information on the key points of the target object; the information on the key points includes position information of the key points and category information of the key points, and the category information of the key points includes a nose, a left eye, a right eye, a left ear, a right ear, a left shoulder, a right shoulder, a left elbow, a right elbow, a left wrist, a right wrist, a left hip, a right hip, a left knee, a right knee, a left ankle, and a right ankle; According to the spatial relationship between the category information of different detection key points, connection edges are established between the detection key points to generate an initial graph structure of the target object.

11. A motion state recognition device, characterized in that: The motion state recognition device comprises: A processing module, used to perform key point detection and graph structure construction on an image to be detected containing a target object, and obtain detection key point information and an initial graph structure corresponding to the target object; the initial graph structure includes detection key points and connecting edges between the detection key points; A completion module, configured to diffuse and update the information of each of the detection key points in the initial graph structure of the target object to obtain an updated graph structure of the target object; the updated graph structure includes all preset key points and connecting edges between the preset key points, and the number of the detection key points included in the initial graph structure does not exceed the number of the preset key points included in the updated graph structure; The recognition module is used to determine the motion state of the target object based on the updated graph structure of the target object.

12. A terminal, characterized in that: The terminal includes a memory, a processor, and a computer program stored in the memory and running on the processor, and the processor is used to execute program data to implement the steps in the motion state recognition method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by the processor, the steps in the motion state recognition method according to any one of claims 1 to 10 are implemented.

Images