A method and system for human behavior recognition based on continuous video

Abstract

The present application discloses a method and system for human behavior recognition based on continuous video, which relates to the field of machine vision technology, and solves the problem of inaccurate human behavior recognition based on the prior art. It includes: extracting multiple key points from continuous single-frame video images in the video, and determining the attribute information of the multiple key points; obtaining single-frame video images from the video in time sequence as target video images for key point position correction; the correction includes: determining the moving trajectory of the target key point according to the positions of the N single-frame target key points before the target video image, so as to predict the moving trajectory of the target key point in the target video image, comparing the predicted target key point trajectory with the extracted actual trajectory, and if the distance exceeds the preset value, using the predicted target key point trajectory of the target video image to determine the final position of the target key point of the target video image; through the present application, the accuracy of human behavior recognition is improved.

Description

Technical Field

[0001] The present application relates to the field of machine vision technology, and in particular to a method and system for identifying human behavior. Background Art

[0002] Human behavior recognition is usually achieved by measuring the positions of key points of the human body in multiple consecutive frames of video, describing the human body posture in each consecutive frame, and finally identifying the human behavior based on the human body posture in each consecutive frame. Human behavior recognition technology currently has broad application prospects in many fields, such as intelligent monitoring, health care, sports training, entertainment, etc.

[0003] However, when depicting human posture, key points are usually identified to depict human posture. For example, the publication number CN112580570A "Key Point Detection Method for Human Posture Image" mentions that by marking key points of human posture images, a human posture estimation model is constructed, and key points of the image to be tested are identified by the human posture estimation model. However, the recognition of key points causes inaccurate depiction of human posture when processing fast movements in continuous videos, or when a single frame image is affected by light and blocked, which leads to inaccurate recognition of human behavior. Summary of the invention

[0004] Based on this, in order to solve the above technical problems, a method and system for human behavior recognition based on continuous video are provided, which can improve the accuracy of human behavior recognition.

[0005] In a first aspect, a method and system for human behavior recognition based on continuous video, the method comprising:

[0006] Acquire continuous single-frame video images in the video, extract multiple key points from the single-frame video images, and obtain the actual coordinates of each key point;

[0007] Determine attribute information of multiple key points based on the distance relationship between the actual coordinates of the multiple key points;

[0008] A single-frame video image is obtained from a video in a time sequence as a target video image for key point position correction to obtain the final position of the key point in each frame of the video image in the video; the correction includes: taking the key points of the same attribute in the target video image and N single-frame video images before the target video image as the target key points, determining the moving trajectory of the target key points in the N single-frame video images before the target video image according to the positions of the target key points in the N single-frame video images before the target video image, predicting the moving trajectory of the target key points in the target video image according to the moving trajectory in the N single-frame video images before the target video image, comparing the predicted target key point trajectory of the target video image with the actual trajectory of the target key points extracted from the target video image, and if the distance exceeds a preset value, determining the final position of the target key points of the target video image using the predicted target key point trajectory of the target video image;

[0009] Action classification is performed based on the final positions of the key points of the video images of all single frames in the video.

[0010] In the above solution, optionally, the action classification based on the positions of key points of the video images of all single frames in the video includes:

[0011] Preprocessing the final positions of key points in the video images of all single frames in the video, wherein the preprocessing includes dimensionality reduction and standardization;

[0012] Characterize the human body movements in a single-frame video image based on the final positions of key points in the single-frame video image;

[0013] Behavior classification based on human actions in all frames of video images.

[0014] In the above solution, optionally, predicting the moving trajectory of the target key point in the target video image according to the moving trajectory in N single-frame video images before the target video image includes:

[0015] Determine the moving trajectory of the target key point in the N single-frame video images before the target video image according to the coordinates of the target key point and the coordinates of other key points in the N single-frame video images before the target video image;

[0016] The moving trajectory of the target key point is input into a pre-built prediction model to obtain the trajectory prediction of the target key point in the target video image.

[0017] In the above solution, further optionally, the construction of the prediction model includes:

[0018] Obtain continuous single-frame images from multiple groups of videos, and annotate key points of different forms of human bodies in the continuous single-frame images of each group of videos;

[0019] The key points of the continuous single-frame images of each group of videos are modeled according to the time domain to obtain a prediction model.

[0020] In the above scheme, optionally, the prediction model is constructed using an LSTM long short-term memory network.

[0021] In the above solution, optionally, when extracting multiple key points from the single-frame video image, the number and attributes of the key points to be extracted are predetermined.

[0022] In the above scheme, optionally, the number of the predetermined key points is 10-15, and the predetermined key point attributes include left knee joint, right knee joint, left hip, right hip, left ankle joint, right ankle joint, upper neck, top of the head, left wrist, right wrist, left elbow, right elbow, left shoulder, right shoulder, or multiple thereof.

[0023] In a second aspect, a human behavior recognition system based on continuous video is provided, the system comprising:

[0024] Key point extraction module: used to obtain continuous single-frame video images in the video, extract multiple key points from the single-frame video images, and obtain the actual coordinates of each key point;

[0025] Key point attribute determination module: used to determine the attribute information of multiple key points based on the distance relationship between the actual coordinates of multiple key points;

[0026] Key point position correction module: used for acquiring single-frame video images from the video in time sequence as target video images for key point position correction, so as to obtain the final position of the key points in each frame of video images in the video; the correction includes: taking the key points with the same attributes in the target video image and the N single-frame video images before the target video image as the target key points, determining the moving trajectory of the target key points in the N single-frame video images before the target video image according to the positions of the target key points in the N single-frame video images before the target video image, predicting the moving trajectory of the target key points in the target video image according to the moving trajectory in the N single-frame video images before the target video image, comparing the predicted target key point trajectory of the target video image with the actual trajectory of the target key points extracted from the target video image, and if the distance exceeds a preset value, determining the final position of the target key points of the target video image using the predicted target key point trajectory of the target video image;

[0027] Action classification module: used to classify actions based on the final positions of the key points of the video images of all single frames in the video.

[0028] In a third aspect, a computer device includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for human behavior recognition based on continuous video described in the first aspect are implemented.

[0029] In a fourth aspect, a computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of a method for human behavior recognition based on continuous video as described in the first aspect.

[0030] This application has at least the following beneficial effects:

[0031] This application first extracts the key points of the image in each frame of the video, and then determines the attributes of the key points according to the positional relationship of the coordinates of all the key points in the image. The trajectory of the target key point of the target image in the video is predicted using the trajectory of the key points with the same attributes as the target key point in the previous multiple frames. If the trajectory is very different from the actual one, the predicted trajectory is used instead of the actual position. Therefore, the problem of inaccurate key point recognition caused by light, occlusion, etc. of the human body is overcome, thereby improving the accuracy of the portrayal of human posture and the accuracy of human behavior. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 A flowchart of a method for human behavior recognition based on continuous video is provided for one embodiment of the present application. DETAILED DESCRIPTION

[0033] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0034] In one embodiment, Figure 1 As shown, a method for human behavior recognition based on continuous video is provided, comprising the following steps:

[0035] Step S1: obtaining continuous single-frame video images in a video, extracting multiple key points from the single-frame video images, and obtaining the actual coordinates of each key point.

[0036] Step S2: Determine the attribute information of the multiple key points based on the distance relationship between the actual coordinates of the multiple key points.

[0037] In step S2, by analyzing the relative positions and connection relationships between the key points, it is possible to further determine which part of the human body they belong to. The attribute information of the key point is the part of the human body that the key point is, including the top of the head, facial features, neck, limbs and other parts.

[0038] Step S3: acquiring a single-frame video image from the video in time sequence as a target video image to perform key point position correction, so as to obtain the final position of the key point in each frame of the video image in the video; the correction comprises: taking the key points of the same attribute in the target video image and the N single-frame video images before the target video image as the target key points, determining the moving trajectory of the target key points in the N single-frame video images before the target video image according to the positions of the target key points in the N single-frame video images before the target video image, predicting the moving trajectory of the target key points in the target video image according to the moving trajectory in the N single-frame video images before the target video image, comparing the predicted target key point trajectory of the target video image with the actual trajectory of the target key points extracted from the target video image, and if the distance exceeds a preset value, determining the final position of the target key points of the target video image using the predicted target key point trajectory of the target video image;

[0039] In step S3, if the predicted trajectory of the target key points of the target video image is compared with the actual trajectory of the extracted target key points of the target video image, if the distance is less than a certain value, no replacement is performed, indicating that the position of the extracted key points is very accurate, and therefore, no replacement is performed.

[0040] After the position of the target key point of the current frame image is determined according to the predicted trajectory, the next frame image of the current frame image uses the position of the target key point determined by the predicted trajectory to re-describe the trajectory for subsequent comparison and replacement.

[0041] Step S4: Action classification is performed based on the final positions of the key points of the video images of all single frames in the video.

[0042] In the above-mentioned method and system for human behavior recognition based on continuous video, the key points of the image in each frame of the video are first extracted, and then the attributes of the key points are determined according to the positional relationship of the coordinates of all the key points in the image. The trajectory of the target key point of the target image in the video is predicted using the trajectory of the key point with the same attribute as the target key point in the previous multiple frames. If the trajectory is very different from the actual one, the predicted trajectory is used to replace the actual position, so the problem of inaccurate key point recognition caused by light, occlusion, etc. of the human body is overcome, thereby improving the accuracy of the portrayal of human posture and the accuracy of human behavior.

[0043] In one embodiment, the action classification based on the positions of the key points of the video images of all single frames in the video includes:

[0044] Preprocessing the final positions of key points in the video images of all single frames in the video, wherein the preprocessing includes dimensionality reduction and standardization;

[0045] Characterize the human body movements in a single-frame video image based on the final positions of key points in the single-frame video image;

[0046] Behavior classification based on human actions in all frames of video images.

[0047] In one embodiment, predicting the moving trajectory of the target key point in the target video image according to the moving trajectory in the N single-frame video images before the target video image includes:

[0048] Determine the moving trajectory of the target key point in the N single-frame video images before the target video image according to the coordinates of the target key point and the coordinates of other key points in the N single-frame video images before the target video image;

[0049] The moving trajectory of the target key point is input into a pre-built prediction model to obtain the trajectory prediction of the target key point in the target video image.

[0050] In one embodiment, the construction of the prediction model includes:

[0051] Obtain continuous single-frame images from multiple groups of videos, and annotate key points of different forms of human bodies in the continuous single-frame images of each group of videos;

[0052] The key points of the continuous single-frame images of each group of videos are modeled according to the time domain to obtain a prediction model.

[0053] In this embodiment, the positions of the key points of the human body in multiple frames of video under different actions are fed into the prediction model, so that the trend of the key points of the next frame can be predicted based on the trend of the key points of the human body.

[0054] In one embodiment, the prediction model is constructed using an LSTM long short-term memory network.

[0055] In one embodiment, when extracting multiple key points from the single-frame video image, the number and attributes of the key points to be extracted are predetermined.

[0056] In one embodiment, the number of predetermined key points is 10-15, the number of predetermined key points is 10-15, and the predetermined key point attributes include left knee joint, right knee joint, left hip, right hip, left ankle joint, right ankle joint, upper neck, top of the head, left wrist, right wrist, left elbow, right elbow, left shoulder, right shoulder, or multiple thereof.

[0057] In one embodiment, a human behavior recognition system based on continuous video is provided, the system comprising:

[0058] Key point extraction module: used to obtain continuous single-frame video images in the video, extract multiple key points from the single-frame video images, and obtain the actual coordinates of each key point;

[0059] Key point attribute determination module: used to determine the attribute information of multiple key points based on the distance relationship between the actual coordinates of multiple key points;

[0060] Key point position correction module: used for acquiring single-frame video images from the video in time sequence as target video images for key point position correction, so as to obtain the final position of the key points in each frame of video images in the video; the correction includes: taking the key points with the same attributes in the target video image and the N single-frame video images before the target video image as the target key points, determining the moving trajectory of the target key points in the N single-frame video images before the target video image according to the positions of the target key points in the N single-frame video images before the target video image, predicting the moving trajectory of the target key points in the target video image according to the moving trajectory in the N single-frame video images before the target video image, comparing the predicted target key point trajectory of the target video image with the actual trajectory of the target key points extracted from the target video image, and if the distance exceeds a preset value, determining the final position of the target key points of the target video image using the predicted target key point trajectory of the target video image;

[0061] Action classification module: used to classify actions based on the final positions of the key points of the video images of all single frames in the video.

[0062] For the specific definition of a human behavior recognition system based on continuous video, please refer to the definition of a human behavior recognition method based on continuous video above, which will not be repeated here. Each module in the above-mentioned human behavior recognition system based on continuous video can be implemented in whole or in part by software, hardware and a combination thereof. The above-mentioned modules can be embedded in or independent of the processor in the computer device in the form of hardware, or can be stored in the memory of the computer device in the form of software, so that the processor can call and execute the operations corresponding to the above modules.

[0063] In one embodiment, a computer device is provided, which may be a server. The computer device includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above-mentioned method for human behavior recognition based on continuous video is implemented.

[0064] In one embodiment, a computer program product is also provided, including a computer program / instruction, which, when executed by a processor, involves all or part of the process in the above-mentioned embodiment method.

[0065] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).

[0066] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0067] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.

Claims

1. A method for human behavior recognition based on continuous video, characterized in that: The method comprises: Obtaining continuous single-frame video images in the video, extracting multiple key points from the single-frame video images, and obtaining the actual coordinates of each key point; Determine attribute information of multiple key points based on the distance relationship between the actual coordinates of the multiple key points; A single-frame video image is obtained from a video in time sequence as a target video image for key point position correction to obtain the final position of the key point in each frame of the video image in the video; the correction includes: taking the target video image and the key points of the same attribute in the N single-frame video images before the target video image as the target key points, determining the moving trajectory of the target key points in the N single-frame video images before the target video image according to the positions of the target key points in the N single-frame video images before the target video image, predicting the moving trajectory of the target key points in the target video image according to the moving trajectory in the N single-frame video images before the target video image, comparing the predicted target key point trajectory of the target video image with the actual trajectory of the target key points extracted from the target video image, and if the distance exceeds a preset value, determining the final position of the target key points of the target video image using the predicted target key point trajectory of the target video image; predicting the moving trajectory of the target key points in the target video image according to the moving trajectory in the N single-frame video images before the target video image includes: Determine the moving trajectory of the target key point in the N single-frame video images before the target video image according to the coordinates of the target key point and the coordinates of other key points in the N single-frame video images before the target video image; Inputting the moving trajectory of the target key point into a pre-built prediction model to obtain the trajectory prediction of the target key point in the target video image; Action classification is performed based on the final positions of the key points of the video images of all single frames in the video.

2. The method for human behavior recognition based on continuous video according to claim 1, characterized in that: The action classification based on the positions of the key points of the video images of all single frames in the video includes: Preprocessing the final positions of key points in the video images of all single frames in the video, wherein the preprocessing includes dimensionality reduction and standardization; Characterize the human body movements in a single-frame video image based on the final positions of key points in the single-frame video image; Behavior classification based on human actions in all frames of video images.

3. The method for human behavior recognition based on continuous video according to claim 1, characterized in that: The construction of the prediction model includes: Obtain continuous single-frame images from multiple groups of videos, and annotate key points of different forms of human bodies in the continuous single-frame images of each group of videos; The key points of the continuous single-frame images of each group of videos are modeled according to the time domain to obtain a prediction model.

4. The method for human behavior recognition based on continuous video according to claim 3 is characterized in that: The prediction model is constructed using LSTM long short-term memory network.

5. The method for human behavior recognition based on continuous video according to claim 1, characterized in that: When extracting multiple key points from the single-frame video image, the number and attributes of the key points to be extracted are predetermined.

6. The method for human behavior recognition based on continuous video according to claim 5, characterized in that: The number of the predetermined key points is 10-15, and the predetermined key point attributes include left knee joint, right knee joint, left hip, right hip, left ankle joint, right ankle joint, upper neck, top of the head, left wrist, right wrist, left elbow, right elbow, left shoulder, right shoulder, or multiple thereof.

7. A human behavior recognition system based on continuous video, characterized in that: The system comprises: Key point extraction module: used to obtain continuous single-frame video images in the video, extract multiple key points from the single-frame video images, and obtain the actual coordinates of each key point; Key point attribute determination module: used to determine the attribute information of multiple key points based on the distance relationship between the actual coordinates of multiple key points; The key point position correction module is used to obtain a single-frame video image from the video in a time sequence as a target video image to perform key point position correction, so as to obtain the final position of the key point in each frame of the video image in the video; the correction includes: taking the key points of the same attribute in the target video image and the N single-frame video images before the target video image as the target key points, determining the moving trajectory of the target key points in the N single-frame video images before the target video image according to the positions of the target key points in the N single-frame video images before the target video image, predicting the moving trajectory of the target key points in the target video image according to the moving trajectory in the N single-frame video images before the target video image, and determining the moving trajectory of the target key points in the target video image according to the predicted target key points in the target video image. The key point trajectory is compared with the actual trajectory of the target key point of the extracted target video image, and if the distance exceeds a preset value, the final position of the target key point of the target video image is determined using the predicted target key point trajectory of the target video image; predicting the moving trajectory of the target key point in the target video image according to the moving trajectory in the N single-frame video images before the target video image includes: determining the moving trajectory of the target key point in the N single-frame video images before the target video image according to the coordinates of the target key point in the N single-frame video images before the target video image and the coordinates of other key points; inputting the moving trajectory of the target key point into a pre-constructed prediction model to obtain the trajectory prediction of the target key point in the target video image; Action classification module: used to classify actions based on the final positions of the key points of the video images of all single frames in the video.

8. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 6 are implemented.

9. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 6 are implemented.

Images