Human body posture recognition method and device

Abstract

The present application discloses a method and device for human posture recognition, which obtains multiple two-dimensional images containing a target object; determines the human motion trajectory of the target object based on the image feature information of at least two two-dimensional images in the multiple two-dimensional images; obtains the first image structure information of the first image and the second image structure information of the second image in the multiple two-dimensional images; and recognizes the current human posture of the target object based on the human motion trajectory, the first image structure information and the second image structure information. The method realizes the recognition of human posture in three-dimensional space through multiple two-dimensional images, and improves the accuracy of the recognition result.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and more specifically to a method and device for human body posture recognition. Background Art

[0002] Human posture recognition is widely used in human-computer interaction, intelligent control, human behavior analysis and other fields. At present, key point detection technology is usually used for human posture recognition. In the key point detection process, the instability of a single two-dimensional image will reduce the accuracy of human posture recognition results. Summary of the invention

[0003] In view of this, this application provides the following technical solutions:

[0004] A human body posture recognition method, comprising:

[0005] Acquire multiple frames of two-dimensional images containing the target object;

[0006] Determining a human body motion trajectory of the target object based on image feature information of at least two frames of two-dimensional images among the multiple frames of two-dimensional images;

[0007] Acquire first image structure information of a first image and second image structure information of a second image in the multiple frames of two-dimensional images, wherein the number of image frames between the first image and the second image meets a specific condition;

[0008] Based on the human body motion trajectory, the first image structure information and the second image structure information, a current human body posture of the target object is identified.

[0009] Optionally, the method further comprises:

[0010] Acquire a reference image of the target object;

[0011] Based on the current human body posture and the target posture corresponding to the reference image, it is detected whether the current human body posture matches the target posture to obtain a detection result.

[0012] Optionally, the method further comprises:

[0013] If the detection result meets the prompt condition, a prompt message is generated, and the prompt message is used to prompt the target object to adjust the current human body posture sub-target posture.

[0014] Optionally, acquiring a reference image containing the target object includes:

[0015] Determine a captured image of the target object in a target posture as a reference image;

[0016] or,

[0017] generating a screening condition corresponding to the target object being in a target posture;

[0018] Based on the screening condition, a reference image is determined in the multiple frames of two-dimensional images.

[0019] Optionally, determining the human body motion trajectory of the target object based on image feature information of at least two frames of images in the multiple frames of two-dimensional images includes:

[0020] Performing face recognition on at least two frames of the multiple two-dimensional images to determine face motion trajectory information;

[0021] Performing key point recognition on at least two frames of the two-dimensional images in the multiple frames of the two-dimensional images to determine key point motion trajectory information, wherein the key points include at least face key points and body key points;

[0022] Based on the face motion trajectory information and the key point motion trajectory information, the body motion trajectory of the target object is determined.

[0023] Optionally, performing face recognition on at least two frames of the multiple two-dimensional images to determine the face recognition trajectory includes:

[0024] Performing face recognition on at least two frames of the multiple two-dimensional images to obtain size and position information of a face recognition frame of each recognized image;

[0025] Determining face motion trajectory information based on the size and position information of the face recognition frame of each recognized image;

[0026] The step of performing key point recognition on at least two frames of two-dimensional images among the multiple frames of two-dimensional images to determine key point motion trajectory information includes:

[0027] Performing key point recognition on at least two frames of the multiple two-dimensional images to obtain position information of key points of each recognized image;

[0028] Based on the position information of the key points of each identified image, the key point motion trajectory information is determined.

[0029] Optionally, determining the human body motion trajectory of the target object based on the face motion trajectory information and the key point motion trajectory information includes:

[0030] Inputting the face motion trajectory information and the key point motion trajectory information into a trajectory recognition model to obtain a plurality of human body motion sub-trajectories and a confidence level corresponding to each human body motion sub-trajectory;

[0031] Based on the confidence level corresponding to each human motion sub-trajectory, a human motion trajectory of the target object is determined from a plurality of human motion sub-trajectories.

[0032] Optionally, the acquiring first image structure information of a first image and second image structure information of a second image in the multiple frames of two-dimensional images includes:

[0033] Obtaining a first face recognition frame and a first group of key points of a first image in the multiple frames of images;

[0034] Determine the first face recognition frame and a first structural relationship formed by the first group of key points as first image structural information;

[0035] Acquire a second face recognition frame and a second set of key points of a second image in the multiple frames of images;

[0036] The first face recognition frame and a second structural relationship formed by the second group of key points are determined as second image structure information.

[0037] Optionally, the identifying and obtaining a current human body posture of the target object based on the human body motion trajectory, the first image structure information, and the second image structure information includes:

[0038] Based on the first image structure information and the second image structure information, identifying and obtaining the posture information of the target object;

[0039] Based on the human body motion trajectory and the posture information, a current human body posture of the target object is determined.

[0040] A human body posture recognition device, comprising:

[0041] An image acquisition unit, used for acquiring multiple frames of two-dimensional images containing a target object;

[0042] A trajectory determination unit, configured to determine a human body motion trajectory of the target object based on image feature information of at least two frames of two-dimensional images among the multiple frames of two-dimensional images;

[0043] An information acquisition unit, configured to acquire first image structure information of a first image and second image structure information of a second image in the plurality of two-dimensional image frames, wherein the number of image frames between the first image and the second image meets a specific condition;

[0044] A posture recognition unit is used to recognize the current human posture of the target object based on the human motion trajectory, the first image structure information and the second image structure information.

[0045] Through the above technical solutions, it can be known that the present application discloses a method and device for human posture recognition, which obtains multiple two-dimensional images containing a target object; determines the human motion trajectory of the target object based on the image feature information of at least two two-dimensional images in the multiple two-dimensional images; obtains the first image structure information of the first image and the second image structure information of the second image in the multiple two-dimensional images; and recognizes the current human posture of the target object based on the human motion trajectory, the first image structure information and the second image structure information. The recognition of human posture in three-dimensional space through multiple two-dimensional images is realized, and the accuracy of the recognition result is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0047] Figure 1 A schematic diagram of a human body posture recognition method provided in an embodiment of the present application;

[0048] Figure 2 A schematic diagram of a process for obtaining a human motion trajectory based on a trajectory recognition model provided in an embodiment of the present application;

[0049] Figure 3 A schematic diagram of image structure information in different frame images provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of image structure information corresponding to a target frame image provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of a human posture recognition device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0053] In the embodiment of the present application, a method for human posture recognition is provided. Human posture recognition is an important part of human behavior and action recognition. Its purpose is to input parameter information of the whole human body or local limbs, such as the position and orientation of the human head. In the present application, multiple frames of two-dimensional images are used to recognize human posture, so that the posture recognition result is more robust, and the posture recognition in three-dimensional space can be completed. For example, the curvature of the cervical vertebra on the side of the human body can be recognized based on multiple frames of frontal human body images.

[0054] See also Figure 1 , which shows a flow chart of a human body posture recognition method provided by an embodiment of the present application, the method may include the following steps:

[0055] S101, acquiring multiple frames of two-dimensional images containing a target object.

[0056] The target object refers to a living organism to be identified, such as a human user. It should be noted that the target object can be one or more. That is, the human posture recognition method provided by the embodiment of the present application can be applied to perform posture recognition on a specific user in front of an electronic device, or it can be applied to perform posture recognition on multiple users captured by a camera, such as recognizing the postures of all students in a classroom.

[0057] The multi-frame two-dimensional image may be a plurality of frames of images in the collected video data. The video may be understood to be composed of at least one frame of image data. In order to recognize the human body posture in the video, the video may be divided into frames of image data, and each frame of image data may be analyzed separately. The multi-frame two-dimensional image may be a continuous multi-frame two-dimensional image, that is, the acquired two-dimensional image is a temporally continuous image. Correspondingly, in order to reduce the amount of data calculation, a specific multi-frame two-dimensional image may be selected from the video data. For example, the video data is a video with a duration of 5 seconds, and two frames of images may be taken from each second to form the final multi-frame two-dimensional image to be processed. The two-dimensional image is a planar image that does not contain depth information. The two-dimensional image is the four directions of left, right, up and down, and there is no front and back direction. The two-dimensional image acquired in the embodiment of the present application may be an RGB (representing the colors of the three channels of red, green, and blue) color two-dimensional image, or a grayscale two-dimensional image, which is not limited.

[0058] S102: Determine a human body motion trajectory of a target object based on image feature information of at least two frames of two-dimensional images among the multiple frames of two-dimensional images.

[0059] The image feature information of each two-dimensional image in the multiple two-dimensional images can be processed. Correspondingly, in order to improve processing efficiency and reduce the occupation of processing resources, at least two two-dimensional images in the multiple two-dimensional images can also be processed. The selected at least two two-dimensional images can be two-dimensional images corresponding to image frames that are continuous in time, or two-dimensional images corresponding to image frames that are continuous in sequence, that is, two-dimensional images of several consecutive frames, such as 8 consecutive two-dimensional images. The specific method can be determined according to the application scenario and the specific recognition accuracy.

[0060] The image feature information of a two-dimensional image refers to the features that can be identified in the image, which may include facial features and key point features. The key points may be facial key points or body key points. For example, they may include key points of the neck, left shoulder, right shoulder and other parts of the human body. Specifically, facial features may be obtained by identifying a face recognition frame, so that the facial features are obtained by the size and position information of the face recognition frame. The key point features are obtained by the position information of the key points. In this way, after obtaining the image feature information, analysis and drawing may be performed based on the facial features and key point features to obtain the human body motion trajectory. For example, the movement trajectory of the key points and the movement trajectory of the face are determined by the coordinate positions of the key points, and the human body motion trajectory is obtained by comprehensive analysis. The specific implementation process of the human body motion trajectory will be described in the subsequent embodiments of this application.

[0061] S103: Acquire first image structure information of a first image and second image structure information of a second image in a plurality of frames of two-dimensional images.

[0062] Among them, the first image and the second image both belong to an image of a certain frame in the multi-frame two-dimensional image, and the number of image frames between the first image and the second image meets a specific condition. The specific condition may be that the number of image frames in the interval meets a specific number threshold, such as the number of image frames in the interval between the first image and the second image is greater than 20 frames. In another possible implementation, the specific condition may be that the number of image frames in the interval meets a specific acquisition time condition. For example, the first image may be any frame image within the first second acquisition time, and the second image may be any frame image within the last second acquisition time. In order to more clearly distinguish the first image from the second image, the first frame of the two-dimensional image in the multi-frame two-dimensional image may be used as the first image, and the last frame of the two-dimensional image in the multi-frame two-dimensional image may be used as the second image.

[0063] Among them, the image structure information is information determined based on the structural relationship between target points in the current image (such as the first image or the second image). The target point can be a key point determined in the target object in the first image or the second image, such as a face key point and a body key point. Correspondingly, the structural relationship between the target points is a relationship obtained by representing a closed-loop structure formed by the target points in a certain order. Correspondingly, a closed-loop structure representation of the face target points can obtain a face recognition frame, and a closed-loop structure obtained by connecting multiple key points on the body in sequence can be a structural relationship of a group of key points. Therefore, the structural relationship between the face recognition frame and a group of key points is determined as the image structure information.

[0064] By analyzing the image structure information of the first image and the second image, the position changes between the key points in different images can be obtained, so that the image depth information can be inferred. For example, if the face recognition box gradually becomes larger, it can be concluded that the target object is gradually approaching the camera. That is, the depth information of the target object's body moving forward is obtained.

[0065] S104: Based on the human body motion trajectory, the first image structure information and the second image structure information, identify and obtain the current human body posture of the target object.

[0066] The human body motion trajectory can represent the trajectory of the movement of the human body's center of gravity, and the first image structure information and the second image structure information can be analyzed to obtain the depth information of the image, that is, the posture information of the human body can be obtained. Then, through the human body motion trajectory and the posture information of the human body, the current human body posture of the target object can be identified, such as the direction and distance information of the head movement. On the basis of the original two-dimensional information, the image depth information is obtained, so that the combined movement information of any one direction or multiple directions of the front, back, left, right, up and down in the human body posture can be obtained. The recognized human body posture is a posture in three-dimensional space, such as whether the body is leaning forward, the degree of leaning forward, the curvature of the cervical spine, etc. The specific recognition process will be described in detail in the subsequent embodiments of this application.

[0067] The embodiment of the present application provides a method for human posture recognition, which obtains multiple two-dimensional images containing a target object; determines the human body motion trajectory of the target object based on the image feature information of at least two two-dimensional images in the multiple two-dimensional images; obtains the first image structure information of the first image and the second image structure information of the second image in the multiple two-dimensional images; and recognizes the current human body posture of the target object based on the human body motion trajectory, the first image structure information and the second image structure information. The method realizes the recognition of human body posture in three-dimensional space through multiple two-dimensional images, and improves the accuracy of the recognition result.

[0068] The human body posture recognition method provided in the embodiment of the present application can be applied to the recognition of human body posture, and further judge the state and behavior of the human body through the recognized posture, so as to achieve the purpose of controlling electronic equipment through the state of the human body, or adjusting the human body posture.

[0069] Taking the control of electronic devices by human body state as an example, it can be detected whether there is a control instruction matching the current human body posture, and if so, the control instruction matching the current human body posture is obtained, and the electronic device is controlled to reach the target state according to the control instruction. For example, the control instruction corresponding to shaking head is pre-stored as the page turning of reading in the reading interface, then when the current human body posture of the target object is identified as shaking head, the reading interface is controlled to turn pages.

[0070] Taking the adjustment of human body posture as an example, it is necessary to compare the current human body posture of the target object obtained by identification with the target posture, and judge whether to remind the target object to adjust the current human body posture by comparing the result. In a possible implementation, a reference image containing the target object can be obtained, and based on the current human body posture and the target posture corresponding to the reference image, whether the current human body posture matches the target posture is detected to obtain a detection result. Correspondingly, if the detection result meets the prompt condition, a prompt message is generated, and the prompt message is used to prompt the target object to adjust the current human body posture to the target posture. Among them, the prompt condition can be a condition that characterizes the matching of the current human body posture with the target posture, such as whether it is within the error range of the target posture. For example, the target posture is a posture with a straight head, and if the target object is detected to be in a posture with a slight tilted head, that is, within the error range of the posture with a straight head, no prompt message is generated. If the target object is in a posture with a tilted head, and the amplitude exceeds the range of the posture with a straight head, a prompt message is generated to prompt the target object to adjust the head posture.

[0071] In the embodiment of the present application, the target posture is the posture of the target object in the reference image. The captured image of the target object in the target posture can be determined as the reference image. For example, an image of the target object in proper posture can be captured as the reference image as a reference for subsequent processing. It is also possible to generate a filtering condition corresponding to the target object in the target posture, and determine the reference image in multiple frames of two-dimensional images based on the filtering condition. Since multiple frames of two-dimensional images containing the target object are to be obtained, the reference image can be determined therein based on the filtering condition. The filtering condition may include the two-dimensional coordinate information of the corresponding key points when in the target posture, and the reference image is determined in the multiple frames of two-dimensional images that have been obtained based on the two-dimensional coordinate information.

[0072] In order to make the human posture recognition result more robust, in the embodiment of the present application, the human body motion trajectory is determined by the face motion trajectory information and the key point motion trajectory information. Correspondingly, the human body motion trajectory of the target object is determined based on the image feature information of at least two frames of two-dimensional images in the multiple frames of two-dimensional images, including:

[0073] Performing face recognition on at least two frames of the multiple two-dimensional images to determine face motion trajectory information;

[0074] Performing face recognition on at least two frames of the multiple two-dimensional images to determine key point motion trajectory information, wherein the key points include face key points and body key points;

[0075] Based on the face motion trajectory information and the key point motion trajectory information, the body motion trajectory of the target object is determined.

[0076] The step of performing face recognition on at least two of the multiple two-dimensional images to determine face motion trajectory information includes: performing face recognition on at least two of the multiple two-dimensional images to obtain size and position information of a face recognition frame of each recognized image; and determining face motion trajectory information based on the size and position information of the face recognition frame of each recognized image.

[0077] The method of performing key point recognition on at least two frames of two-dimensional images among the multiple frames of two-dimensional images to determine the key point motion trajectory information includes: performing key point recognition on at least two frames of images among the multiple frames of two-dimensional images to obtain the position information of the key points of each recognized image; and determining the key point motion trajectory information based on the position information of the key points of each recognized image.

[0078] Usually, facial features are identified through a face recognition frame. When the target object is in motion, the face recognition frame used to identify facial features will also change position with the movement of the target object. Correspondingly, the size and position of the face recognition frame can be used to determine the corresponding face motion trajectory information. The size of the face recognition frame can determine the distance between the target object and the camera, that is, the depth information in the two-dimensional image containing the target object can be obtained, and then the position of the face recognition frame can be analyzed to obtain the face position offset information. Therefore, the face motion trajectory information can be determined by the size and position of the face recognition frame.

[0079] Correspondingly, the motion trajectory of the key points can be drawn through the position coordinates of the key points in different frames, thereby obtaining the motion trajectory information of the key points. Then, the depth information and face position offset information in the image can be obtained by analyzing the motion trajectory information of the face. The position offset information of the key points can be obtained by analyzing the motion trajectory information of the key points. By combining the above information, the motion trajectory information of the human body, that is, the motion trajectory of the center of gravity of the human body, can be obtained.

[0080] On the basis of the above embodiments, an artificial intelligence neural network model can also be used to process the above information. A face motion trajectory recognition model is obtained by training a neural network with a sample set including multiple face feature samples, wherein the face feature samples include face recognition frame size and position information, and the face motion trajectory recognition model can identify and obtain the face motion trajectory. Correspondingly, a key point motion trajectory recognition model is obtained by training a neural network with a sample set including multiple key point feature samples, wherein the key point feature samples include key point two-dimensional coordinate information, and the key point motion trajectory recognition is used to identify and obtain the key point trajectory. The trajectory recognition model obtained by training with the face motion trajectory and the key point motion trajectory as training samples is used to identify and obtain the human body motion trajectory. Among them, the present application does not limit the types and training processes of the face motion trajectory recognition model, the key point motion trajectory recognition model and the trajectory recognition model, such as a convolutional neural network model or a recurrent neural network model.

[0081] See also Figure 2 , which shows a schematic diagram of a process of obtaining a human motion trajectory based on a trajectory recognition model provided in an embodiment of the present application.

[0082] Acquire multiple two-dimensional images containing the target object, perform image feature recognition on at least two of the images, determine the image for which the image feature recognition is performed as the recognized image, obtain the size and position information of the face recognition frame of each recognized image, use the size and position information of the face recognition frame of each recognized image as a face information queue, input it into the face motion trajectory recognition model, and obtain face motion trajectory information. Then obtain the position information of the key points of each recognized image, use the position information of the key points of each recognized image as a key point information queue, input it into the key point motion trajectory recognition model, and obtain key point motion trajectory information. Then input the face motion trajectory information and the key point motion trajectory information into the trajectory recognition model to obtain the human body motion trajectory of the target object.

[0083] In one embodiment, the trajectory recognition model in the embodiment of the present application is a neural network model that can output confidence. Wherein, confidence refers to the probability that the trajectory recognition model outputs the current human motion trajectory when identifying the human motion trajectory. The trajectory recognition model is provided with confidence. For example, input information containing face motion trajectory information and key point motion trajectory information is input into the trajectory recognition model, and the trajectory recognition model outputs the corresponding confidence. When the confidence is greater than the confidence threshold, the human motion sub-trajectory corresponding to the confidence is determined as the human motion trajectory of the target object. That is, the face motion trajectory information and the key point motion trajectory information are input into the trajectory recognition model to obtain several human motion sub-trajectories and the confidence corresponding to each human motion sub-trajectory; based on the confidence corresponding to each human motion sub-trajectory, the human motion trajectory of the target object is determined in the several human motion sub-trajectories. For example, the value of the confidence is a value between 0 and 1, 1 represents the highest accuracy, and 0 represents the lowest accuracy. If the confidence of the identified human motion sub-trajectory A is 0.9, and the confidence of the identified human motion sub-trajectory B is 0.3, the human motion sub-trajectory A is determined as the human motion trajectory of the target object.

[0084] See also Figure 3 , which shows a schematic diagram of image structure information in different frame images provided by an embodiment of the present application. The image structure information presented in images of different frames is different. Figure 3 This is reflected in the size and position of the face recognition box, as well as the shape of the key points connected by lines. It can be seen that the structural information of the target object is different in different states. For a detailed description, please refer to Figure 4 The description in the embodiments will not be repeated here.

[0085] After obtaining the human body motion trajectory in the present application, the current human body posture is not determined directly based on the human body motion trajectory, but needs to be determined based on the first image structure information of the first image and the second image structure information of the second image in multiple frames of two-dimensional images.

[0086] Image structure information is information determined based on the structural relationship between target points in the current image (such as the first image or the second image). The target point may be a key point determined in the target object in the first image or the second image, such as a face key point and a body key point. Correspondingly, the structural relationship between the target points is a relationship obtained by representing a closed-loop structure formed by the target points in a certain order. Correspondingly, a closed-loop structure representation of the face target points can obtain a face recognition frame, and a closed-loop structure obtained by connecting multiple key points on the body in sequence can be a structural relationship of a group of key points. Therefore, the structural relationship between the face recognition frame and a group of key points is determined as image structure information.

[0087] Therefore, the first face recognition frame and the first group of key points of the first image in the multi-frame image are obtained; the first face recognition frame and the first structural relationship composed of the first group of key points are determined as the first image structural information; the second face recognition frame and the second group of key points of the second image in the multi-frame image are obtained; the first face recognition frame and the second structural relationship composed of the second group of key points are determined as the second image structural information. Among them, the key point types of the first group of key points and the second group of key points are the same, for example, both include left and right pupil key points, chin key points, left and right shoulder key points, etc. In this way, the closed-loop frame formed by connecting the first group of key points in a certain order is used as the first structural relationship, such as being connected in a clockwise order. Similarly, the closed-loop frame formed by connecting the second group of key points in a certain order is used as the second structural relationship.

[0088] By identifying the first image structure information and the second image structure information, the posture information of the target object can be obtained. Since the first image and the second image are separated by a preset image frame, the two can better reflect the actual motion state of the target object. For example, the first image is the first frame image of a multi-frame two-dimensional image, and the second image is the last frame image of the multi-frame two-dimensional image. The posture information of the target object, that is, the information of the posture change, can be obtained from the image structure information in the first frame image and the last frame image. The trajectory information of the target object, such as the center of gravity offset information, can be obtained through the human body motion trajectory, and the posture information, that is, the information of the posture change that causes the current motion trajectory, can be obtained through the image structure information. For example, if the center of gravity trajectory information of the target object is determined through the human body motion trajectory, the information of whether the target object has tilted forward, moved left and right, etc. can be obtained through the image structure information. The current human body posture obtained in this way is more in line with the actual situation of the target object, and the actual motion state of the target object that produces the posture can be more clearly defined. Thus, it can be accurately compared with the target posture of the reference image.

[0089] See also Figure 4 , which shows a schematic diagram of image structure information corresponding to a target frame image provided in an embodiment of the present application. Figure 4 The target frame in refers to the image frame where the reference image is located, the first frame and the last frame, which correspond to the reference image, the first frame image and the last frame image. The first frame image is the image corresponding to the first frame in the multi-frame two-dimensional image, that is, the first image in the above embodiment, and the last frame image is the image corresponding to the last frame in the multi-frame two-dimensional image, that is, the second image in the above embodiment. The dotted frame in the figure is the face recognition frame, and the solid frame is the closed figure obtained by connecting the key points in sequence, that is, the structural relationship composed of the key points. Figure 4The frames of the various images in the figure are consistent, that is, the size of the shooting screen is consistent. By analyzing the different image structure information presented in the figure, the posture information of the human body can be analyzed. Correspondingly, in Figure 4 In the embodiment shown, the key points are selected from the left and right eyes, mouth, left and right shoulders and chest key points. After connecting these points, the structural relationship of the key points can be obtained, and the structural relationship between the face recognition frame and the key points can be determined as the image structure information. Figure 4 The dotted and solid boxes in the figure determine the image structure information.

[0090] Depend on Figure 4 As shown in FIG. 1 , in the reference image and the first frame image, the posture presented in the user image is relatively close, but the face recognition frame (i.e. Figure 4 The dashed box in the middle is larger than the face recognition box in the first frame image. Usually, when the user is close to the camera, the image captured by the camera will be larger. Therefore, it can be concluded that the distance between the user and the camera in the first frame image is greater than the distance between the user and the camera in the reference image, that is, when the first frame image is taken, the user is farther away from the camera than when the reference image is taken. The shape of the key points after connecting the lines is as follows: Figure 4 The solid line frame in the middle is the structural information of the key points. For a clear description, please refer to Figure 4 The shape of the key points extracted on the right is basically the same as the shape of the key points in the benchmark image, except that the distance between the key points has changed, that is, the side lengths of the connected shapes are inconsistent. Therefore, it can be analyzed that compared with the benchmark image, the user in the first frame image has only moved in the front-to-back direction, that is, moved away from the camera.

[0091] Comparing the last frame with the first frame, the size and position of the face recognition frame in the last frame have changed. The size of the face recognition frame can be reflected by the vertex coordinates of the face recognition frame, that is, Figure 4 The user moves forward and backward, which generates depth information. Figure 4 The shapes of the extracted key points shown on the right are also different. It can be seen that the positions of the key points have changed, and this change is not only caused by the user's forward and backward movement, but also caused by changes in the user's left-right or up-down posture. Therefore, the user's posture information can be obtained through the shapes of the key points and the size and position of the face recognition box. By analyzing the image structure information of the first frame and the last frame, it can be concluded that the face recognition box becomes larger, that is, the user is approaching the camera. Correspondingly, the shapes of the key points can also be used to determine whether the user moves left and right while moving toward the camera, such as Figure 4The user tilted his head in the image. The size, position and structural relationship of the key points of the face recognition frame in the reference image can represent the information that the user is in the target posture. Then the user's current posture is judged by the user's motion trajectory and the structural information in the first and last frames. Figure 4 The structural information presented in the image can determine that the user's current posture is close to the camera and the head is tilted, which does not match the information presented in the reference image. If the camera is located on the electronic device that the user is watching, it will prompt the user to move away from the electronic device and correct the head posture to ensure the user experience.

[0092] The above embodiment is described by taking the user's forward leaning posture as an example. At the same time, the present application can also determine whether the side cervical vertebra of the human body is bent based on the two-dimensional image, that is, the corresponding depth information can be obtained based on the two-dimensional image. This process has been described in the above embodiment and will not be discussed in detail here. Therefore, the embodiment of the present application realizes the recognition of human body posture in three-dimensional space through multiple frames of two-dimensional images, thereby improving the accuracy of the recognition result.

[0093] See also Figure 5 In an embodiment of the present application, a human body posture recognition device is also provided, comprising:

[0094] An image acquisition unit 10 is used to acquire multiple frames of two-dimensional images containing a target object;

[0095] A trajectory determination unit 20, configured to determine a human body motion trajectory of the target object based on image feature information of at least two frames of two-dimensional images among the multiple frames of two-dimensional images;

[0096] An information acquisition unit 30 is used to acquire first image structure information of a first image and second image structure information of a second image in the multiple frames of two-dimensional images, wherein the number of image frames between the first image and the second image meets a specific condition;

[0097] The posture recognition unit 40 is used to recognize the current human posture of the target object based on the human motion trajectory, the first image structure information and the second image structure information.

[0098] exist Figure 4 Based on the embodiment, the device further includes:

[0099] A reference image acquisition unit, used to acquire a reference image containing the target object;

[0100] The posture detection unit is used to detect whether the current human body posture matches the target posture based on the current human body posture and the target posture corresponding to the reference image, and obtain a detection result.

[0101] Optionally, the device further comprises:

[0102] The prompt information generating unit is used to generate prompt information if the detection result meets the prompt condition, and the prompt information is used to prompt the target object to adjust the current human body posture to the target posture.

[0103] In one implementation, the reference image acquisition unit is specifically used to:

[0104] Determine a captured image of the target object in a target posture as a reference image;

[0105] or,

[0106] generating a screening condition corresponding to the target object being in a target posture;

[0107] Based on the screening condition, a reference image is determined in the multiple frames of two-dimensional images.

[0108] Based on the above embodiment, the trajectory determination unit includes:

[0109] A first recognition subunit is used to perform face recognition on at least two frames of the multiple two-dimensional images to determine face motion trajectory information;

[0110] A second recognition subunit is used to perform key point recognition on at least two frames of the two-dimensional image in the multiple frames of the two-dimensional image to determine the key point motion trajectory information, wherein the key points include at least face key points and body key points;

[0111] The first determination subunit is used to determine the human body motion trajectory of the target object based on the face motion trajectory information and the key point motion trajectory information.

[0112] Optionally, the first identification subunit is specifically used for:

[0113] Performing face recognition on at least two frames of the multiple two-dimensional images to obtain size and position information of a face recognition frame of each recognized image;

[0114] Determining face motion trajectory information based on the size and position information of the face recognition frame of each recognized image;

[0115] The second identification subunit is specifically used for:

[0116] Performing key point recognition on at least two frames of the multiple two-dimensional images to obtain position information of key points of each recognized image;

[0117] Based on the position information of the key points of each identified image, the key point motion trajectory information is determined.

[0118] Optionally, the first determining subunit is specifically configured to:

[0119] Inputting the face motion trajectory information and the key point motion trajectory information into a trajectory recognition model to obtain a plurality of human body motion sub-trajectories and a confidence level corresponding to each human body motion sub-trajectory;

[0120] Based on the confidence level corresponding to each human motion sub-trajectory, a human motion trajectory of the target object is determined from a plurality of human motion sub-trajectories.

[0121] Based on the above embodiment, the information acquisition unit includes:

[0122] A first acquisition subunit, used to acquire a first face recognition frame and a first group of key points of a first image in the multiple frames of images;

[0123] A second determining subunit is used to determine the first face recognition frame and the first structural relationship formed by the first group of key points as first image structural information;

[0124] A second acquisition subunit, used to acquire a second face recognition frame and a second group of key points of a second image in the multiple frames of images;

[0125] The third determining subunit is used to determine the first face recognition frame and the second structural relationship formed by the second group of key points as second image structure information.

[0126] Optionally, the gesture recognition unit is specifically used for:

[0127] Based on the first image structure information and the second image structure information, identifying and obtaining the posture information of the target object;

[0128] Based on the human body motion trajectory and the posture information, a current human body posture of the target object is determined.

[0129] It should be noted that the specific implementation of each unit in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0130] The embodiment of the present application provides a human posture recognition device, wherein an image acquisition unit acquires multiple two-dimensional images containing a target object; a trajectory determination unit determines a human motion trajectory of the target object based on image feature information of at least two two-dimensional images in the multiple two-dimensional images; an information acquisition unit acquires first image structure information of a first image and second image structure information of a second image in the multiple two-dimensional images; and a posture recognition unit recognizes the current human posture of the target object based on the human motion trajectory, the first image structure information, and the second image structure information. The recognition of human posture in three-dimensional space through multiple two-dimensional images is realized, and the accuracy of the recognition result is improved.

[0131] An embodiment of the present application also provides an electronic device. The technical solution provided in this embodiment mainly realizes the analysis of multiple frames of two-dimensional images, recognizes human body postures in three-dimensional space, and improves the accuracy of recognition results.

[0132] Specifically, the electronic device in this embodiment may include the following structure:

[0133] A memory, used to store applications and data generated by the operation of the applications;

[0134] A processor, configured to execute the application program to implement:

[0135] Acquire multiple frames of two-dimensional images containing the target object;

[0136] Determining a human body motion trajectory of the target object based on image feature information of at least two frames of two-dimensional images among the multiple frames of two-dimensional images;

[0137] Acquire first image structure information of a first image and second image structure information of a second image in the multiple frames of two-dimensional images, wherein the number of image frames between the first image and the second image meets a specific condition;

[0138] Based on the human body motion trajectory, the first image structure information and the second image structure information, a current human body posture of the target object is identified.

[0139] It should be noted that the specific implementation of the processor in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0140] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0141] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0142] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0143] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A human body posture recognition method, comprising: Acquire multiple frames of two-dimensional images containing the target object; Determining a human body motion trajectory of the target object based on image feature information of at least two frames of two-dimensional images among the multiple frames of two-dimensional images; Acquire first image structure information of a first image and second image structure information of a second image in the multiple frames of two-dimensional images, wherein the number of image frames between the first image and the second image meets a specific condition; Based on the human body motion trajectory, the first image structure information and the second image structure information, the current human body posture of the target object is identified; wherein the image depth information is obtained by analyzing the first image structure information and the second image structure information; The step of acquiring first image structure information of a first image and second image structure information of a second image in the multiple frames of two-dimensional images includes: Obtaining a first face recognition frame and a first group of key points of a first image in the multiple frames of two-dimensional images; Determine the first face recognition frame and a first structural relationship formed by the first group of key points as first image structural information; Acquire a second face recognition frame and a second set of key points of a second image in the multiple frames of two-dimensional images; The first face recognition frame and a second structural relationship formed by the second group of key points are determined as second image structure information.

2. The method according to claim 1, further comprising: Acquire a reference image containing the target object; Based on the current human body posture and the target posture corresponding to the reference image, it is detected whether the current human body posture matches the target posture to obtain a detection result.

3. The method according to claim 2, further comprising: If the detection result meets the prompt condition, a prompt message is generated, and the prompt message is used to prompt the target object to adjust the current human body posture to the target posture.

4. The method according to claim 2, wherein acquiring a reference image containing the target object comprises: Determine a captured image of the target object in a target posture as a reference image; or, generating a screening condition corresponding to the target object being in a target posture; Based on the screening condition, a reference image is determined in the multiple frames of two-dimensional images.

5. The method according to claim 1, wherein determining the human body motion trajectory of the target object based on the image feature information of at least two frames of two-dimensional images in the multiple frames of two-dimensional images comprises: Performing face recognition on at least two frames of the multiple two-dimensional images to determine face motion trajectory information; Performing key point recognition on at least two frames of the two-dimensional images in the multiple frames of the two-dimensional images to determine key point motion trajectory information, wherein the key points include at least face key points and body key points; Based on the face motion trajectory information and the key point motion trajectory information, the body motion trajectory of the target object is determined.

6. The method according to claim 5, wherein performing face recognition on at least two of the multiple two-dimensional images to determine face motion trajectory information comprises: Performing face recognition on at least two frames of the multiple two-dimensional images to obtain size and position information of a face recognition frame of each recognized image; Determining face motion trajectory information based on the size and position information of the face recognition frame of each recognized image; The step of performing key point recognition on at least two frames of two-dimensional images among the multiple frames of two-dimensional images to determine key point motion trajectory information includes: Performing key point recognition on at least two frames of the multiple two-dimensional images to obtain position information of key points of each recognized image; Based on the position information of the key points of each identified image, the key point motion trajectory information is determined.

7. The method according to claim 5, wherein determining the human body motion trajectory of the target object based on the face motion trajectory information and the key point motion trajectory information comprises: Inputting the face motion trajectory information and the key point motion trajectory information into a trajectory recognition model to obtain a plurality of human body motion sub-trajectories and a confidence level corresponding to each human body motion sub-trajectory; Based on the confidence level corresponding to each human motion sub-trajectory, a human motion trajectory of the target object is determined from a plurality of human motion sub-trajectories.

8. The method according to claim 1, wherein the step of identifying the current human body posture of the target object based on the human body motion trajectory, the first image structure information, and the second image structure information comprises: Based on the first image structure information and the second image structure information, identifying and obtaining the posture information of the target object; Based on the human body motion trajectory and the posture information, a current human body posture of the target object is determined.

9. A human body posture recognition device, comprising: An image acquisition unit, used for acquiring multiple frames of two-dimensional images containing a target object; A trajectory determination unit, configured to determine a human body motion trajectory of the target object based on image feature information of at least two frames of two-dimensional images among the multiple frames of two-dimensional images; An information acquisition unit, configured to acquire first image structure information of a first image and second image structure information of a second image in the plurality of two-dimensional image frames, wherein the number of image frames between the first image and the second image meets a specific condition; a posture recognition unit, configured to recognize a current human posture of the target object based on the human motion trajectory, the first image structure information, and the second image structure information; wherein image depth information is obtained by analyzing the first image structure information and the second image structure information; The information acquisition unit acquires the first image structure information of the first image and the second image structure information of the second image in the multiple frames of two-dimensional images, specifically for: Obtaining a first face recognition frame and a first group of key points of a first image in the multiple frames of two-dimensional images; Determine the first face recognition frame and a first structural relationship formed by the first group of key points as first image structural information; Acquire a second face recognition frame and a second set of key points of a second image in the multiple frames of two-dimensional images; The first face recognition frame and a second structural relationship formed by the second group of key points are determined as second image structure information.

Images