Motion recognition method and device, electronic device and computer-readable storage medium
By using a two-dimensional human body key point recognition method in image processing, combining the character orientation and knee key point position, the problem of large data processing volume in the prior art is solved, and the efficiency of turning direction recognition is improved.
Patent Information
- Application Number
- CN202310143399.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2023-02-09
- Publication Date
- 2024-11-19
- Estimated Expiration
- 2043-02-09
AI Technical Summary
The prior art requires obtaining three-dimensional information when identifying the direction of a person's turning, resulting in large data processing volume and low efficiency.
By obtaining the two-dimensional human key points of the character in the image to be processed (such as hips, knees, ankle key points), determine the reference line and the direction of the character, combine the positional relationship between the knee key points and the reference line, determine the turning direction of the character, and reduce the direction of the person's turn Data processing volume.
The turning direction recognition based on two-dimensional human body key points is realized, which reduces the data processing volume and improves the recognition efficiency.
Smart Images

Figure CN117746491B_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer vision technology, and in particular to a motion recognition method and device, an electronic device, and a computer-readable storage medium. Background Art
[0002] With the development of image processing technology, its applications are becoming increasingly widespread, including identifying the turning direction of a person in an image. Current methods typically identify the turning direction of a person based on the person's three-dimensional information in the image. This means that identifying the turning direction requires obtaining the person's three-dimensional information in the image. Acquiring this information typically requires a large amount of data processing, which in turn leads to a high data processing load for current methods of identifying the turning direction. Summary of the Invention
[0003] The present application provides a motion recognition method and device, an electronic device, and a computer-readable storage medium.
[0004] In a first aspect, a method for motion recognition is provided, the method comprising:
[0005] Acquire three or more key points of a person in an image to be processed, wherein the person is in a turned state, and the three or more key points of the person include: a hip key point, a knee key point, and an ankle key point, and the hip key point, the knee key point, and the ankle key point are all located on the same side of the person;
[0006] Determine a straight line passing through the hip key point and the ankle key point to obtain a reference line;
[0007] Obtaining the orientation of the person in the image to be processed;
[0008] The turning direction of the character is determined according to the orientation, the positional relationship between the knee key point and the reference line.
[0009] In combination with any embodiment of the present application, determining the turning direction of the character based on the orientation, the positional relationship between the knee key point and the reference line includes:
[0010] Determining, based on a positional relationship between the knee key point and the reference line, that the knee key point is located on a first side of the reference line;
[0011] In a case where the orientation is facing the camera, determining the turning direction of the person is turning toward the first side;
[0012] In the case that the orientation is facing away from the camera, the turning direction of the person is determined to be turning toward the second side, and the first side and the second side are two sides of the person.
[0013] In combination with any embodiment of the present application, the three or more key points of the human body include a left hip key point and a right hip key point, and obtaining the orientation of the person in the image to be processed includes:
[0014] The orientation of the person in the image to be processed is obtained according to the positional relationship between the left hip key point and the right hip key point.
[0015] In combination with any embodiment of the present application, obtaining the orientation of the person in the image to be processed based on the positional relationship between the left hip key point and the right hip key point includes:
[0016] When the horizontal coordinate of the left hip key point in the pixel coordinate system of the image to be processed is smaller than the horizontal coordinate of the right hip key point in the pixel coordinate system, determining that the orientation of the person is facing away from the camera;
[0017] When the horizontal coordinate of the left hip key point in the pixel coordinate system is greater than the horizontal coordinate of the right hip key point in the pixel coordinate system, the orientation of the person is determined to be facing the camera.
[0018] In conjunction with any embodiment of the present application, the step of obtaining three or more key points of a person in the image to be processed includes:
[0019] Obtaining a video to be processed including the person, and four or more body key points of the person in each video frame in the video to be processed, wherein the four or more body key points include the hip key point, the knee key point, and the ankle key point, the hip key point includes a left hip key point and a right hip key point, and the knee key point and the ankle key point are both located on the same side of the person;
[0020] Determine the distance between the left hip key point and the right hip key point in the video to be processed;
[0021] Determining a target video segment in which the distance first decreases and then increases from the video to be processed;
[0022] A frame of image is determined from the target video segment as the image to be processed, and three or more human body key points of the person in the image to be processed are obtained based on the four or more human body key points of the person in the image to be processed.
[0023] In conjunction with any embodiment of the present application, the four or more key points of the human body further include a shoulder key point, an elbow key point, and a wrist key point, and the shoulder key point, the elbow key point, and the wrist key point are all located on the same side of the character;
[0024] After obtaining the video to be processed and four or more key points of the human body of the person in each video frame in the video to be processed, the method further includes:
[0025] Determining an elbow bending angle in each video frame according to the shoulder key point, the elbow key point, and the wrist key point in each video frame;
[0026] If there is a waving video segment that meets a preset condition in the video to be processed, it is determined that the video to be processed includes a waving action, and the preset condition includes: an elbow bending angle is greater than or equal to a first threshold.
[0027] In combination with any embodiment of the present application, the preset condition also includes: the duration is greater than or equal to a second threshold.
[0028] In combination with any embodiment of the present application, when there is a waving video segment that meets a preset condition in the video to be processed, before determining that the video to be processed includes a waving action, the method further includes:
[0029] Determining a shoulder joint flexion angle in each video frame according to the hip key point, the shoulder key point, and the elbow key point in each video frame;
[0030] The preset condition further includes: a difference between the shoulder joint flexion angle in the starting frame and the shoulder joint flexion angle in the ending frame is greater than or equal to a third threshold.
[0031] In combination with any embodiment of the present application, the preset condition also includes: the shoulder joint flexion angle continues to increase, and the increased acceleration is greater than or equal to a fourth threshold value. The preset condition may also include: the shoulder joint flexion angle continues to decrease, and the decreased acceleration is greater than or equal to the fourth threshold value.
[0032] In conjunction with any embodiment of the present application, after obtaining the video to be processed, obtaining four or more key points of the human body of the person in each video frame in the video to be processed includes:
[0033] Sending the video to be processed and a human key point detection instruction for the video to be processed to the server;
[0034] Receive the four or more human body key points of the person in each video frame in the video to be processed returned by the server.
[0035] In combination with any embodiment of the present application, after determining the turning direction of the character, the method further includes:
[0036] Adding a special effect matching the turning direction to the image to be processed.
[0037] In a second aspect, a motion recognition device is provided, the motion recognition device comprising:
[0038] an acquisition unit, configured to acquire three or more key points of a person in an image to be processed, wherein the person is in a turned state, and the three or more key points of the person include: a hip key point, a knee key point, and an ankle key point, wherein the hip key point, the knee key point, and the ankle key point are all located on the same side of the person;
[0039] a determining unit, configured to determine a straight line passing through the hip key point and the ankle key point to obtain a reference line;
[0040] The acquisition unit is further configured to acquire the orientation of the person in the image to be processed;
[0041] The determining unit is further configured to determine the turning direction of the character based on the orientation, the positional relationship between the knee key point and the reference line.
[0042] In combination with any embodiment of the present application, the determining unit is configured to:
[0043] Determining, based on a positional relationship between the knee key point and the reference line, that the knee key point is located on a first side of the reference line;
[0044] In a case where the orientation is facing the camera, determining the turning direction of the person is turning toward the first side;
[0045] In the case that the orientation is facing away from the camera, the turning direction of the person is determined to be turning toward the second side, and the first side and the second side are two sides of the person.
[0046] In conjunction with any embodiment of the present application, the three or more key points of the human body include a left hip key point and a right hip key point, and the acquisition unit is used to:
[0047] The orientation of the person in the image to be processed is obtained according to the positional relationship between the left hip key point and the right hip key point.
[0048] In combination with any embodiment of the present application, the acquiring unit is configured to:
[0049] When the horizontal coordinate of the left hip key point in the pixel coordinate system of the image to be processed is smaller than the horizontal coordinate of the right hip key point in the pixel coordinate system, determining that the orientation of the person is facing away from the camera;
[0050] When the horizontal coordinate of the left hip key point in the pixel coordinate system is greater than the horizontal coordinate of the right hip key point in the pixel coordinate system, the orientation of the person is determined to be facing the camera.
[0051] In combination with any embodiment of the present application, the acquiring unit is configured to:
[0052] Obtaining a video to be processed including the person, and four or more body key points of the person in each video frame in the video to be processed, wherein the four or more body key points include the hip key point, the knee key point, and the ankle key point, the hip key point includes a left hip key point and a right hip key point, and the knee key point and the ankle key point are both located on the same side of the person;
[0053] Determine the distance between the left hip key point and the right hip key point in the video to be processed;
[0054] Determining a target video segment in which the distance first decreases and then increases from the video to be processed;
[0055] A frame of image is determined from the target video segment as the image to be processed, and three or more human body key points of the person in the image to be processed are obtained based on the four or more human body key points of the person in the image to be processed.
[0056] In conjunction with any embodiment of the present application, the four or more key points of the human body further include a shoulder key point, an elbow key point, and a wrist key point, and the shoulder key point, the elbow key point, and the wrist key point are all located on the same side of the character;
[0057] The determining unit is further configured to:
[0058] Determining an elbow bending angle in each video frame according to the shoulder key point, the elbow key point, and the wrist key point in each video frame;
[0059] If there is a waving video segment that meets a preset condition in the video to be processed, it is determined that the video to be processed includes a waving action, and the preset condition includes: an elbow bending angle is greater than or equal to a first threshold.
[0060] In combination with any embodiment of the present application, the preset condition also includes: the duration is greater than or equal to a second threshold.
[0061] In combination with any embodiment of the present application, the determining unit is further configured to:
[0062] Determining a shoulder joint flexion angle in each video frame according to the hip key point, the shoulder key point, and the elbow key point in each video frame;
[0063] The preset condition further includes: a difference between the shoulder joint flexion angle in the starting frame and the shoulder joint flexion angle in the ending frame is greater than or equal to a third threshold.
[0064] In combination with any embodiment of the present application, the preset condition also includes: the shoulder joint flexion angle continues to increase, and the increased acceleration is greater than or equal to a fourth threshold value. The preset condition may also include: the shoulder joint flexion angle continues to decrease, and the decreased acceleration is greater than or equal to the fourth threshold value.
[0065] In combination with any embodiment of the present application, the motion recognition device further includes: a sending unit, configured to send the video to be processed and a human key point detection instruction for the video to be processed to a server;
[0066] The determining unit is configured to receive the four or more body key points of the person in each video frame in the video to be processed returned by the server.
[0067] In combination with any embodiment of the present application, the action recognition device further includes: an adding unit, configured to add a special effect matching the turning direction to the image to be processed.
[0068] In a third aspect, an electronic device is provided, characterized in that it includes: a processor and a memory, the memory is used to store computer program code, the computer program code includes computer instructions, and when the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.
[0069] In a fourth aspect, another electronic device is provided, comprising: a processor, a sending device, an input device, an output device and a memory, wherein the memory is used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device executes the method as described in the first aspect above and any possible implementation method thereof.
[0070] In a fifth aspect, a computer-readable storage medium is provided, in which a computer program is stored. The computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method as described in the first aspect above and any possible implementation method thereof.
[0071] In a sixth aspect, a computer program product is provided, which includes a computer program or instructions, and when the computer program or instructions are run on a computer, the computer is enabled to execute the method of the above-mentioned first aspect and any possible implementation thereof.
[0072] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present application.
[0073] In an embodiment of the present application, an image processing device first obtains three or more key points of the human body of a person in an image to be processed, wherein the three or more key points of the human body include a hip key point, a knee key point, and an ankle key point located on the same side of the person. A straight line between the hip key point and the ankle key point is then determined to obtain a reference line. Furthermore, when the orientation of the person in the image to be processed is obtained, the direction of the person's turn can be determined based on the orientation, the positional relationship between the knee key point, and the reference line. This allows the person's turn direction to be determined based on the two-dimensional key points of the human body and the person's orientation, thereby reducing the amount of data processing required to determine the turn direction. BRIEF DESCRIPTION OF THE DRAWINGS
[0074] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the background technology, the drawings required for use in the embodiments of the present application or the background technology will be described below.
[0075] The drawings herein are incorporated into and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and, together with the specification, are used to illustrate the technical solutions of the present application.
[0076] Figure 1 A flowchart of an action recognition method provided in an embodiment of the present application;
[0077] Figure 2 A schematic diagram of an image to be processed provided in an embodiment of the present application;
[0078] Figure 3 A schematic diagram of another image to be processed provided in an embodiment of the present application;
[0079] Figure 4 A schematic diagram of another image to be processed provided in an embodiment of the present application;
[0080] Figure 5 A schematic diagram of the architecture of a motion recognition system provided in an embodiment of the present application;
[0081] Figure 6 A schematic diagram of a processing flow of a motion recognition system provided in an embodiment of the present application;
[0082] Figure 7A schematic diagram of the process of detecting key points of a human body in an image by a server provided in an embodiment of the present application;
[0083] Figure 8 A schematic diagram of key points of the human body provided in an embodiment of the present application;
[0084] Figure 9 A schematic diagram of the structure of a motion recognition device provided in an embodiment of the present application;
[0085] Figure 10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0086] In order to enable those skilled in the art to better understand the present invention, the following will clearly and completely describe the technical solutions in the embodiments of the present invention in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0087] The terms "first," "second," and the like in the specification and claims of this application and the accompanying drawings are used to distinguish between different objects, not to describe a particular order. Furthermore, the terms "including," "having," and any variations thereof, are intended to cover non-exclusive inclusions. For example, a process, method, system, product, or apparatus comprising a series of steps or elements is not limited to the listed steps or elements but may optionally include steps or elements not listed, or may optionally include other steps or elements inherent to the process, method, product, or apparatus.
[0088] References herein to "embodiments" mean that a particular feature, structure, or characteristic described in connection with the embodiments may be included in at least one embodiment of the present application. The appearance of this phrase in various places in the specification does not necessarily refer to the same embodiment, nor does it constitute an independent or alternative embodiment that is mutually exclusive of other embodiments. It is understood, both explicitly and implicitly, by those skilled in the art that the embodiments described herein may be combined with other embodiments.
[0089] The execution subject of the embodiments of the present application is a motion recognition device, wherein the motion recognition device can be any electronic device that can execute the technical solution disclosed in the method embodiments of the present application. Optionally, the motion recognition device can be one of the following: a mobile phone, a computer, a tablet computer, or a wearable smart device.
[0090] It should be understood that the method embodiment of the present application can also be implemented by a processor executing computer program code. The following describes the embodiment of the present application in conjunction with the drawings in the embodiment of the present application. Figure 1 , Figure 1 This is a flow chart of an action recognition method provided in an embodiment of the present application.
[0091] 101. Obtain three or more key points of the human body of a person in the image to be processed.
[0092] In an embodiment of the present application, the person in the image to be processed is in a turning state, that is, the person in the image to be processed is in the process of turning. For example, the image to be processed is a frame of a video of a person turning around, and the person in the image to be processed is in the turning state. For another example, the image to be processed is an image captured of the person in the process of turning around.
[0093] In an embodiment of the present application, the three or more human body key points include: a hip key point, a knee key point, and an ankle key point, wherein the hip key point, the knee key point, and the ankle key point are all located on the same side of the character. For example, the hip key point is the left hip key point, the knee key point is the left knee key point, and the ankle key point is the left ankle key point. In this case, the hip key point, the knee key point, and the ankle key point are all located on the left side of the character. For another example, the hip key point is the right hip key point, the knee key point is the right knee key point, and the ankle key point is the right ankle key point. In this case, the hip key point, the knee key point, and the ankle key point are all located on the right side of the character.
[0094] In one implementation for obtaining three or more key points of a person in an image to be processed, a motion recognition device receives input from a user via an input component. The input component includes a keyboard, a mouse, a touch screen, a touchpad, and an audio input device.
[0095] In another implementation of obtaining three or more key points of a person in an image to be processed, the motion recognition device receives three or more key points of a person in an image to be processed from a terminal. Optionally, the terminal can be any of the following: a mobile phone, a computer, a tablet computer, or a server.
[0096] In another implementation method of obtaining three or more human body key points of a person in an image to be processed, the motion recognition device, when obtaining the image to be processed, performs human body key point detection processing on the image to be processed to obtain three or more human body key points of the person in the image to be processed.
[0097] 102. Determine a straight line passing through the above-mentioned hip key point and the above-mentioned ankle key point to obtain a reference line.
[0098] In this embodiment of the present application, the three or more human key points include the position of any one human key point in the image to be processed. That is, based on the three or more human key points, the position of the hip key point, the position of the knee key point, and the position of the ankle key point in the image to be processed can be determined. Therefore, based on the position of the hip key point and the position of the ankle key point in the image to be processed, the motion recognition device can determine a straight line passing through the hip key point and the ankle key point to obtain a reference line.
[0099] 103. Obtain the orientation of the person in the image to be processed.
[0100] In an embodiment of the present application, the orientation of the person in the image includes facing the camera, facing away from the camera, and facing sideways to the camera, wherein facing the camera means that the person is facing the front of the imaging device when the imaging device obtains the image by photographing the person, facing away from the camera means that the person is facing the back of the imaging device when the imaging device obtains the image by photographing the person, and facing sideways to the camera means that the person is facing the side of the imaging device when the imaging device obtains the image by photographing the person.
[0101] The orientation of the person in the image to be processed includes one of the following: facing the camera, facing away from the camera, and facing sideways to the camera. Specifically, the orientation of the person in the image to be processed is facing the camera, that is, in the process of the imaging device obtaining the image to be processed by photographing the person, the person is facing the front of the imaging device. At this time, the image to be processed shows the front of the person. The orientation of the person in the image to be processed is facing away from the camera, that is, in the process of the imaging device obtaining the image to be processed by photographing the person, the person is facing the back of the imaging device. At this time, the image to be processed shows the back of the person. The orientation of the person in the image to be processed is facing sideways to the camera, that is, in the process of the imaging device obtaining the image to be processed by photographing the person, the person is facing the side of the imaging device. At this time, the image to be processed shows the side of the person.
[0102] In an implementation method of obtaining the orientation of a person in an image to be processed, a motion recognition device receives the orientation of the person in the image to be processed input by a user through an input component.
[0103] In another implementation of obtaining the orientation of a person in the image to be processed, the motion recognition device receives the orientation of the person in the image to be processed sent by the terminal.
[0104] In another implementation method of obtaining the orientation of a person in an image to be processed, the motion recognition device, when obtaining the image to be processed, performs orientation detection processing on the person in the image to be processed to obtain the orientation of the person in the image, wherein the orientation detection processing is used to detect the orientation of the person in the image.
[0105] 104. Determine the turning direction of the character based on the orientation, the positional relationship between the knee key point and the reference line.
[0106] When a person turns under their own power, their knees bend in the direction of the turn. Specifically, when a person turns to the left, their knees bend to the left, and when they turn to the right, their knees bend to the right. When the knees bend, the knee, ankle, and hip points are not collinear; that is, the reference line does not pass through the knee point. Specifically, when the knee bends to the left, the knee point is to the left of the reference line, and when the knee bends to the right, the knee point is to the right of the reference line.
[0107] Therefore, the motion recognition device can determine which side the person is turning to based on the positional relationship between the knee key point and the reference line. Specifically, if the knee key point is to the left of the reference line, the person is determined to be turning to the left, i.e., the turning direction is leftward. If the knee key point is to the right of the reference line, the person is determined to be turning to the right, i.e., the turning direction is rightward.
[0108] It should be understood that the left and right sides described above both correspond to positions in the image to be processed. In one possible implementation, the left side is the side closer to the negative direction of the horizontal axis of the pixel coordinate system of the image to be processed, and the right side is the side closer to the positive direction of the horizontal axis of the pixel coordinate system of the image to be processed. That is, the reference line divides the image to be processed into two parts, and these two parts are located on either side of the reference line. The side closer to the positive direction of the horizontal axis is the right side, and the side closer to the negative direction of the horizontal axis is the left side.
[0109] In another possible implementation, the right side is the side closer to the negative direction of the horizontal axis of the pixel coordinate system of the image to be processed, and the left side is the side closer to the positive direction of the horizontal axis of the pixel coordinate system of the image to be processed. That is, the reference line divides the image to be processed into two parts, and these two parts are located on either side of the reference line. The side closer to the positive direction of the horizontal axis is the left side, and the side closer to the negative direction of the horizontal axis is the right side.
[0110] Based on the positional relationship between the knee key point and the reference line, the motion recognition device determines whether the knee key point is located on the side of the reference line that is closer to the positive direction of the horizontal axis, or on the side of the reference line that is closer to the negative direction of the horizontal axis. However, it cannot determine whether the knee key point is on the left or right side of the reference line. Therefore, the motion recognition device needs to further determine whether the side closer to the positive direction of the horizontal axis is the left or right side.
[0111] Because the person's orientation in the image to be processed directly determines whether the side closest to the positive direction of the horizontal axis is the left or right side, the motion recognition device can determine whether the side closest to the positive direction of the horizontal axis is the left or right side based on the person's orientation in the image to be processed. Specifically, if the person in the image to be processed is facing the camera and the person's left side is the side closest to the positive direction of the horizontal axis, if the person in the image to be processed is facing away from the camera, then the side closest to the positive direction of the horizontal axis is the person's right side.
[0112] For example, in Figure 2 In the image to be processed shown, the pixel coordinate system is XOY, where O is the origin of the coordinate system, the X axis is the horizontal axis, and the Y axis is the vertical axis. If the person in the image to be processed is facing the camera, the positive direction of the horizontal axis of the pixel coordinate system is to the left of the person. In this pixel coordinate system, if the person in the image to be processed is facing away from the camera, the negative direction of the horizontal axis of the pixel coordinate system is to the right of the person.
[0113] Therefore, when the motion recognition device determines whether the side close to the positive direction of the horizontal axis is the left or right side based on the orientation of the person in the image to be processed, it can further determine the direction of the person's turn based on the positional relationship between the knee key point and the reference line.
[0114] In an embodiment of the present application, an image processing device first obtains three or more key points of the human body of a person in an image to be processed, wherein the three or more key points of the human body include a hip key point, a knee key point, and an ankle key point located on the same side of the person. A straight line between the hip key point and the ankle key point is then determined to obtain a reference line. Furthermore, when the orientation of the person in the image to be processed is obtained, the direction of the person's turn can be determined based on the orientation, the positional relationship between the knee key point, and the reference line. This allows the person's turn direction to be determined based on the two-dimensional key points of the human body and the person's orientation, thereby reducing the amount of data processing required to determine the turn direction.
[0115] As an optional implementation, the motion recognition device performs the following steps during step 104:
[0116] 201. According to the positional relationship between the knee key point and the reference line, determine that the knee key point is located on a first side of the reference line.
[0117] In an embodiment of the present application, the first side is one of the two sides of the reference line. For example, the first side is the side of the two sides of the reference line close to the positive direction of the horizontal axis. For another example, the first side is the side of the two sides of the reference line close to the negative direction of the horizontal axis.
[0118] In one possible implementation, the motion recognition device determines the equation of a reference line based on the hip key point and the ankle key point. The knee key point is substituted into the equation of the reference line to obtain a reference value. When the reference value is a positive number, the knee key point is determined to be located on the side of the reference line that is closer to the positive direction of the horizontal axis, that is, the first side is the side of the reference line that is closer to the positive direction of the horizontal axis. When the reference value is a negative number, the knee key point is determined to be located on the side of the reference line that is closer to the negative direction of the horizontal axis, that is, the first side is the side of the reference line that is closer to the negative direction of the horizontal axis.
[0119] 202. When the above-mentioned orientation is facing the camera, determine that the turning direction of the above-mentioned person is turning toward the above-mentioned first side.
[0120] In one possible implementation, the first side is the side close to the positive direction of the horizontal axis, and the character's turning direction is to turn toward the positive direction of the horizontal axis. Figure 2 In the image to be processed shown, the side close to the positive direction of the horizontal axis is the left side of the character, so the direction of the character's turn is to turn towards the left side of the character.
[0121] In one possible implementation, the first side is the side close to the negative direction of the horizontal axis, and the character's turning direction is to turn toward the negative direction of the horizontal axis. Figure 2 In the image to be processed shown, the side close to the negative direction of the horizontal axis is the right side of the character, so the direction of the character's turn is to turn towards the right side of itself.
[0122] 203. When the orientation is facing away from the camera, determine that the turning direction of the person is toward the second side, and the first side and the second side are two sides of the reference line.
[0123] In the embodiment of the present application, the first side and the second side are two sides of the reference line. For example, the first side is the side close to the positive direction of the horizontal axis, and the second side is the side close to the negative direction of the horizontal axis.
[0124] In one possible implementation, the first side is the side close to the negative direction of the horizontal axis, and the second side is the side close to the positive direction of the horizontal axis, so the character's turning direction is to turn towards the positive direction of the horizontal axis. Figure 3In the image to be processed, the pixel coordinate system is XOY, where O is the origin, the X axis is the horizontal axis, and the Y axis is the vertical axis. The side closest to the positive horizontal axis is the right side of the character, so the character's turn direction is toward their right side.
[0125] In one possible implementation, the first side is the side close to the positive direction of the horizontal axis, and the second side is the side close to the negative direction of the horizontal axis, so the character's turning direction is to turn towards the negative direction of the horizontal axis. Figure 3 In the image to be processed shown, the side close to the negative direction of the horizontal axis is the left side of the character, so the direction of the character's turn is to turn towards the left side of the character.
[0126] In this embodiment, the motion recognition device determines that the knee key point is located on a first side of the reference line based on the positional relationship between the knee key point and the reference line. If the person is facing the camera, the person's turning direction is determined to be toward the first side. If the person is facing away from the camera, the person's turning direction is determined to be toward the second side, where the first side and the second side are two sides of the person.
[0127] As an optional implementation manner, the three or more human body key points include a left hip key point and a right hip key point, and the action recognition device performs the following steps during the execution of step 103:
[0128] 301. Obtain the orientation of the person in the image to be processed based on the positional relationship between the left hip key point and the right hip key point.
[0129] In one possible implementation, the pixel coordinate system of the image to be processed is as follows: Figure 2 As shown in the pixel coordinate system in , at this time, when the orientation of the person in the image to be processed is facing the camera, the right side of the person is closer to the positive direction of the horizontal axis than the left side of the person, and at this time the horizontal coordinate of the left hip key point is greater than the horizontal coordinate of the right hip key point. Therefore, when the positional relationship between the left hip key point and the right hip key point is that the horizontal coordinate of the left hip key point is less than the horizontal coordinate of the right hip key point, the action recognition device determines that the orientation of the person in the image to be processed is facing away from the camera. When the positional relationship between the left hip key point and the right hip key point is that the horizontal coordinate of the left hip key point is greater than the horizontal coordinate of the right hip key point, the orientation of the person in the image to be processed is facing the camera. When the positional relationship between the left hip key point and the right hip key point is that the horizontal coordinate of the left hip key point is equal to the horizontal coordinate of the right hip key point, the orientation of the person in the image to be processed is determined to be sideways to the camera.
[0130] In another possible implementation, Figure 4In the image to be processed shown, the pixel coordinate system is a coordinate system XOY, wherein O is the origin of the coordinate system, the X axis is the horizontal axis, and the Y axis is the vertical axis. At this time, when the orientation of the person in the image to be processed is facing the camera, the left side of the person is closer to the positive direction of the horizontal axis than the right side of the person, and at this time the horizontal coordinate of the left hip key point is smaller than the horizontal coordinate of the right hip key point. Therefore, when the positional relationship between the left hip key point and the right hip key point is such that the horizontal coordinate of the left hip key point is smaller than the horizontal coordinate of the right hip key point, the action recognition device determines that the orientation of the person in the image to be processed is facing the camera. When the positional relationship between the left hip key point and the right hip key point is such that the horizontal coordinate of the left hip key point is larger than the horizontal coordinate of the right hip key point, the orientation of the person in the image to be processed is determined to be facing away from the camera. When the positional relationship between the left hip key point and the right hip key point is such that the horizontal coordinate of the left hip key point is equal to the horizontal coordinate of the right hip key point, the orientation of the person in the image to be processed is determined to be sideways toward the camera.
[0131] In this embodiment, the motion recognition device can obtain the orientation of the person in the image to be processed based on the positional relationship between the left hip key point and the right hip key point.
[0132] As an optional implementation, the motion recognition device performs the following steps during the execution of step 101:
[0133] 401. Obtain a video to be processed that includes the person, and four or more body key points of the person in each video frame of the video to be processed.
[0134] In an embodiment of the present application, the four or more human body key points include a hip key point, a knee key point and an ankle key point, wherein the hip key point includes a left hip key point and a right hip key point, and the knee key point and the ankle key point are both located on the same side of the character.
[0135] The motion recognition device can obtain the video to be processed and four or more body key points of the characters in each frame of the video to be processed by executing step 401. It should be understood that each frame of the video to be processed includes the above-mentioned characters.
[0136] In one implementation of obtaining a video to be processed, a motion recognition device receives the video to be processed input by a user through an input component.
[0137] In another implementation of obtaining the video to be processed, the motion recognition device receives the video to be processed sent by the terminal.
[0138] In another implementation of obtaining the video to be processed, there is a communication connection between the motion recognition device and the camera. The motion recognition device obtains the video to be processed by shooting the dancer while dancing with the camera through the communication connection.
[0139] In an implementation method for obtaining four or more human body key points of a person in each video frame in a video to be processed, a motion recognition device receives four or more human body key points of a person in each video frame in a video to be processed input by a user through an input component.
[0140] In another implementation of obtaining four or more key points of a person in each frame of a video to be processed, the motion recognition device receives four or more key points of a person in each frame of the video to be processed from a terminal. Optionally, the terminal can be any of the following: a mobile phone, a computer, a tablet computer, or a server.
[0141] In an embodiment of the present application, the step of the action recognition device obtaining the video to be processed and the step of the action recognition device obtaining four or more human body key points of the person in each video frame in the video to be processed can be executed separately or simultaneously, and this application does not limit this.
[0142] 402. Determine the distance between the left hip key point and the right hip key point in the video to be processed.
[0143] The action recognition device can determine the distance between the left hip key point and the right hip key point in each video frame according to the left hip key point and the right hip key point in each video frame in the video to be processed, and then obtain the distance between the left hip key point and the right hip key point in the video to be processed.
[0144] 403. Determine, from the video to be processed, a target video segment in which the distance first decreases and then increases.
[0145] During the process of a person turning around, the distance between the left hip key point and the right hip key point will first decrease and then increase. Therefore, the action recognition device can determine the video segment including the person turning around from the video to be processed based on the distance between the left hip key point and the right hip key point. Specifically, the action recognition device determines the target video segment in which the distance first decreases and then increases from the video to be processed as the video segment in which the person turns around. For example, the action recognition device determines that from the 10th frame to the 50th frame of the video to be processed, the distance between the left hip key point and the right hip key point is in a process of continuously decreasing, and then from the 51st frame to the 100th frame, the distance between the left hip key point and the right hip key point is in a process of continuously increasing. At this time, the target video segment is the 10th frame to the 100th frame.
[0146] 404 : Determine a frame of image from the target video segment as the image to be processed, and obtain three or more human body key points of the person in the image to be processed based on the four or more human body key points of the person in the image to be processed.
[0147] Since the target video segment is a segment where a person turns around, the person in any frame of the target video segment is in a turning position. Therefore, the action recognition device determines a frame from the target video segment as the image to be processed, and can obtain three or more key points of the person in the image to be processed based on the four or more key points of the person in the image to be processed.
[0148] In this embodiment, the action recognition device obtains a video to be processed and four or more key points of a person in each frame of the video to be processed. The device then determines the distance between the left hip key point and the right hip key point in the video to be processed. The device then determines a target video segment from the video to be processed in which the distance first decreases and then increases, thereby obtaining a video segment of the person turning around. This allows a frame of image to be determined from the target video segment as the image to be processed, and based on the four or more key points of the person in the image to be processed, three or more key points of the person in the image to be processed can be obtained.
[0149] Based on this implementation, the embodiment of the present application also provides an implementation method for determining the turning direction of the character in the target video segment. After the action recognition device determines the target video segment from the video to be processed, it can determine the turning direction of the character in each frame in the target video segment based on the implementation method for determining the turning direction of the character in the image to be processed provided above. Then the turning direction with the largest proportion in the target video segment is used as the turning direction of the character in the target video segment. For example, the target video segment includes image a, image b, and image c, wherein the turning direction of the character in image a is turning toward the negative direction of the horizontal axis, and the turning direction of the character in image b and the turning direction of the character in image c are both turning toward the positive direction of the horizontal axis. At this time, the turning direction with the largest proportion in the target video segment is turning toward the positive direction of the horizontal axis, so the turning direction of the character in the target video segment is turning toward the positive direction of the horizontal axis.
[0150] As an optional embodiment, the four or more human body key points further include a shoulder key point, an elbow key point, and a wrist key point, wherein the shoulder key point, the elbow key point, and the wrist key point are all located on the same side of the person. For example, the shoulder key point is the left shoulder key point, the elbow key point is the left elbow key point, and the wrist key point is the left wrist key point. In this case, the shoulder key point, the elbow key point, and the wrist key point are all located on the left side of the person. For another example, the shoulder key point is the right shoulder key point, the elbow key point is the right elbow key point, and the wrist key point is the right wrist key point. In this case, the shoulder key point, the elbow key point, and the wrist key point are all located on the right side of the person.
[0151] In this embodiment, after executing step 401, the motion recognition device further executes the following steps:
[0152] 501. Determine the elbow bending angle in each of the video frames according to the shoulder key point, the elbow key point, and the wrist key point in each of the video frames.
[0153] In one possible implementation, the motion recognition device determines a first vector pointing from the elbow keypoint to the shoulder keypoint, and determines a second vector pointing from the elbow keypoint to the wrist keypoint. The angle between the first and second vectors is determined as the elbow flexion angle. The motion recognition device can determine the elbow flexion angle in each video frame.
[0154] 502. When there is a waving video segment that meets a preset condition in the video to be processed, determine that the video to be processed includes a waving action.
[0155] When a person waves, their elbow is bent, i.e., at a certain angle. Therefore, the motion recognition device can use this elbow bending angle as a basis for determining whether the person is waving, i.e., whether the video to be processed includes a waving action. Therefore, the pre-set condition in this step includes the elbow bending angle being greater than or equal to a first threshold, i.e., the elbow bending angle in the waving video segment is greater than or equal to the first threshold.
[0156] In one possible implementation, the first threshold is 135 degrees, that is, when the bending angle of the person's elbow is greater than or equal to 135 degrees, the person is in a waving state, and it is determined that the video to be processed includes a waving action.
[0157] In this embodiment, when the motion recognition device determines that there is a waving video segment in the video to be processed, it can determine that the video to be processed includes a waving motion.
[0158] As an optional implementation, the preset condition further includes that the duration is greater than or equal to a second threshold.
[0159] In this embodiment, the elbow flexion angle of each frame in the waving video segment is greater than or equal to a first threshold, and the duration of the waving video segment is greater than or equal to a second threshold. When a waving video segment is present in the video to be processed, the motion recognition device determines that the video to be processed includes a waving action. For example, if the first threshold is 135 degrees and the second threshold is 2 seconds, then the waving video segment is a video segment with a duration greater than or equal to 2 seconds, and the elbow flexion angle in each frame is greater than or equal to 135 degrees.
[0160] In this embodiment, when there is a waving video segment in the video to be processed in which the elbow bending angle is greater than or equal to a first threshold and the duration of the waving video segment is greater than or equal to a second threshold, the motion recognition device determines that the video to be processed includes a waving action, which can reduce the recognition error of the waving action and thus improve the recognition accuracy of the waving action.
[0161] As an optional implementation, the motion recognition device further performs the following steps before performing step 502:
[0162] 601. Determine the shoulder joint flexion angle in each of the video frames according to the hip key point, the shoulder key point, and the elbow key point in each of the video frames.
[0163] In one possible implementation, the motion recognition device determines a third vector pointing from the shoulder keypoint to the elbow keypoint, and determines a fourth vector pointing from the shoulder keypoint to the hip keypoint. The angle between the third and fourth vectors is determined as the shoulder flexion angle. The motion recognition device can determine the shoulder flexion angle in each video frame.
[0164] In this embodiment, the preset conditions also include: the difference between the shoulder flexion angle in the starting frame and the shoulder flexion angle in the ending frame is greater than or equal to a third threshold. Since the waving action will cause the shoulder flexion angle to change, specifically, there will be a large difference between the shoulder flexion angle at the starting moment and the shoulder flexion angle at the ending moment. In the embodiment of the present application, the action recognition device determines whether the difference in the shoulder flexion angle is large or small based on the third threshold. Specifically, if the difference in the shoulder flexion angle is greater than or equal to the third threshold, it means that the difference in the shoulder flexion angle is large, and if the difference in the shoulder flexion angle is less than the third threshold, it means that the difference in the shoulder flexion angle is small. Therefore, the preset conditions also include that the difference between the shoulder flexion angle in the starting frame and the shoulder flexion angle in the ending frame is greater than or equal to the third threshold, that is, in the waving video segment, the difference between the shoulder flexion angle in the starting frame and the shoulder flexion angle in the ending frame is greater than or equal to the third threshold.
[0165] In this embodiment, the action recognition device determines the shoulder joint flexion angle in each video frame based on the above-mentioned hip key points, shoulder key points, and elbow key points in each video frame, and can determine whether there is a waving video segment that meets the preset conditions in the video to be processed. Further, when it is determined that there is a waving video segment in the video to be processed, it is determined that there is a waving video in the video to be processed, thereby improving the accuracy of waving action recognition.
[0166] As an optional embodiment, the preset condition also includes: the shoulder joint flexion angle continues to increase, and the increased acceleration is greater than or equal to the fourth threshold value. The preset condition may also include: the shoulder joint flexion angle continues to decrease, and the decreased acceleration is greater than or equal to the fourth threshold value.
[0167] The waving action will cause the shoulder joint flexion angle to be in a continuous changing process. Specifically, it will either cause the shoulder joint flexion angle to be in a state of continuous increase or to be in a state of continuous decrease. Moreover, there is a large change acceleration in the continuous change process of the shoulder joint flexion angle. For example, the shoulder joint flexion angle is in a state of continuous increase, and the increase acceleration is greater than or equal to 10 degrees per second, that is, the shoulder joint flexion angle increases by at least 10 degrees per second. For another example, the shoulder joint flexion angle is in a state of continuous decrease, and the decrease acceleration is greater than or equal to 10 degrees per second, that is, the shoulder joint flexion angle decreases by at least 10 degrees per second.
[0168] Therefore, the motion recognition device further includes, under the preset conditions, that the shoulder joint flexion angle continues to increase, and the increasing acceleration is greater than or equal to the fourth threshold, or further includes that the shoulder joint flexion angle continues to decrease. In this case, the waving video segment determined by the motion recognition device from the video to be processed is a video segment in which the elbow flexion angle is greater than or equal to the first threshold, the duration of the waving is greater than or equal to the second threshold, the difference between the shoulder joint flexion angle in the starting frame and the shoulder joint flexion angle in the ending frame is greater than or equal to the third threshold, the shoulder joint flexion angle continues to increase, and the increasing acceleration is greater than or equal to the fourth threshold.
[0169] The motion recognition device determines a waving video segment from the video to be processed, or a video segment in which the elbow bending angle is greater than or equal to a first threshold, and the duration is greater than or equal to a second threshold, and the difference between the shoulder joint flexion angle in the starting frame and the shoulder joint flexion angle in the ending frame is greater than or equal to a third threshold, and the shoulder joint flexion angle continues to decrease, and the decreasing acceleration is greater than or equal to a fourth threshold.
[0170] It should be understood that the fourth threshold can be acceleration in time units, for example, increasing by 10 degrees per second, or decreasing by 10 degrees per second. The fourth threshold can also be acceleration in frames units, for example, increasing by 1 degree per frame, or decreasing by 1 degree per frame.
[0171] In this embodiment, the motion recognition device determines whether there is a waving video segment in the video to be processed based on the preset conditions, when the preset conditions also include: the shoulder joint flexion angle continues to increase, and the increased acceleration is greater than or equal to the fourth threshold, or also include: the shoulder joint flexion angle continues to decrease, and the decreased acceleration is greater than or equal to the fourth threshold. This can improve the judgment accuracy and thus improve the recognition accuracy of the waving action.
[0172] As an optional implementation, after obtaining the video to be processed, the action recognition device obtains four or more human body key points of the person in each video frame in the video to be processed by performing the following steps:
[0173] 701. Sending the video to be processed and a human body key point detection instruction for the video to be processed to the server.
[0174] In the embodiment of the present application, there is a communication connection between the server and the motion recognition device, wherein the server includes a server. In one possible implementation, Figure 5 The following is a schematic diagram of the architecture of the action recognition system. Figure 5 As shown, the motion recognition system 1 includes a motion recognition device 11 and a server 12 , and there is a communication connection between the motion recognition device 11 and the server 12 .
[0175] See also Figure 6 , Figure 6 The following is a schematic diagram of the processing flow of the action recognition system. Figure 6 As shown, after the image is input to the server, the server (i.e., the above-mentioned server 12) uses a 2D human skeleton key point detection module (i.e., a two-dimensional human skeleton key point detection module) to perform human skeleton key point detection processing on the image to obtain human skeleton points (i.e., human key points). Figure 7 The figure shows the process flow of the server detecting key points of the image. Figure 7 As shown, the server uses the human body detection module to perform human body detection on the image, determines the human body area from the image and intercepts the human body area, and then uses the key point detection module to perform human body key point detection processing on the human body area to obtain human body key points.
[0176] Optional, Figure 8 The figure shows a schematic diagram of the key points of the human body obtained through the key point detection process. Figure 8 As shown, the key points of the human body include the key point of the top of the head, the key point of the neck, the key point of the left shoulder, the key point of the right shoulder, the key point of the left upper arm, the key point of the right upper arm, the key point of the left elbow, the key point of the right elbow, the key point of the left forearm, the key point of the right forearm, the key point of the left wrist, the key point of the right wrist, the key point of the left hip, the key point of the right hip, the key point of the left thigh, the key point of the right thigh, the key point of the left knee, the key point of the right knee, the key point of the left ankle, and the key point of the right ankle.
[0177] The server is connected to the client (i.e., the motion recognition device 11) via a communication connection, and the server can transmit data to the client via the communication connection. After obtaining the human skeleton points, the server transmits the human skeleton points to the client via the communication connection. The client processes the human skeleton points using the human hand-waving motion recognition module to identify whether the image includes hand-waving, and processes the human skeleton points using the human turn motion recognition module to identify whether the image includes turn. The implementation process of using the human hand-waving motion recognition module to identify whether the image includes hand-waving can be seen in the above-mentioned steps 501 to 502 and step 601, and the implementation process of using the human turn motion recognition module to identify whether the image includes hand-waving can be seen in the above-mentioned steps 101 to 104, steps 201 to 203, step 301, and steps 401 to 404.
[0178] The motion recognition device sends the video to be processed and the human key point detection instruction for the video to be processed to the server, so that the server performs human key point detection processing on each frame image in the video to be processed.
[0179] 702. Receive the four or more body key points of the person in each video frame in the video to be processed returned by the server.
[0180] In this embodiment, after obtaining the video to be processed, the action recognition device sends the video to be processed and a human key point detection instruction for the video to be processed to the server, so that the server performs human key point detection processing on each frame of the video to be processed, and then receives four or more human key points of the person in each video frame of the video to be processed returned by the server. In this way, when the server performs human key point detection processing on each video frame of the video to be processed, four or more human key points of the person in each video frame of the video to be processed can be obtained, thereby reducing the data processing amount of the action recognition device and improving the processing speed of the action recognition device.
[0181] As an optional implementation, after determining the direction of the person's turn, the motion recognition device further performs the following steps:
[0182] 801. Add a special effect matching the turning direction to the image to be processed.
[0183] In the embodiments of the present application, the special effect that matches the turning direction is a special effect that can represent the turning direction. In one possible implementation, the motion recognition device can add a special effect indicating the turning direction to the image to be processed. For example, the turning effect is in the form of a "dynamic sticker," where a motion trajectory similar to an elliptical arc is dynamically displayed near the waist of the character. This motion trajectory can be rendered and then presented.
[0184] In this embodiment, the action recognition device can increase the interest of the image to be processed by adding special effects that match the turning direction to the image to be processed.
[0185] Optionally, when the motion recognition device determines that the video to be processed includes a waving action, it adds special effects matching the waving action to the video to be processed, for example, adding a special effect of a roc spreading its wings to the waving video segment, wherein the special effect of a roc spreading its wings includes the effect obtained by rendering the motion trajectory of the wrist key points during the waving process.
[0186] Those skilled in the art will understand that in the above-mentioned method of the specific implementation method, the writing order of each step does not mean a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.
[0187] If the technical solution of this application involves personal information, the product that applies the technical solution of this application has clearly informed the personal information processing rules and obtained the individual's voluntary consent before processing personal information. If the technical solution of this application involves sensitive personal information, the product that applies the technical solution of this application has obtained the individual's separate consent before processing sensitive personal information, and at the same time meets the "explicit consent" requirement. For example, on personal information collection devices such as cameras, a clear and prominent sign is set to inform that the personal information collection scope has been entered and personal information will be collected. If the individual voluntarily enters the collection scope, it is deemed that they agree to the collection of their personal information; or on the personal information processing device, when the personal information processing rules are notified by obvious signs / information, the individual's authorization is obtained through pop-up information or by asking the individual to upload their personal information; among which, personal information processing may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the type of personal information processed.
[0188] The above describes in detail the method of the embodiment of the present application, and the following provides an apparatus of the embodiment of the present application.
[0189] See also Figure 9 , Figure 9 This is a structural diagram of a motion recognition device provided in an embodiment of the present application. The motion recognition device 2 includes: an acquisition unit 21, a determination unit 22, and optionally, the motion recognition device 2 further includes: a sending unit 23, and an adding unit 24. Specifically:
[0190] an acquisition unit 21 for acquiring three or more key points of a person in an image to be processed, wherein the person is in a turned state, and the three or more key points of the person include a hip key point, a knee key point, and an ankle key point, wherein the hip key point, the knee key point, and the ankle key point are all located on the same side of the person;
[0191] a determination unit 22, configured to determine a straight line passing through the hip key point and the ankle key point to obtain a reference line;
[0192] The acquisition unit 21 is further configured to acquire the orientation of the person in the image to be processed;
[0193] The determining unit 22 is further configured to determine the turning direction of the character according to the orientation and the positional relationship between the knee key point and the reference line.
[0194] In combination with any embodiment of the present application, the determining unit 22 is configured to:
[0195] Determining, based on a positional relationship between the knee key point and the reference line, that the knee key point is located on a first side of the reference line;
[0196] In a case where the orientation is facing the camera, determining the turning direction of the person is turning toward the first side;
[0197] In the case that the orientation is facing away from the camera, the turning direction of the person is determined to be turning toward the second side, and the first side and the second side are two sides of the person.
[0198] In combination with any embodiment of the present application, the three or more key points of the human body include a left hip key point and a right hip key point, and the acquisition unit 21 is used to:
[0199] The orientation of the person in the image to be processed is obtained according to the positional relationship between the left hip key point and the right hip key point.
[0200] In combination with any embodiment of the present application, the acquiring unit 21 is configured to:
[0201] When the horizontal coordinate of the left hip key point in the pixel coordinate system of the image to be processed is smaller than the horizontal coordinate of the right hip key point in the pixel coordinate system, determining that the orientation of the person is facing away from the camera;
[0202] When the horizontal coordinate of the left hip key point in the pixel coordinate system is greater than the horizontal coordinate of the right hip key point in the pixel coordinate system, the orientation of the person is determined to be facing the camera.
[0203] In combination with any embodiment of the present application, the acquiring unit 21 is configured to:
[0204] Obtaining a video to be processed including the person, and four or more body key points of the person in each video frame in the video to be processed, wherein the four or more body key points include the hip key point, the knee key point, and the ankle key point, the hip key point includes a left hip key point and a right hip key point, and the knee key point and the ankle key point are both located on the same side of the person;
[0205] Determine the distance between the left hip key point and the right hip key point in the video to be processed;
[0206] Determining a target video segment in which the distance first decreases and then increases from the video to be processed;
[0207] A frame of image is determined from the target video segment as the image to be processed, and three or more human body key points of the person in the image to be processed are obtained based on the four or more human body key points of the person in the image to be processed.
[0208] In conjunction with any embodiment of the present application, the four or more key points of the human body further include a shoulder key point, an elbow key point, and a wrist key point, and the shoulder key point, the elbow key point, and the wrist key point are all located on the same side of the character;
[0209] The determining unit 22 is further configured to:
[0210] Determining an elbow bending angle in each video frame according to the shoulder key point, the elbow key point, and the wrist key point in each video frame;
[0211] If there is a waving video segment that meets a preset condition in the video to be processed, it is determined that the video to be processed includes a waving action, and the preset condition includes: an elbow bending angle is greater than or equal to a first threshold.
[0212] In combination with any embodiment of the present application, the preset condition also includes: the duration is greater than or equal to a second threshold.
[0213] In combination with any embodiment of the present application, the determining unit 22 is further configured to:
[0214] The shoulder joint flexion angle in each video frame is determined based on the hip key point, the shoulder key point, and the elbow key point in each video frame; the preset condition also includes: the difference between the shoulder joint flexion angle in the starting frame and the shoulder joint flexion angle in the ending frame is greater than or equal to a third threshold.
[0215] In combination with any embodiment of the present application, the preset condition also includes: the shoulder joint flexion angle continues to increase, and the increased acceleration is greater than or equal to a fourth threshold value. The preset condition may also include: the shoulder joint flexion angle continues to decrease, and the decreased acceleration is greater than or equal to the fourth threshold value.
[0216] In combination with any embodiment of the present application, the action recognition device 2 further includes: a sending unit 23, configured to send the video to be processed and a human key point detection instruction for the video to be processed to the server;
[0217] The determining unit 22 is configured to receive the four or more body key points of the person in each video frame in the video to be processed returned by the server.
[0218] In combination with any embodiment of the present application, the action recognition device 2 further includes: an adding unit 24, configured to add a special effect matching the turning direction to the image to be processed.
[0219] In an embodiment of the present application, an image processing device first obtains three or more key points of the human body of a person in an image to be processed, wherein the three or more key points of the human body include a hip key point, a knee key point, and an ankle key point located on the same side of the person. A straight line between the hip key point and the ankle key point is then determined to obtain a reference line. Furthermore, when the orientation of the person in the image to be processed is obtained, the direction of the person's turn can be determined based on the orientation, the positional relationship between the knee key point, and the reference line. This allows the person's turn direction to be determined based on the two-dimensional key points of the human body and the person's orientation, thereby reducing the amount of data processing required to determine the turn direction.
[0220] In some embodiments, the functions or modules included in the device provided in the embodiments of the present application can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments. For the sake of brevity, it will not be repeated here.
[0221] Figure 10 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device 3 includes a processor 31 and a memory 32. Optionally, the electronic device 3 also includes an input device 33 and an output device 34. The processor 31, the memory 32, the input device 33 and the output device 34 are coupled via a connector, and the connector includes various interfaces, transmission lines or buses, etc., which are not limited in the embodiments of the present application. It should be understood that in each embodiment of the present application, coupling refers to mutual connection in a specific manner, including direct connection or indirect connection through other devices, for example, connection through various interfaces, transmission lines, buses, etc.
[0222] The processor 31 may include one or more processors, for example, one or more central processing units (CPUs). In the case where the processor is a CPU, the CPU may be a single-core CPU or a multi-core CPU. Alternatively, the processor 31 may be a processor group consisting of multiple CPUs, wherein the multiple processors are coupled to each other via one or more buses. Alternatively, the processor may also be other types of processors, etc., which are not limited in the embodiments of the present application.
[0223] The memory 32 can be used to store computer program instructions and various computer program codes, including the program code for executing the solution of the present application. Optionally, the memory includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or portable compact disc read-only memory (CD-ROM), which is used for related instructions and data.
[0224] The input device 33 is used to input data and / or signals, and the output device 34 is used to output data and / or signals. The input device 33 and the output device 34 can be independent devices or an integrated device.
[0225] It can be understood that in the embodiment of the present application, the memory 32 can be used not only to store relevant instructions, but also to store relevant data. The embodiment of the present application does not limit the specific data stored in the memory.
[0226] It is understandable that Figure 10 Only a simplified design of an electronic device is shown. In actual applications, the electronic device may further include other necessary components, including but not limited to any number of input / output devices, processors, memories, etc., and all electronic devices that can implement the embodiments of the present application are within the scope of protection of the present application.
[0227] Those skilled in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.
[0228] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices, and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here. Those skilled in the art will also clearly understand that the descriptions of the various embodiments of this application have different focuses. For the convenience and brevity of description, the same or similar parts may not be repeated in different embodiments. Therefore, for parts not described or not described in detail in a certain embodiment, reference can be made to the descriptions of other embodiments.
[0229] In the several embodiments provided in this application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are merely schematic. For example, the division of the units is merely a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0230] The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.
[0231] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
[0232] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted via the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, server or data center to another website, computer, server or data center via wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that can be accessed by a computer or a data storage device such as a server or data center that includes one or more available media integrated therein. The available medium may be a magnetic medium (eg, a floppy disk, a hard disk, a magnetic tape), an optical medium (eg, a digital versatile disc (DVD)), or a semiconductor medium (eg, a solid state disk (SSD)).
[0233] Those skilled in the art will appreciate that all or part of the processes in the above-described method embodiments can be implemented by a computer program instructing related hardware to perform the processes. The program can be stored in a computer-readable storage medium, and when executed, the program can include the processes in the above-described method embodiments. The aforementioned storage medium includes various media capable of storing program code, such as read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
Claims
1. A method for motion recognition, characterized in that: The method comprises: Acquire three or more key points of a person in the image to be processed, wherein the person is in a turned state, and the three or more key points of the person include: a hip key point, a knee key point, and an ankle key point, and the hip key point, the knee key point, and the ankle key point are all located on the same side of the person; Determine a straight line passing through the hip key point and the ankle key point to obtain a reference line; Obtaining the orientation of the person in the image to be processed; The turning direction of the character is determined according to the orientation, the positional relationship between the knee key point and the reference line.
2. The method according to claim 1, characterized in that: The step of determining the turning direction of the character according to the orientation, the positional relationship between the knee key point and the reference line includes: According to the positional relationship between the knee key point and the reference line, determining that the knee key point is located on a first side of the reference line; In a case where the orientation is facing the camera, determining the turning direction of the person is turning toward the first side; In the case that the orientation is facing away from the camera, the turning direction of the person is determined to be turning toward the second side, and the first side and the second side are two sides of the person.
3. The method according to claim 2, characterized in that The three or more key points of the human body include a left hip key point and a right hip key point, and obtaining the orientation of the person in the image to be processed includes: The orientation of the person in the image to be processed is obtained according to the positional relationship between the left hip key point and the right hip key point.
4. The method according to claim 3, characterized in that The obtaining the orientation of the person in the image to be processed according to the positional relationship between the left hip key point and the right hip key point includes: When the horizontal coordinate of the left hip key point in the pixel coordinate system of the image to be processed is smaller than the horizontal coordinate of the right hip key point in the pixel coordinate system, determining the orientation of the person as facing away from the camera; When the horizontal coordinate of the left hip key point in the pixel coordinate system is greater than the horizontal coordinate of the right hip key point in the pixel coordinate system, the orientation of the person is determined to be facing the camera.
5. The method according to any one of claims 1 to 4, characterized in that: The step of obtaining three or more key points of a person in the image to be processed includes: Acquire a video to be processed including the person, and four or more human body key points of the person in each video frame in the video to be processed, wherein the four or more human body key points include the hip key point, the knee key point, and the ankle key point, the hip key point includes a left hip key point and a right hip key point, and the knee key point and the ankle key point are both located on the same side of the person; Determine the distance between the left hip key point and the right hip key point in the video to be processed; Determine, from the video to be processed, a target video segment in which the distance first decreases and then increases; A frame of image is determined from the target video segment as the image to be processed, and three or more human body key points of the person in the image to be processed are obtained based on the four or more human body key points of the person in the image to be processed.
6. The method according to claim 5, characterized in that The four or more human body key points further include a shoulder key point, an elbow key point, and a wrist key point, and the shoulder key point, the elbow key point, and the wrist key point are all located on the same side of the character; After obtaining the video to be processed and four or more human body key points of the person in each video frame in the video to be processed, the method further includes: Determine the elbow bending angle in each video frame according to the shoulder key point, the elbow key point, and the wrist key point in each video frame; In the case that there is a waving video segment satisfying a preset condition in the video to be processed, it is determined that the video to be processed includes a waving action, and the preset condition includes: an elbow bending angle is greater than or equal to a first threshold.
7. The method according to claim 6, characterized in that The preset condition also includes: the duration is greater than or equal to a second threshold.
8. The method according to claim 7, characterized in that In the case where there is a waving video segment that meets a preset condition in the video to be processed, before determining that the video to be processed includes a waving action, the method further includes: Determine the shoulder joint flexion angle in each video frame according to the hip key point, the shoulder key point, and the elbow key point in each video frame; The preset condition also includes: a difference between the shoulder joint flexion angle in the start frame and the shoulder joint flexion angle in the end frame is greater than or equal to a third threshold.
9. The method according to claim 8, characterized in that The preset condition also includes: the shoulder joint flexion angle continues to increase, and the increasing acceleration is greater than or equal to the fourth threshold value. The preset condition may also include: the shoulder joint flexion angle continues to decrease, and the decreasing acceleration is greater than or equal to the fourth threshold value.
10. The method according to any one of claims 5 to 9, characterized in that After obtaining the video to be processed, obtaining four or more human body key points of the person in each video frame in the video to be processed includes: Sending the video to be processed and a human body key point detection instruction for the video to be processed to the server; Receive the four or more human body key points of the person in each video frame in the video to be processed returned by the server.
11. The method according to any one of claims 1 to 4, characterized in that: After determining the turning direction of the character, the method further includes: Adding a special effect matching the turning direction to the image to be processed.
12. A motion recognition device, characterized in that: The device comprises: An acquisition unit is used to acquire three or more key points of a person in the image to be processed, wherein the person is in a turned state, and the three or more key points of the person include: a hip key point, a knee key point, and an ankle key point, and the hip key point, the knee key point, and the ankle key point are all located on the same side of the person; A determination unit, used for determining a straight line passing through the hip key point and the ankle key point to obtain a reference line; The acquisition unit is further used to acquire the orientation of the person in the image to be processed; The determination unit is further used to determine the turning direction of the character according to the orientation, the positional relationship between the knee key point and the reference line.
13. An electronic device, characterized in that: include: A processor and a memory, wherein the memory is used to store computer program codes, wherein the computer program codes include computer instructions, and when the processor executes the computer instructions, the electronic device executes the method according to any one of claims 1 to 11.
14. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions. When the program instructions are executed by a processor, the processor is enabled to execute the method according to any one of claims 1 to 11.
Citation Information
Patent Citations
Method and device for detecting posture of human body
CN107358149A
Human body posture recognition method and device, equipment and storage medium
CN113989855A
Method and device for determining test action, equipment and medium
CN115299934A
Hand-raising recognition method and apparatus, electronic device, and storage medium
WO2022237481A1
