Low-calculation-amount method and system for simulating human body actions by robot
By identifying the key poses in the human posture video and processing only these frames, the problem of excessive calculation and power consumption during robot imitation is solved, and a low-computation robot simulates human movements is achieved, which significantly reduces computing power and power consumption. .
Patent Information
- Application Number
- CN202411055370.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-08-01
- Publication Date
- 2024-11-15
AI Technical Summary
The prior art leads to problems of excessive calculation amount, power consumption and time delay when a robot imitates independent, discontinuous posture or action.
By obtaining human posture videos, identifying key points and reference points, calculating angular velocity and angular acceleration, filtering out frames containing key postures, processing only these frames to adjust the robot posture, using low-pass filtering and tertiary processing to reduce the amount of calculation.
It significantly reduces computing power, power consumption and time delay, and improves the efficiency and accuracy of robot imitation.
Smart Images

Figure CN118952167A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of robotics technology, and in particular to a method and system for simulating human motions by a robot with low computational complexity. Background Art
[0002] Motion imitation is an important function of robots to interact with people. In some fields, robots imitate a series of independent, non-continuous postures or movements. For example, in museums, science and technology exhibitions or product displays, robots can show specific postures or movements to attract the audience's attention or demonstrate a specific technology or concept; in sports training, robots can imitate specific sports movements to help athletes learn and practice technical movements. In amusement parks or interactive experiences, robots can imitate specific movements to interact with tourists and provide entertainment experiences.
[0003] However, for the imitation of independent, non-continuous postures or movements, the existing technology is that the robot uses a camera to shoot a video of a person's movement, identifies the human body's movements from each frame of the video, and then uses the kinematic equation to calculate the instructions to be executed when the robot completes the same movement. However, the video has a fixed frame rate. The frame rate of smooth video requires no less than 29 frames per second, and the frame rate of high-definition video even reaches 60 frames per second. Each frame of the image contains an intermediate state of the person's movement process. Processing all the frames in the video will result in a large amount of calculation, power consumption and delay. Summary of the invention
[0004] Based on this, in order to solve the above technical problems, a method and system for a robot to simulate human movements with low computational complexity is provided to solve the problems of large amount of computation, power consumption and delay caused by robot imitation in the prior art.
[0005] In a first aspect, a method for a robot with low computational load to simulate human motions, the method comprising:
[0006] Obtain a human posture video, identify the human joint information in each frame, and determine the reference points and reference lines in each frame in a preset manner;
[0007] Obtain the connecting lines of multiple key points and reference points in each frame of human posture image respectively, and calculate the angles between the connecting lines of multiple key points and reference points and the reference line respectively, which are recorded as the angles of the key points;
[0008] Calculate the difference between the angle of each key point in each frame of human posture image and the angle of the corresponding key point in the previous frame of human posture image, record it as the angular velocity of the key point, and obtain the angular velocity of all key points in each frame of human posture image in the human posture video;
[0009] The difference between the angular velocity of each key point in each frame and the angular velocity of the corresponding key point in the previous frame is calculated respectively, recorded as the angular acceleration of the key point, and the angular acceleration of all key points in each frame of the human posture image in the human posture video is obtained;
[0010] A human posture image in which the angular velocities of all key points in a human posture video are 0 and the angular acceleration of at least one key point is not 0 is taken as a frame containing a key posture; or a still video segment in a human posture video is determined, and a human posture image in the middle of the still video segment is taken as a frame containing a key posture; the still video segment is obtained as a continuous plurality of frames of human posture images in which the angular velocities of all key points are 0 and the angular acceleration of at least one key point changes from non-0 to 0 and then to non-0;
[0011] The posture in the human posture image corresponding to the frame of the key posture is obtained, and the robot posture is adjusted.
[0012] In the above scheme, optionally, the preset method includes: taking the center point of the quadrilateral formed by the left shoulder, the right shoulder, the left hip and the right hip as the reference point, and taking the line connecting the midpoints of the two shoulders and the midpoints of the two hips as the reference line.
[0013] In the above scheme, optionally, the key points include: left hand, right hand, left foot, right foot.
[0014] In the above scheme, optionally, after obtaining the angular velocity of all key points in each frame of the human posture image in the human posture video, the method further includes: performing low-pass filtering on the angular velocity sequence of each key point.
[0015] In the above scheme, optionally, after obtaining the angular acceleration of all key points in each frame of the human posture image in the human posture video, the step further includes: performing low-pass filtering and ternary processing on the angular acceleration series of each key point.
[0016] In a second aspect, a system for simulating human motions by a robot with low computational load is provided, the system comprising:
[0017] After obtaining the angular velocity of all key points in each frame of the human posture image in the human posture video, the method further includes: performing low-pass filtering on the angular velocity sequence of each key point.
[0018] In the above scheme, optionally, the preset method includes: taking the center point of the quadrilateral formed by the left shoulder, the right shoulder, the left hip and the right hip as the reference point, and taking the line connecting the midpoints of the two shoulders and the midpoints of the two hips as the reference line.
[0019] In the above scheme, optionally, the key points include: left hand, right hand, left foot, right foot.
[0020] In a third aspect, a computer device comprises a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the steps of a method for a low-computational robot to simulate human motion as described in the first aspect are implemented.
[0021] In a fourth aspect, a computer program product includes a computer program / instruction, which, when executed by a processor, implements the steps of a method for a low-computational robot to simulate human motions as described in the first aspect.
[0022] This application has at least the following beneficial effects:
[0023] The present application constructs reference points and reference lines for each frame of human posture image in a human posture video, obtains the angle between the line connecting each key point and the reference point and the reference line in each frame of human posture image, and calculates the angular velocity of all key points in each frame of human posture image in the human posture video; at the same time, calculates the angular acceleration of all key points in each frame of human posture image in the human posture video; obtains a frame of human posture image in which the angular velocity of all key points is 0 and the angular acceleration of at least one of the key points is not 0, which corresponds to the frame containing the key posture; and also obtains a still video clip in the human posture video, and uses the obtained still video clip The human body posture image in the middle position in the time sequence is taken as the frame containing the key posture; that is, the image with angular velocity of 0 and angular acceleration not 0 is taken as the frame of switching between movement and stillness, and all frames with angular velocity of 0 between adjacent frames of switching between movement and stillness are taken as static segments; the frames of switching between movement and stillness can be selected as the frames of key posture, or the frames in the middle of the time sequence in the static segment can be selected as the frames of key posture, and the imitation is performed only by calculating the human body posture images corresponding to the frames of key posture, because the calculation amount of determining the frames of key posture is much less than the calculation amount of calculating the postures of all frames, so only processing the frames containing these static postures will greatly reduce computing power, power consumption and delay. BRIEF DESCRIPTION OF THE DRAWINGS
[0024] Figure 1 A flowchart of a method for a robot to simulate human motion with low computational complexity is provided for one embodiment of the present application. DETAILED DESCRIPTION
[0025] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.
[0026] Human motion can be broken down into the arrangement of static postures in time sequence, and the motion between adjacent static postures is continuous. The robot's motion imitation only needs to reproduce the static postures of the human motion process and the motion time between adjacent static postures in time sequence. Processing only the frames containing these static postures will greatly reduce computing power, power consumption and latency. This application refers to these static postures as key postures and proposes a method for detecting key postures in motion videos.
[0027] In one embodiment, Figure 1 As shown, a method for a robot to simulate human motion with low computational effort is provided, the method comprising:
[0028] Step S1: obtaining a human posture video, identifying the human joint information in each frame image, and determining the reference point and reference line in each frame image in a preset manner;
[0029] Step S2: respectively obtaining the connecting lines of multiple key points and the reference points in each frame of the human body posture image, and respectively calculating the angles between the connecting lines of the multiple key points and the reference points and the reference line, which are recorded as the angles of the key points;
[0030] Step S3: Calculate the difference between the angle of each key point in each frame of the human posture image and the angle of the corresponding key point in the previous frame of the human posture image, record it as the angular velocity of the key point, and obtain the angular velocity of all key points in each frame of the human posture image in the human posture video;
[0031] Step S4: Calculate the difference between the angular velocity of each key point in each frame and the angular velocity of the corresponding key point in the previous frame, record it as the angular acceleration of the key point, and obtain the angular acceleration of all key points in each frame of the human posture image in the human posture video;
[0032] Step S5: taking a human posture image in which the angular velocities of all key points in the human posture video are 0 and the angular acceleration of at least one key point is not 0 as a frame containing the key posture; or judging a still video segment in the human posture video, taking a human posture image in the middle position of the still video segment as a frame containing the key posture; obtaining the still video segment as a continuous plurality of frames of human posture images in which the angular velocities of all key points are 0 and the angular acceleration of at least one key point changes from non-zero to zero and then to non-zero;
[0033] Step S6: obtaining the posture in the human posture image corresponding to the frame of the key posture, and adjusting the robot posture.
[0034] Specifically, the present application implements a method for a robot to simulate human motion with low computational effort, which specifically includes the following steps:
[0035] 1. Identify the human body posture from each frame of the video and obtain the coordinates of the joints of the left shoulder, left hand, right shoulder, right hand, left hip, left foot, right hip, right foot, etc. The center point of the quadrilateral formed by the right shoulder, right shoulder, left hip, and right hip is used as the reference point, and the line connecting the midpoints of the two shoulders and the midpoints of the two hips is used as the reference line.
[0036] 2. Calculate the angles between the line connecting the left hand and the reference point, the line connecting the right hand and the reference point, the line connecting the left foot and the reference point, and the line connecting the right foot and the reference point and the reference line, and record them as ∠A, ∠B, ∠C, and ∠D respectively.
[0037] 3. For two adjacent frames, subtract ∠A, ∠B, ∠C, and ∠D of the previous frame from ∠A, ∠B, ∠C, and ∠D of the latter frame, and record the differences as DA, DB, DC, and DD. These four differences can be regarded as the angular velocity of the movement of the extremities of the human body.
[0038] 4. Perform low-pass filtering on the DA, DB, DC, and DD sequences of the entire video to remove high-frequency angular velocity noise caused by factors such as camera shake during shooting and inaccurate human posture recognition.
[0039] 5. For two adjacent frames, subtract DA, DB, DC, DD of the previous frame from DA, DB, DC, DD of the latter frame, and record the differences as SA, SB, SC, SD. These four differences can be regarded as the angular acceleration of the movement of the extremities of the human body.
[0040] 6. Perform low-pass filtering on the entire SA, SB, SC, and SD sequences to remove high-frequency components, and then perform ternary processing to divide SA, SB, SC, and SD into three situations: greater than 0, equal to 0, and less than 0.
[0041] 7. All frames with one or more angular accelerations greater than 0 and all angular velocities equal to 0 can be selected as frames containing key gestures, or one or more frames with angular accelerations less than 0 and all angular velocities equal to 0 can be selected as frames containing key gestures. After a person completes a key gesture during movement, there will be a short pause, which is shown in the video as a segment that maintains a static key gesture. Therefore, a frame in the static segment can also be selected as a frame of the key gesture. This static video segment has the following two characteristics: one or more angular accelerations change from non-0 to 0 and then to non-0; all angular velocities are equal to 0. In order to increase the accuracy of key gesture detection, the above two characteristics can be used to determine the static video segment, and then the frame in the middle of the segment in time sequence can be used as the frame containing the key gesture.
[0042] In the above-mentioned method of simulating human motion by a robot with low computational complexity, the present application constructs reference points and reference lines for each frame of human posture image in a human posture video, obtains the angle between the line connecting each key point and the reference point and the reference line in each frame of human posture image, and calculates the angular velocity of all key points in each frame of human posture image in the human posture video; at the same time, calculates the angular acceleration of all key points in each frame of human posture image in the human posture video; obtains a frame of human posture image in which the angular velocity of all key points is 0 and the angular acceleration of at least one of the key points is not 0, which corresponds to the frame containing the key posture; and still video clips in the human posture video can also be obtained. , the human body posture image in the middle position of the acquired static video clip is taken as the frame containing the key posture; that is, the image with angular velocity of 0 and angular acceleration of not 0 is taken as the frame of motion-static switching, and all frames with angular velocity of 0 between adjacent motion-static switching frames are taken as static clips; the frames of motion-static switching can be selected as the frames of key posture, or the frames in the middle of the static clip can be selected as the frames of key posture, and only the human body posture images corresponding to the frames of key posture are calculated for imitation, because the calculation amount of determining the frames of key posture is much less than the calculation amount of calculating the postures of all frames, so only processing the frames containing these static postures will greatly reduce computing power, power consumption and delay.
[0043] In one embodiment, the preset method includes: taking the center point of a quadrilateral formed by the left shoulder, the right shoulder, the left hip, and the right hip as a reference point, and taking the line connecting the midpoints of the two shoulders and the midpoints of the two hips as a reference line.
[0044] In one embodiment, the key points include: left hand, right hand, left foot, and right foot.
[0045] In one embodiment, after obtaining the angular velocity of all key points in each frame of the human posture image in the human posture video, the method further includes: performing low-pass filtering on the angular velocity sequence of each key point.
[0046] In one embodiment, after obtaining the angular accelerations of all key points in each frame of the human posture image in the human posture video, the step further includes: performing low-pass filtering and ternary processing on the angular acceleration series of each key point.
[0047] In one embodiment, a system for simulating human motion by a robot with low computational load is provided, the system comprising:
[0048] Reference point and reference line determination module: used to obtain human posture video, identify human joint point information in each frame image, and determine the reference point and reference line in each frame image in a preset manner;
[0049] Key point angle calculation module: used to obtain the line connecting each key point and the reference point in each frame of human posture image, and calculate the angle between the line connecting each key point and the reference point and the reference line, which is recorded as the angle of the key point;
[0050] Angular velocity calculation module of key points: used to calculate the difference between the angle of each key point in each frame of human posture image and the angle of the corresponding key point in the previous frame of human posture image, recorded as the angular velocity of the key point, and obtain the angular velocity of all key points in each frame of human posture image in the human posture video;
[0051] Angular acceleration calculation module of key points: used to calculate the difference between the angular velocity of each key point in each frame and the angular velocity of the corresponding key point in the previous frame, recorded as the angular acceleration of the key point, and obtain the angular acceleration of all key points in each frame of the human posture image in the human posture video;
[0052] The key posture frame determination module is used to take a human posture image in which the angular velocity of all key points in a human posture video is 0 and the angular acceleration of at least one key point is not 0 as a frame containing a key posture; or determine a still video segment in a human posture video, and take a human posture image in the middle of the still video segment as a frame containing a key posture; obtain the still video segment as a continuous multiple-frame human posture image in which the angular velocity of all key points is 0 and the angular acceleration of at least one key point changes from non-0 to 0 and then to non-0;
[0053] Robot simulation module: used to obtain the posture in the human posture image corresponding to the frame of the key posture and adjust the robot posture.
[0054] In one embodiment, the preset method includes: taking the center point of a quadrilateral formed by the left shoulder, the right shoulder, the left hip, and the right hip as a reference point, and taking the line connecting the midpoints of the two shoulders and the midpoints of the two hips as a reference line.
[0055] In one embodiment, the key points include: left hand, right hand, left foot, and right foot.
[0056] The specific definition of a system for simulating human motion by a robot with low computational load can be found in the above definition of a method for simulating human motion by a robot with low computational load, which will not be repeated here. Each module in the above system for simulating human motion by a robot with low computational load can be implemented in whole or in part by software, hardware and a combination thereof. Each of the above modules can be embedded in or independent of a processor in a computer device in the form of hardware, or can be stored in a memory in a computer device in the form of software, so that the processor can call and execute the operations corresponding to each of the above modules.
[0057] In one embodiment, a computer device is provided, which may be a server, and includes a processor, a memory, and a network interface connected via a system bus. The processor of the computer device is used to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operation of the operating system and the computer program in the non-volatile storage medium. The network interface of the computer device is used to communicate with an external terminal via a network connection. When the computer program is executed by the processor, the above-mentioned method of simulating human body motion by a robot is implemented.
[0058] In one embodiment, a computer program product is also provided, including a computer program / instruction, which involves all or part of the process in the above embodiment method.
[0059] Those of ordinary skill in the art can understand that all or part of the processes in the above-mentioned embodiment methods can be completed by instructing the relevant hardware through a computer program, and the computer program can be stored in a non-volatile computer-readable storage medium. When the computer program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, any reference to memory, storage, database or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory or optical memory, etc. Volatile memory can include random access memory (RAM) or external cache memory. As an illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM).
[0060] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0061] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the invention patent. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application. Therefore, the protection scope of the patent of the present application shall be subject to the attached claims.
Claims
1. A method for simulating human motion by a robot with low computational complexity, characterized in that: The method comprises: Obtain a human posture video, identify the human joint information in each frame, and determine the reference points and reference lines in each frame in a preset manner; Obtain the connecting lines of multiple key points and reference points in each frame of human posture image respectively, and calculate the angles between the connecting lines of multiple key points and reference points and the reference line respectively, which are recorded as the angles of the key points; Calculate the difference between the angle of each key point in each frame of human posture image and the angle of the corresponding key point in the previous frame of human posture image, record it as the angular velocity of the key point, and obtain the angular velocity of all key points in each frame of human posture image in the human posture video; The difference between the angular velocity of each key point in each frame and the angular velocity of the corresponding key point in the previous frame is calculated respectively, recorded as the angular acceleration of the key point, and the angular acceleration of all key points in each frame of the human posture image in the human posture video is obtained; A human posture image in which the angular velocities of all key points in a human posture video are 0 and the angular acceleration of at least one key point is not 0 is taken as a frame containing a key posture; or a still video segment in a human posture video is determined, and a human posture image in the middle of the still video segment is taken as a frame containing a key posture; the still video segment is obtained as a continuous plurality of frames of human posture images in which the angular velocities of all key points are 0 and the angular acceleration of at least one key point changes from non-0 to 0 and then to non-0; The posture in the human posture image corresponding to the frame of the key posture is obtained, and the posture of the robot is adjusted.
2. The method for simulating human motion by a robot with low computational complexity according to claim 1, characterized in that: The preset method includes: taking the center point of a quadrilateral formed by the left shoulder, the right shoulder, the left hip, and the right hip as a reference point, and taking the line connecting the midpoints of the two shoulders and the midpoints of the two hips as a reference line.
3. The method for simulating human motion by a robot with low computational complexity according to claim 1, characterized in that: The key points include: left hand, right hand, left foot, and right foot.
4. The method for simulating human motion by a robot with low computational complexity according to claim 1, characterized in that: After obtaining the angular velocity of all key points in each frame of the human posture image in the human posture video, the method further includes: performing low-pass filtering on the angular velocity sequence of each key point.
5. The method for simulating human motion by a robot with low computational complexity according to claim 1, characterized in that: After obtaining the angular accelerations of all key points in each frame of the human posture image in the human posture video, the method further includes: performing low-pass filtering and ternary processing on the angular acceleration series of each key point.
6. A system for simulating human motion by a robot with low computational load, characterized in that: The system comprises: Reference point and reference line determination module: used to obtain human posture video, identify human joint point information in each frame image, and determine the reference point and reference line in each frame image in a preset manner; Key point angle calculation module: used to obtain the line connecting each key point and the reference point in each frame of human posture image, and calculate the angle between the line connecting each key point and the reference point and the reference line, which is recorded as the angle of the key point; Angular velocity calculation module of key points: used to calculate the difference between the angle of each key point in each frame of human posture image and the angle of the corresponding key point in the previous frame of human posture image, recorded as the angular velocity of the key point, and obtain the angular velocity of all key points in each frame of human posture image in the human posture video; Angular acceleration calculation module of key points: used to calculate the difference between the angular velocity of each key point in each frame and the angular velocity of the corresponding key point in the previous frame, recorded as the angular acceleration of the key point, and obtain the angular acceleration of all key points in each frame of the human posture image in the human posture video; The key posture frame determination module is used to take a human posture image in which the angular velocity of all key points in a human posture video is 0 and the angular acceleration of at least one key point is not 0 as a frame containing a key posture; or determine a still video segment in a human posture video, and take a human posture image in the middle of the still video segment as a frame containing a key posture; obtain the still video segment as a continuous multiple-frame human posture image in which the angular velocity of all key points is 0 and the angular acceleration of at least one key point changes from non-0 to 0 and then to non-0; Robot simulation module: used to obtain the posture in the human posture image corresponding to the frame of the key posture and adjust the robot posture.
7. The low-computation robot human motion simulation system according to claim 6, characterized in that: The preset method includes: taking the center point of a quadrilateral formed by the left shoulder, the right shoulder, the left hip, and the right hip as a reference point, and taking the line connecting the midpoints of the two shoulders and the midpoints of the two hips as a reference line.
8. The low-computation robot simulation human motion system according to claim 6, characterized in that: The key points include: left hand, right hand, left foot, and right foot.
9. A computer device comprising a memory and a processor, wherein the memory stores a computer program, wherein: When the processor executes the computer program, the steps of the method according to any one of claims 1 to 5 are implemented.
10. A computer program product comprising a computer program / instructions, characterized in that When the computer program / instructions are executed by a processor, the steps of the method according to any one of claims 1 to 5 are implemented.
