ROS2-based real-time human body posture estimation method, device and system

Abstract

The invention discloses a real-time human body posture estimation method, device and system based on ROS2. The method comprises the following steps: constructing a camera node and a processing node based on ROS2; the camera node issues image data to the real-time image topic in real time; the processing node subscribes to the topic and carries out human body existence judgment on the image frame by utilizing a lightweight human body detection model YOLO; and selectively calling a human body posture estimation algorithm according to a judgment result, identifying key skeleton points of a human body, and respectively issuing the image marked with the skeleton points and the coordinate data of the key points to a real-time image topic and a skeleton point topic. According to the method, the overall consumption of the system can be reduced under the condition that the computing power is limited, meanwhile, the method can serve as a functional plate to be integrally embedded into a larger robot application system, and the practicability and expansibility of the system are improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of computer vision and robotics, and more particularly, to a real-time human pose estimation method, device and system based on ROS2. BACKGROUND

[0002] With the rapid development of robotics, real-time and accurate perception and estimation of human pose become the key to realizing human-computer interaction. Deploying on mobile embedded devices, perception can be achieved through an external RGB camera, and the application of human-computer interaction can be expanded in scenarios with lower cost.

[0003] In the past research, high-precision human pose estimation algorithms (such as HybrIK) are computationally complex. If the estimation algorithm is called for each image frame, it will bring a large computational pressure to the embedded device, making it difficult to guarantee the real-time performance of the entire system, resulting in frame loss, inaccurate recognition, etc. Most research focuses on the accuracy of the algorithm itself, and does not fully consider its integration and application in a complete robot system. There is a lack of a modular system to convert algorithm capabilities into real-time perceptual information available to robots. SUMMARY

[0004] In order to solve the technical problems existing in the prior art, the present application provides a real-time human pose estimation method based on ROS2.

[0005] Specifically, the first aspect of the present application provides a real-time human pose estimation method based on ROS2, comprising: An RGB camera is externally connected to an embedded device, and a camera node and a processing node are constructed based on a robot operating system ROS2; The camera node acquires images in real time and publishes the acquired images to a real-time image topic; The processing node subscribes to the real-time image topic published by the camera node, acquires an image frame at the current time, and performs human detection on the image frame at the current time. Based on the detection result, a human pose estimation algorithm is selectively called to obtain a human pose estimation result.

[0006] In one embodiment, the human existence detection on the image frame at the current time comprises: A lightweight human detection model YOLO is used to detect humans in the image frame.

[0007] In one embodiment, based on the detection result, the human pose estimation algorithm is selectively called to obtain a human pose estimation result, comprising: If the detection result is no human body detected, the processing node processes the acquired current time image and publishes it to a processed image topic, wherein the processed image topic is used to transmit the image containing semantic information processed by the processing node; If the detection result is human body detected, a preset human body pose estimation algorithm is called to perform key skeleton point recognition on the image frame of the current time, to obtain a human body pose result, wherein the human body pose result includes an image with skeleton point annotation and skeleton point coordinates, and the image with skeleton point annotation is published to the processed image topic, and the skeleton point coordinates are published to a skeleton point topic, which is used to transmit structured data that can be directly understood by a standardized robot system.

[0008] In an embodiment, the method further comprises: Respectively setting the node message buffer size of the camera node and the processing node.

[0009] In an embodiment, before the camera node publishes the collected image to the real-time image topic, the method further comprises: Converting the collected OpenCV type image to an Image type.

[0010] In an embodiment, before the processing node publishes the image with skeleton point annotation to the processed image topic, the method further comprises: Converting the image with skeleton point annotation to an Image type.

[0011] In an embodiment, before the processing node publishes the image with skeleton point annotation to the processed image topic, the method further comprises: Converting the skeleton point coordinates to an array type of floating-point numbers.

[0012] Based on the same inventive concept, the second aspect of the present application provides a real-time human body pose estimation device based on ROS2, comprising: A node construction module is configured to externally connect an RGB camera to an embedded device, and construct a camera node and a processing node based on a robot operating system (ROS2); An image acquisition module is configured to acquire images in real time through the camera node, and publish the acquired images to a real-time image topic; An image processing and information publishing module is configured to subscribe to the real-time image topic published by the camera node through the processing node, acquire an image frame of the current time, and perform human body detection on the image frame of the current time. Based on the detection result, a human body pose estimation algorithm is selectively called to obtain a human body pose estimation result.

[0013] Based on the same inventive concept, the third aspect of the present application provides a ROS2-based real-time human pose estimation system, comprising the ROS2-based real-time human pose estimation device of embodiment two.

[0014] Based on the same inventive concept, the fourth aspect of the present application provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the ROS2-based real-time human pose estimation method of the first aspect when executing the program.

[0015] Compared with the prior art, the present application has the following advantages and beneficial technical effects: The ROS2-based real-time human pose estimation method disclosed in the present application externally connects an RGB camera on an embedded device, constructs a camera node and a processing node based on a robot operating system ROS2, communicates between nodes through topics, the camera node collects images in real time and publishes the collected images to a real-time image topic; the processing node subscribes to the real-time image topic published by the camera node to obtain an image frame at the current time, first performs human detection on the image frame at the current time, selectively calls a human pose estimation algorithm based on the detection result, avoids unnecessary complex operation in a no-person scene, saves computing power, especially when deployed on an embedded device with limited computing power such as Jetson Orin NX, reduces the overall consumption of the system.

[0016] Further, with the help of the good communication mechanism of ROS2, the method can publish messages as topics of visualized labeled images and key skeleton point coordinates, greatly facilitating subsequent human-machine interaction, action recognition, or behavior analysis and prediction of robots.

[0017] The ROS2-based real-time human pose estimation device of the present application is designed based on the ROS2 (robot operating system) framework, realizes smooth communication between different nodes, including image acquisition, image processing and publishing modules, so that the device can be embedded as a functional block into a larger robot application system, improving the practicality and expandability of the system. BRIEF DESCRIPTION OF DRAWINGS

[0018] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiment or prior art description. Obviously, the drawings in the following description are some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0019] Figure 1 The overall flowchart of the ROS2-based real-time human pose estimation method provided by the embodiments of the present application; Figure 2 A detailed flowchart of the ROS2-based real-time human pose estimation method in the embodiment of the application is shown in FIG. 1. Figure 3 A simulation diagram of the key skeleton points recognized by the HybrIK algorithm from the input image in the embodiment of the application is shown in FIG. 2. DETAILED DESCRIPTION

[0020] The embodiment provides a ROS2-based real-time human pose estimation method, which is shown in FIG. 1 and includes the following steps. Figure 1 S1: An RGB camera is externally connected to an embedded device, and a camera node and a processing node are constructed based on a robot operating system ROS2.

[0021] In one embodiment, the method further includes the following steps. The node message buffer size of the camera node and the processing node is respectively set.

[0022] S2: The camera node collects images in real time and publishes the collected images to a real-time image topic.

[0023] Specifically, the camera node and the processing node are constructed based on ROS2, and the nodes communicate with each other through publishing and subscribing topics.

[0024] The real-time image topic is responsible for stably transmitting the original image data collected by the camera node to the processing node.

[0025] In one embodiment, the camera node collects images at a fixed frequency, converts the collected OpenCV images into Image type, and then publishes the images to the real-time image topic.

[0026] S3: The processing node subscribes to the real-time image topic published by the camera node, acquires the image frame at the current time, and performs human detection on the image frame at the current time. Based on the detection result, the human pose estimation algorithm is selectively called to obtain the human pose estimation result.

[0027] Specifically, the processing node acquires the original image data collected by the camera node, i.e., the image frame at the current time, by subscribing to the real-time image topic. The subscribed image message is converted into OpenCV format, and human existence detection is performed. By first performing human existence detection on the image and then determining whether to call the human pose estimation algorithm, unnecessary complex operation is avoided in the absence of people, and computing power is saved.

[0028] In S3, the human existence detection on the image frame at the current time includes the following steps. A lightweight human detection model YOLO is used to detect humans in the image frame. ​

[0029] The human body detection is performed by using YOLO, and the detection efficiency can be improved.

[0030] The human body pose estimation algorithm is selectively called based on the detection result to obtain a human body pose estimation result, including: If the detection result is that no human body is detected, the processing node processes the obtained current time image and publishes the processed image to a processing image topic, where the processing image topic is used to transmit the image containing semantic information processed by the processing node. If the detection result is that a human body is detected, a preset human body pose estimation algorithm is called to perform key skeleton point recognition on the current time image frame to obtain a human body pose result, where the human body pose result includes an image with skeleton point labeling and skeleton point coordinates, the image with skeleton point labeling is published to the processing image topic, and the skeleton point coordinates are published to a skeleton point topic, and the skeleton point topic is used to transmit structured data that can be directly understood by a standardized robot system.

[0031] Specifically, please refer to Figure 2 for the detailed flowchart of the ROS2-based real-time human body pose estimation method in the embodiment of the application.

[0032] If no human body is detected, the image data is converted into an Image type and directly published to the processing image topic; If a human body is detected, a human body pose estimation algorithm is called to perform key skeleton point recognition, and since different subscription nodes have different data requirements, the image message with skeleton point labeling is converted into an Image type and published to the processing image topic, and the skeleton point coordinates are converted into a Float32MultiArray (a floating-point number array type) and published to the skeleton point topic; the processing image topic transmits the image processed by the processing node and rich in semantic information. It is no longer raw data, but a visual medium that can be read by humans and analyzed with the assistance of machines; the skeleton point topic publishes standardized, symbolic and structured data that can be directly understood by a robot system.

[0033] The application will be described in detail below. Figures 1-3 The application will be described in detail below.

[0034] An RGB camera is externally connected to an embedded device, and a camera node and a processing node are constructed based on ROS2 (Robot Operating System); In this embodiment, a Jetson Orin NX is used, an RGB camera is externally connected to the device, which is used to capture real-time images, the device system is Ubuntu 20.04, ROS2: foxy is used, python is selected as the development language, and the camera node and the processing node are constructed; The camera node collects images in real time and publishes the images to a real-time image topic; In this embodiment, the camera node is named camera_publisher, a topic / camera / image_raw is registered to the ROS2 system, the message type is Image, the message buffer and the fixed frequency of image collection are set, the OpenCV image is converted to the Image message of ROS2, and the message is published to the topic / camera / image_raw; The processing node subscribes to the real-time image topic published by the camera node, obtains the image frame at the current time, and performs human existence detection on the image frame at the current time; In this embodiment, the processing node is named image_process_node, subscribes to the topic / camera / image_raw, the message type is Image, and the message type is converted to the OpenCV image, two topics / camera / processed_result and / camera / target_coord are registered to the ROS2 system, the message types are Image and Float32MultiArray respectively, the message buffer is set, and the YOLO method is used to perform human existence detection on the subscribed image message; Based on the detection result, the human pose estimation algorithm is selectively called; If no human is detected, the processing node directly publishes the received image at the current time to the processed image topic; If a human is detected, the human pose estimation algorithm is called to identify the key skeleton points of the image frame at the current time, the image with the skeleton points marked is published to the processed image topic, and the skeleton point coordinates are published to the skeleton point topic In this embodiment, if no human existence is detected, the received image data at the current time is directly published to the topic / camera / processed_result, and if human existence is detected, the human pose estimation algorithm is called to process the image frame; In this embodiment, for the image message with human existence, the message is first converted to the OpenCV image, the human pose estimation algorithm HybrIK is called to identify the key skeleton points, the image with the skeleton points marked is converted to the Image message type, and then the message is published to the topic / camera / processed_result, and the skeleton point coordinate data is encapsulated as the Float32MultiArray message type and published to the topic / camera / target_coord; The human pose estimation algorithm is based on the open source project HybrIK (a three-dimensional human grid recovery algorithm based on inverse kinematics), which uses swing-torsion decomposition, solves swing rotation through analytical calculation to ensure joint positioning accuracy, and predicts torsion rotation through a neural network to recover orientation, thereby achieving high-fidelity full-body mesh reconstruction from a single image. The human pose estimation algorithm is extracted from it, the algorithm is packaged as an ImageProcessor class, the class is imported into the processing node, the HybrIK pose estimation model is called, and human pose estimation is realized; In the embodiment, the method is deployed on an embedded device Jetson Orin NX, and real-time image topics / camera / image_raw and processed image topics / camera / processed_result can be subscribed and displayed by a Rviz2 visualization tool in real time. The specific embodiments of the application will be described in more detail below based on simulation on the Jetson Orin NX: In order to have better migration, the embodiment uses Docker to build an environment image of the specific implementation, first pulls an NVIDIA official base image nvcr.io / nvidia / l4t-base:r35.2.1 suitable for Jetson Orin NX according to requirements, which contains a basic system Ubuntu 20.04 and CIDA 11.4; Start the container, install the dependencies required by HybrIK, YOLO and ROS2 in the container; Encapsulate HybrIK as an ImageProcessor class; The algorithm uses a backbone neural network (HRNet-W48) to extract features from the image and input the HybrIK model. The model finally performs high-fidelity full-body mesh reconstruction by driving the SMPL-X model, which has a total of 144 key points, including 17 facial contour points; In the embodiment, the 17 facial contour points are removed, and only the three-dimensional coordinates of the 127 key skeleton points are output. The recognized key skeleton points are plotted using matplotlib, as shown in Figure 3 ; Create a ROS workspace / ros2_ws / src, enter the workspace and create a Python node package; Use Python to build a camera node camera_publisher, which requires the following nodes: Set the message buffer size to 10; Set the fixed frequency of image acquisition to 20 per second; Convert the collected image to Image type; Register topic / camera / image_raw; Publish real-time image to / camera / image_raw topic; Create a processing node image_process_node with Python, and the node requirements are as follows: Set the message buffer size to 10; Subscribe to the / camera / image_raw topic, and the message type is Image, and convert the message type to OpenCV image; Register two topics, / camera / processed_result and / camera / target_coord, with message types Image and Float32MultiArray, respectively; Import the packaged ImageProcessor class and YOLO, and finally realize: Detect the presence of human body for each frame of image; If no human body is detected, the current time image data received is directly published to the / camera / processed_result topic; If a human body is detected, first convert the message to an OpenCV image, call the human pose estimation algorithm HybrIK to identify the key skeleton points, identify 127 key skeleton points, convert the skeleton point labeled image to Image message type, and then publish the message to the / camera / processed_result topic, and encapsulate the 127x3-dimensional skeleton point coordinate data as Float32MultiArray message type and publish it to the / camera / target_coord topic; After completing the framework construction, the container is packaged into an image for easy migration in the future; Start the camera node and processing node in the container; Subscribe to the / camera / image_raw topic and / camera / processed_result topic using the Rviz2 tool to view the simulation results; Embodiment Two Based on the same inventive concept, the embodiment discloses a real-time human pose estimation device based on ROS2, comprising: A node construction module is used for externally connecting an RGB camera to an embedded device, and constructing a camera node and a processing node based on a robot operating system ROS2; An image acquisition module is configured to acquire images in real time through the camera node and publish the acquired images to a real-time image topic; An image processing and information publishing module is configured to subscribe to the real-time image topic published by the camera node through the processing node, acquire an image frame at the current time, perform human body detection on the image frame at the current time, and selectively invoke a human body pose estimation algorithm based on the detection result to obtain a human body pose estimation result.

[0035] Specifically, the ROS2-based real-time human body pose estimation device is deployed on an embedded device, and the real-time image topic and the processed image topic can be subscribed and displayed in real time by the Rviz2 visualization tool, which facilitates the viewing of the recognition effect and can be subscribed to related topics by other functional nodes in the same local area network, and can be used to implement human-computer interaction, action recognition, behavior analysis and prediction, etc.

[0036] For example, a gesture instruction node subscribes to hand-related skeleton point coordinates, recognizes gestures by analyzing the relative positions and motion trajectories of the skeleton points to obtain instructions for controlling the robot, and implements simple human-computer interaction; an action recognition node subscribes to the skeleton point topic to obtain a continuous sequence of skeleton points, recognizes the skeleton sequence by a time series model such as LSTM, and classifies the action, thereby implementing recognition of simple actions such as "walking" and "raising hands"; a behavior analysis node subscribes to the skeleton point topic to obtain a continuous sequence of human skeleton points, compares the real-time skeleton sequence with a pre-trained normal behavior model by constructing a time series model such as LSTM, thereby implementing behavior analysis, predicting the movement path of the pelvis of a pedestrian in the next few seconds based on historical trajectory data of the skeleton points of the pelvis, and implementing behavior prediction.

[0037] Since the device introduced in the second embodiment of the present application is a device used to implement the ROS2-based real-time human body pose estimation method of the first embodiment of the present application, the specific structure and modifications of the device can be understood by those skilled in the art based on the method introduced in the first embodiment of the present application, and therefore will not be described here. Any device used in the method of the first embodiment of the present application belongs to the scope of the present application.

[0038] Embodiment Three Based on the same inventive concept, the present application also provides a ROS2-based real-time human body pose estimation system, which includes the ROS2-based real-time human body pose estimation device of the second embodiment.

[0039] Since the system introduced in the third embodiment of the present application is a system including the ROS2-based real-time human body pose estimation device of the second embodiment of the present application, the specific structure and modifications of the system can be understood by those skilled in the art based on the device introduced in the first embodiment of the present application, and therefore will not be described here.

[0040] Embodiment Four Based on the same inventive concept, the present application also provides an electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, wherein the processor implements the ROS2-based real-time human pose estimation method described in the embodiment when executing the program.

[0041] Since the electronic device introduced in the fourth embodiment of the present application is the electronic device used to implement the ROS2-based real-time human pose estimation method in the first embodiment of the present application, the specific structure and variations of the electronic device can be understood by those skilled in the art based on the method introduced in the first embodiment of the present application, and thus will not be described here again. Any electronic device used by the method in the first embodiment of the present application belongs to the scope of the present application.

[0042] Those skilled in the art will understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of an entirely hardware embodiment, an entirely software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROMs, optical storage, etc.) containing computer-usable program code.

[0043] The present application is described with reference to flowcharts and / or block diagrams of the method, device (system), and computer program product according to the embodiments of the present application. It should be understood that each flow and / or block in the flowcharts and / or block diagrams, as well as combinations of flows and / or blocks in the flowcharts and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing apparatus to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing apparatus produce a device that implements the functions specified in the flowcharts and / or block diagrams. Figure 1 The function specified in one or more flows and / or blocks. Figure 1 The means for implementing the function specified in one or more flows and / or blocks.

[0044] Although the preferred embodiments of the present application have been described, those skilled in the art can make additional changes and modifications to the embodiments once they know the basic inventive concept. Therefore, the appended claims are intended to be interpreted as including all the preferred embodiments and all the changes and modifications falling within the scope of the present application. Obviously, those skilled in the art can make various modifications and variations to the embodiments of the present application without departing from the spirit and scope of the embodiments of the present application. Thus, if these modifications and variations of the embodiments of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A ROS2-based real-time human pose estimation method, characterized in that, The method comprises the following steps: An RGB camera is externally connected to an embedded device, and a camera node and a processing node are constructed based on a robot operating system ROS2; An image is collected in real time through the camera node, and the collected image is published to a real-time image topic; The processing node subscribes to the real-time image topic published by the camera node, obtains an image frame at the current time, and performs human body detection on the image frame at the current time, and based on the detection result, selectively calls a human body pose estimation algorithm to obtain a human body pose estimation result.

2. The ROS2-based real-time human pose estimation method of claim 1, wherein, The human body existence detection on the image frame at the current time comprises the following steps: A lightweight human detection model YOLO is used to detect the human body in the image frame.

3. The ROS2-based real-time human pose estimation method of claim 1, wherein, Based on the detection result, the human body pose estimation algorithm is selectively called to obtain the human body pose estimation result, which comprises the following steps: If the detection result is that no human body is detected, the processing node publishes the processed image at the current time to a processing image topic, wherein the processing image topic is used to transmit the image containing semantic information processed by the processing node; If the detection result is that a human body is detected, a preset human body pose estimation algorithm is called to recognize key skeleton points of the image frame at the current time to obtain a human body pose result, wherein the human body pose result comprises an image with skeleton point labeling and skeleton point coordinates, the image with skeleton point labeling is published to the processing image topic, and the skeleton point coordinates are published to a skeleton point topic, and the skeleton point topic is used to transmit structured data that can be directly understood by a standardized robot system.

4. The ROS2-based real-time human pose estimation method of claim 1, wherein, The method further comprises the following steps: The node message buffer size of the camera node and the processing node is respectively set.

5. The ROS2-based real-time human pose estimation method of claim 1, wherein, Before the camera node publishes the collected image to the real-time image topic, the method further comprises the following steps: The collected OpenCV type image is converted to an Image type.

6. The ROS2-based real-time human pose estimation method of claim 1, wherein, Before the processing node publishes the image with skeleton point labeling to the processing image topic, the method further comprises the following steps: The image with skeleton point labeling is converted to an Image type.

7. The ROS2-based real-time human pose estimation method of claim 1, wherein, Before the processing node publishes the image with skeleton point labeling to the processing image topic, the method further comprises the following steps: The skeleton point coordinates are converted to an array type of floating-point numbers.

8. A ROS2-based real-time human pose estimation device, comprising: The method comprises the following steps: A node construction module is configured to externally connect an RGB camera to an embedded device, and construct a camera node and a processing node based on a robot operating system ROS2; An image collection module is configured to collect an image in real time through the camera node, and publish the collected image to a real-time image topic; An image processing and information publishing module is configured to subscribe to the real-time image topic published by the camera node through the processing node, obtain an image frame at the current time, and perform human body detection on the image frame at the current time, and based on the detection result, selectively call a human body pose estimation algorithm to obtain a human body pose estimation result.

9. A ROS2-based real-time human pose estimation system, characterized in that, The device comprises the ROS2-based real-time human body pose estimation device according to claim 8.

10. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor, characterized in that, The processor executes the program to implement the ROS2-based real-time human body pose estimation method according to any one of claims 1 to 7.

Images