Human body three-dimensional posture estimation method and system based on multi-view scene, terminal and storage medium

Abstract

The invention discloses a human body three-dimensional posture estimation method and system based on a multi-view scene, a terminal and a storage medium, and the method comprises the steps: determining a multi-view camera, and carrying out the video stream synchronous reading setting and camera calibration processing of the multi-view camera, and obtaining a target multi-view camera; collecting a target human body video stream through the target multi-view camera, and performing key point extraction processing on the target human body video stream to obtain initial human body key points; performing image distortion removal processing on the initial human body key points to obtain target human body key points; according to the target human body key points, posture recognition processing and three-dimensional reconstruction processing are carried out, and a human body three-dimensional posture estimation result is obtained. According to the method, the accuracy and the real-time performance of human body three-dimensional posture estimation of multiple cameras in a complex dynamic scene can be effectively improved.

Description

Technical Field

[0001] The present invention relates to the technical field of data processing, and in particular, to a method, system, terminal, and computer-readable storage medium for human three-dimensional pose estimation based on a multi-view scenario. Background Art

[0002] With the rapid development of computer vision and artificial intelligence technologies, human pose recognition technology has been widely applied in many fields, such as virtual reality, augmented reality, human-computer interaction, robot control, and medical rehabilitation. The goal of human pose recognition is to extract the key points of the human body from images or videos and analyze their relative positions and movements, so as to identify information such as the movement patterns and postures of the human body.

[0003] Currently, human pose recognition technology is mainly divided into two types: single-camera-based pose recognition and multi-camera-based pose recognition. Multi-camera pose recognition technology performs relatively stably in static scenarios. However, in dynamic scenarios, especially when the human body moves rapidly or multiple targets appear in the field of view simultaneously, the recognition accuracy and real-time performance are often affected. Factors such as camera synchronization problems, motion blur, and target overlap will lead to a decrease in the estimation accuracy of motion postures, making it difficult to meet the requirements of rapidly changing or complex interaction scenarios.

[0004] Therefore, the existing technology still needs to be improved and developed. Summary of the Invention

[0005] The main purpose of the present invention is to provide a method, system, terminal, and computer-readable storage medium for human three-dimensional pose estimation based on a multi-view scenario, aiming to solve the problems of poor recognition accuracy and inaccurate motion pose estimation of multi-camera pose recognition technology in dynamic scenarios in the existing technology.

[0006] To achieve the above object, the present invention provides a method for human three-dimensional pose estimation based on a multi-view scenario, and the method for human three-dimensional pose estimation based on a multi-view scenario includes the following steps:

[0007] Determine multi-view cameras, and perform video stream synchronization reading settings and camera calibration processing on the multi-view cameras to obtain target multi-view cameras;

[0008] Collect a target human body video stream through the target multi-view cameras, and perform key point extraction processing on the target human body video stream to obtain initial human key points;

[0009] Perform image distortion removal processing on the initial human key points to obtain target human key points;

[0010] Perform pose recognition processing and 3D reconstruction processing based on the target human body key points to obtain the human body 3D pose estimation result.

[0011] Optionally, in the human body 3D pose estimation method based on a multi-view scenario, the determining of the multi-view cameras and the setting of video stream synchronous reading and camera calibration processing for the multi-view cameras to obtain the target multi-view cameras specifically includes:

[0012] Set multiple preset positions, and set cameras at the multiple preset positions to obtain multi-view cameras, where the preset positions include directly in front of, diagonally in front of the right, and directly to the right of the target human body;

[0013] Use a multi-threaded method to perform video stream synchronous reading settings for the multi-view cameras, and use the checkerboard calibration method to perform camera calibration processing on the multi-view cameras to obtain the camera internal parameters;

[0014] When the multi-view cameras complete the video stream synchronous reading settings and the camera calibration processing, the target multi-view cameras are obtained.

[0015] Optionally, in the human body 3D pose estimation method based on a multi-view scenario, the using of the checkerboard calibration method to perform camera calibration processing on the multi-view cameras to obtain the camera internal parameters specifically includes:

[0016] Set a preset checkerboard, and obtain multi-angle checkerboard images of the preset checkerboard;

[0017] Extract the checkerboard corner points of the multi-angle checkerboard images, and perform quality optimization processing on the checkerboard corner points to obtain the target checkerboard corner point coordinates;

[0018] Obtain the calibration size information of the preset checkerboard, and use the non-linear optimization solution method to calculate based on the calibration size information and the target checkerboard corner point coordinates to obtain the camera internal parameters, where the camera internal parameters include focal length, principal point position, and distortion coefficient.

[0019] Optionally, in the human body 3D pose estimation method based on a multi-view scenario, the collecting of the target human body video stream by the target multi-view cameras and the performing of key point extraction processing on the target human body video stream to obtain the initial human body key points specifically includes:

[0020] Collect a video stream through the target multi-view cameras to obtain an initial human body video stream, and perform size adjustment processing and format conversion processing on the initial human body video stream to obtain the target human body video stream;

[0021] Perform the first key point extraction process on the target human body video stream through the MediaPipe pose estimation model to obtain human joint key points;

[0022] Perform the second key point extraction process on the target human body video stream through the MediaPipe hand estimation model to obtain hand key points;

[0023] Perform absolute coordinate conversion processing on the human joint key points and the hand key points to obtain initial human key points.

[0024] Optionally, in the human body three-dimensional pose estimation method based on a multi-view scenario, where the image distortion removal process is performed on the initial human key points to obtain target human key points, specifically:

[0025] Obtain the key point video where the initial human key points are located, and use a distortion removal algorithm to perform radial distortion elimination and tangential distortion elimination on each frame image in the key point video to obtain target human key points.

[0026] Optionally, in the human body three-dimensional pose estimation method based on a multi-view scenario, where the pose recognition process and three-dimensional reconstruction process are performed according to the target human key points to obtain the human body three-dimensional pose estimation result, specifically including:

[0027] Obtain the key point two-dimensional coordinates corresponding to the target human key points, and perform coordinate reconstruction processing and coordinate conversion processing on the key point two-dimensional coordinates to obtain key point three-dimensional coordinates;

[0028] Perform three-dimensional reconstruction processing on the key point three-dimensional coordinates to obtain the human body three-dimensional pose estimation result.

[0029] Optionally, in the human body three-dimensional pose estimation method based on a multi-view scenario, where the pose recognition process and three-dimensional reconstruction process are performed according to the target human key points to obtain the human body three-dimensional pose estimation result, and then it further includes:

[0030] Obtain the spatial vectors corresponding to each key point in the human body three-dimensional pose estimation result, and calculate the vector angles between the spatial vectors;

[0031] Determine the target type angles according to the vector angles, where the target type angles include three-dimensional angles, two-dimensional plane angles, and direction angles;

[0032] Determine the target robotic arm, and map the target type angles to the target robotic arm to enable the target robotic arm to complete the specified actions of the target human body.

[0033] In addition, to achieve the above object, the present invention also provides a human three-dimensional pose estimation system based on a multi-view scenario. Among them, the human three-dimensional pose estimation system based on a multi-view scenario includes:

[0034] A multi-view camera setting module for determining a multi-view camera and performing video stream synchronous reading setting and camera calibration processing on the multi-view camera to obtain a target multi-view camera;

[0035] A key point extraction module for collecting a target human body video stream through the target multi-view camera and performing key point extraction processing on the target human body video stream to obtain initial human body key points;

[0036] An image distortion removal module for performing image distortion removal processing on the initial human body key points to obtain target human body key points;

[0037] A pose estimation result generation module for performing pose recognition processing and three-dimensional reconstruction processing according to the target human body key points to obtain a human three-dimensional pose estimation result.

[0038] In the present invention, a multi-view camera is determined, and video stream synchronous reading setting and camera calibration processing are performed on the multi-view camera to obtain a target multi-view camera; a target human body video stream is collected through the target multi-view camera, and key point extraction processing is performed on the target human body video stream to obtain initial human body key points; image distortion removal processing is performed on the initial human body key points to obtain target human body key points; pose recognition processing and three-dimensional reconstruction processing are performed according to the target human body key points to obtain a human three-dimensional pose estimation result. The present invention collects a target human body video stream by using the target multi-view camera after video stream synchronous reading setting and camera calibration processing, and performs key point extraction, image distortion removal, pose recognition, and three-dimensional reconstruction processing on the target human body video stream, thereby obtaining an accurate and real-time human three-dimensional pose estimation result, effectively improving the accuracy of pose estimation of multiple cameras in a dynamic scenario. Description of the Drawings

[0039] Figure 1 is a flowchart of a preferred embodiment of the human three-dimensional pose estimation method based on a multi-view scenario of the present invention;

[0040] Figure 2 is a schematic diagram of the overall implementation process of a preferred embodiment of the human three-dimensional pose estimation method based on a multi-view scenario of the present invention;

[0041] Figure 3 is a schematic diagram of the shooting positions of multi-view cameras in a preferred embodiment of the human three-dimensional pose estimation method based on a multi-view scenario of the present invention;

[0042] Figure 4 It is a structural diagram of a preferred embodiment of the human body three-dimensional pose estimation system based on a multi-view scenario of the present invention;

[0043] Figure 5 It is a structural diagram of a preferred embodiment of the terminal of the present invention. Specific embodiments

[0044] To make the objectives, technical solutions and advantages of the present invention clearer and more definite, the following further elaborates on the present invention with reference to the accompanying drawings and by way of examples. It should be understood that the specific embodiments described herein are merely for explaining the present invention and are not intended to limit the present invention.

[0045] With the rapid development of computer vision and artificial intelligence technologies, human pose recognition technology has been widely applied in many fields, such as virtual reality, augmented reality, human-computer interaction, robot control, and medical rehabilitation. The goal of human pose recognition is to extract the key points of the human body from images or videos and analyze their relative positions and movements, so as to identify information such as the movement patterns and postures of the human body.

[0046] Currently, human pose recognition technology is mainly divided into two types: single-camera-based pose recognition and multi-camera-based pose recognition.

[0047] Single-camera pose recognition is the most basic pose recognition method, which mainly relies on feature point extraction in images and deep learning models to infer the joint positions of the human body. By analyzing the human features in the image, the coordinates of each part of the human body relative to the image plane are estimated. This method is usually applicable to the recognition in static environments and simple postures and has low hardware requirements. However, the single-camera pose recognition method also has the following significant problems: 1. Viewpoint limitation: A single camera can only observe the human body from a fixed angle. Therefore, in complex movements or viewpoints such as the side or back, occlusion is likely to occur, affecting the accuracy of key point extraction and pose estimation. 2. Lack of depth information: A single camera cannot provide the depth information of the scene, resulting in a lack of accurate spatial position information when performing three-dimensional pose estimation and being unable to accurately infer the three-dimensional structure of the human body.

[0048] To address the limitations of single-camera pose recognition, multi-camera systems have been widely used in recent years. The multi-camera system captures human body images from multiple viewpoints by installing multiple cameras at different positions, and then calculates the three-dimensional depth information through technologies such as stereo vision and triangulation, so as to more comprehensively and accurately reconstruct the three-dimensional pose of the human body.

[0049] Multi-camera systems can generally overcome the limitations of single cameras in rear or large-angle movements. By combining different perspectives, they can provide more accurate three-dimensional pose estimation. However, although multi-camera systems have a significant improvement in accuracy compared to single-camera systems, they still face the following technical challenges: 1. Camera calibration and synchronization issues: The perspectives of multiple cameras must be precisely aligned, so strict camera calibration is required. However, there may be certain errors in the calibration process. If the synchronization and calibration between multiple cameras are not precise enough, it will lead to deviations in depth information in three-dimensional reconstruction, thus affecting the recognition accuracy. 2. High computational resource consumption: Multi-camera systems need to process real-time video streams from multiple cameras simultaneously, resulting in high computational resource consumption. Especially in real-time processing applications, problems such as computational latency and poor real-time performance may be faced, which is particularly prominent in some embedded devices or resource-constrained environments. 3. Perspective blind spot problems: Although multi-camera systems can provide a more comprehensive perspective, in practical applications, there may be insufficient overlapping areas or perspective blind spots between multiple cameras. Especially when the camera layout is unreasonable or the perspective angle is too large, problems such as data loss or failure in depth information reconstruction may occur.

[0050] Existing multi-camera pose recognition technologies perform relatively stably in static scenarios. However, in dynamic scenarios, especially when the human body moves rapidly or multiple targets appear in the field of view simultaneously, the recognition accuracy and real-time performance are often affected. Factors such as camera synchronization problems, motion blur, and target overlap will lead to a decrease in the estimation accuracy of motion postures, making it difficult to meet the requirements of rapidly changing or complex interaction scenarios. In some applications with high-precision and real-time feedback requirements, such as robot control and interactive virtual reality, there is still much room for improvement in the existing technologies in terms of motion capture and interaction real-time performance. The existing technologies are difficult to fully meet the strict requirements for accuracy, stability, and real-time performance in dynamic scenarios. Although the existing single-camera and multi-camera pose recognition technologies have made significant progress in static scenarios and simple applications, they still have drawbacks such as perspective limitations, excessive computational resource consumption, inaccurate camera calibration, synchronization problems, and poor adaptability to dynamic scenarios. Therefore, how to optimize the accuracy, real-time performance, and computational efficiency of multi-camera systems and solve the perspective blind spot problem remains a major challenge in current technologies.

[0051] To solve the above problems, the present invention proposes a method for estimating the three-dimensional pose of the human body in a multi-perspective scenario, aiming to solve problems such as accuracy and real-time performance in human pose recognition in the existing technologies. By using multiple cameras to collect the pose data of the human body from different perspectives and combining deep learning and three-dimensional reconstruction technologies, the present invention can accurately and real-time estimate the three-dimensional pose of the human body in complex dynamic scenarios and apply this pose data to fields such as robot control and intelligent teaching.

[0052] The present invention uses three cameras, which are respectively arranged in the front, right front, and right of the human body, ensuring that all-round pose information of the human body can be captured and effectively avoiding occlusion problems. The video streams of multiple cameras are read synchronously to ensure data consistency. Through precise camera calibration, the internal parameters of each camera are calculated to ensure that the viewing angles and parameters of all cameras are consistent, thereby improving the accuracy of 3D reconstruction. During the data processing, the system first performs distortion removal on the images collected by each camera to eliminate the influence of lens distortion, then extracts two-dimensional key point data from each video stream, and finally reconstructs the 3D pose model of the human body through multi-view data fusion, 3D coordinate reconstruction, and spatial relationship calculation. By converting the 3D pose data into angular data, the present invention can generate precise control signals suitable for robot control, and these signals can be sent to devices such as robotic arms in real time through a router to complete intelligent teaching and action execution. The multi-view data fusion and precise 3D reconstruction of the present invention enable it to effectively improve the accuracy of pose estimation, especially in complex and dynamic scenarios, and solve the deficiencies of traditional single-camera or multi-camera systems in terms of accuracy, real-time performance, and viewing angle blind spots.

[0053] In summary, through innovative multi-view data fusion, 3D reconstruction, and intelligent control technologies, the present invention provides an efficient, precise, and real-time feedback capable pose recognition system, which can be widely applied to multiple fields such as robot control, virtual reality, rehabilitation training, and interactive entertainment, and has important technical advantages and broad application prospects.

[0054] The method for estimating the 3D pose of a human body based on a multi-view scenario according to a preferred embodiment of the present invention, as Figure 1 shown, the method for estimating the 3D pose of a human body based on a multi-view scenario includes the following steps:

[0055] Step S10: Determine multi-view cameras, and perform video stream synchronous reading settings and camera calibration processing on the multi-view cameras to obtain target multi-view cameras.

[0056] The present invention proposes a method for estimating the 3D pose of a human body based on a multi-view scenario, aiming to solve problems such as insufficient accuracy, poor real-time performance, and viewing angle blind spots in the prior art. By introducing multiple cameras and combining deep learning and 3D reconstruction technologies, the present invention can achieve high-precision real-time pose estimation in complex dynamic scenarios and apply it to fields such as robot control and intelligent teaching.

[0057] Specifically, set multiple preset positions, and set cameras at the multiple preset positions to obtain multi-view cameras, where the preset positions include the front, right front, and right of the target human body.

[0058] As Figure 2As shown in the figure, for the selection and layout of the camera: In order to achieve low-cost and efficient three-dimensional pose recognition, the present invention selects the Logitech C920e as the camera for video stream acquisition. This camera performs stably under standard lighting conditions and can provide sufficient image quality for human pose estimation.

[0059] Although the Logitech C920e camera does not have depth perception capabilities, in the present invention, combined with the MediaPipe algorithm (the MediaPipe algorithm is a gesture recognition algorithm based on machine learning technology, which is characterized by high accuracy, supports five-finger and gesture tracking, and can infer 21 three-dimensional nodes of a single hand from a single frame of image), efficient human key point extraction can be performed through RGB images, meeting the requirements for accurate estimation of key parts (such as shoulders, elbows, wrists, and palms). Through reasonable camera layout and multi-view data fusion, the limitations of the lack of a depth camera can be compensated, providing stable real-time data support.

[0060] As Figure 3 shown in the figure, in order to ensure that the joint movements of the human body can be accurately captured, the present invention uses three Logitech C920e cameras and installs them respectively at 1. In front of the human body: mainly responsible for capturing the front actions of the human body to ensure accurate recognition of the front pose and joint positions. 2. Installed on the right side of the human body: specifically responsible for capturing the side actions of the human body, especially parts such as the right shoulder, right elbow, and right wrist. The perspective of the right-side camera can effectively supplement the blind area of the front camera, especially providing accurate support for the movement of the upper limb joints. 3. Installed in the front right of the human body, forming a certain angle with the right-side camera, used to further supplement the perspective of the right-side camera and help improve the accuracy of the right shoulder and right elbow. Its main role is to reduce errors in depth estimation and pose estimation to ensure more accurate three-dimensional data can be obtained. At the same time, the distances and heights of the three cameras are initially calibrated to achieve higher accuracy. The three cameras are all located on a circle with a radius of 1.35m and a height of 1.35m. Such an arrangement can ensure that the upper limbs of the human body can be recognized by all three cameras without causing too much useless footage, resulting in resource waste and reduced accuracy. This layout design of the three cameras can effectively solve the limitations brought by the perspective of a traditional single camera. Especially in the case where parts such as the side and back of the human body are easily blocked, multiple cameras can provide all-round perspective support to ensure that the movement of each joint can be accurately captured.

[0061] The present invention mainly realizes a three-dimensional reconstruction and control system based on multi-camera data fusion and human pose estimation. To achieve this goal, a series of key modules are designed, including camera selection and layout, synchronous reading of video streams, camera calibration, key point extraction and processing, image de-distortion, three-dimensional pose reconstruction, and action control mapping, etc.

[0062] In addition, a depth camera can be used to replace the Logitech C920e camera: If the budget allows, a depth camera (such as Microsoft Kinect or Intel RealSense, etc.) can be considered to replace the RGB camera. The depth camera can directly provide depth information, avoiding the complexity of calculating depth through three-view image fusion. However, this may increase the hardware cost and complexity of the system. At the same time, an extended layout with more cameras can also be adopted: If higher-precision 3D reconstruction or more complex environmental monitoring is required, more cameras can be considered, such as adding cameras at the back, left, or above to increase the coverage of the human body.

[0063] The multi-view camera is set to synchronously read the video stream in a multi-threaded manner, and a preset checkerboard is set, and multi-angle checkerboard images of the preset checkerboard are obtained; the checkerboard corner points of the multi-angle checkerboard images are extracted, and quality optimization processing is performed on the checkerboard corner points to obtain target checkerboard corner point coordinates; the calibration size information of the preset checkerboard is obtained, and a non-linear optimization solution method is used to calculate according to the calibration size information and the target checkerboard corner point coordinates to obtain the camera internal parameters, where the camera internal parameters include focal length, principal point position, and distortion coefficient; when the multi-view camera completes the video stream synchronous reading setting and the camera calibration processing, a target multi-view camera is obtained.

[0064] As Figure 2 shown, for the synchronous reading process of the video stream: In a multi-camera system, the synchronous reading of the video stream is the key to ensuring data consistency and pose estimation accuracy. Since the system relies on image data from multiple perspectives for 3D reconstruction and action recognition, ensuring that the data collected by different cameras can be aligned in time and avoiding misalignment caused by time differences is the basis for achieving high-precision results. To ensure video stream synchronization, the present invention uses parallel processing and multi-threaded technology to enable the video stream of each camera to be independently read and processed in a timely manner, and at the same time, data consistency is ensured through synchronous control.

[0065] It can be understood that the present invention starts an independent processing thread for each camera in a multi-threaded manner, enabling each camera to capture the video stream in parallel and extract key points. To optimize the processing efficiency, the present invention performs image processing regularly through frame number control. Every certain number of frames (preferably set to every 5 frames, which can not only ensure the recognition effect but also ensure good real-time performance without jamming), the system will perform a key point detection once, avoiding excessive resource consumption caused by frequent calculations. This method not only improves the processing efficiency but also reduces the calculation delay, ensuring the real-time performance and stability of the system.

[0066] AsFigure 2 As shown, a camera calibration system is set up in the present invention. To ensure the accuracy of the multi-camera system, the internal parameters of the cameras are first calibrated. Since the specific positions of the three cameras are already known, the calibration process focuses on calculating the internal parameters of each camera, including the focal length, the position of the principal point, and the distortion coefficient, etc. These parameters are essential because they ensure that each camera can accurately project the three-dimensional world onto the two-dimensional image plane.

[0067] For camera calibration, the present invention uses the checkerboard calibration method. A checkerboard with a design size of 9*12 and a grid spacing of 2 cm is designed and printed on A4 paper. By taking checkerboard images at multiple different angles, the positions of the checkerboard corner points are extracted, and these data are used to calculate the internal parameters of each camera. During this process, the OpenCV tool is used for corner point extraction (corner detection is realized through cv2.cornerHarris, the corner quality is optimized by combining the Shi-Tomasi algorithm, and the sub-pixel coordinates are output by inputting the image, the number of corner points, and the quality threshold. It is applicable to structured scenarios such as checkerboard calibration boards, with strong robustness and high computational efficiency) and internal parameter calculation (the internal parameters are calculated through checkerboard calibration board images at multiple angles using cv2.calibrateCamera. First, the corner coordinates of each image are extracted, and combined with the known physical size information of the calibration board, the camera matrix, distortion coefficient, and other parameters are solved through non-linear optimization. The internal parameter matrix contains core information such as the focal length and the principal point, and the reprojection error is used to evaluate the calibration accuracy). To ensure the calibration accuracy, the present invention visualizes the calibration process during calibration and takes manual photos. After the camera accurately detects the corner points, photos are taken to ensure that each photo is at a different angle to improve the quality of the photos. Finally, the reprojection error is also calculated (the 3D points of the calibration board are projected onto the image plane through cv2.projectPoints, compared with the measured corner point coordinates, and the root mean square of the Euclidean distance between the two is calculated. The error is directly output by cv2.calibrateCamera or manually traversed through all corner points to find the mean. The error value reflects the calibration accuracy. If it is lower than the threshold (such as 0.1 pixel), it indicates that the internal parameters (focal length, principal point) and the distortion coefficient estimation are reliable and meet the accuracy requirements of subsequent stereo vision or multi-view geometry applications), and the threshold is set to 0.1. When the error is less than 0.1, it proves that this calibration meets the requirements. Through this calibration step, the present invention can obtain the internal parameters of each camera, such as the focal length, the position of the principal point, and the distortion coefficient.

[0068] After completing the internal parameter calibration, the internal parameters of each camera are saved to a specified file. During each data acquisition, these internal parameters are loaded to ensure that the output of each camera conforms to an accurate geometric projection, thus guaranteeing the accuracy of subsequent image processing, pose estimation, and 3D reconstruction. Through the internal parameter calibration of the camera, the internal parameters of the camera are obtained, providing accurate parameter support for subsequent image de-distortion, multi-camera data fusion, and 3D reconstruction. The calibration process provides a stable and reliable foundation for the system, ensuring that data from different cameras can be accurately fused in the same coordinate system.

[0069] In addition, the present invention can also adopt hardware synchronization: in addition to using software synchronization (multi-threading), multi-camera synchronization can also be achieved through hardware triggering. Some high-end cameras support synchronization triggers, which can ensure that different cameras capture the same picture at the same moment, avoiding errors caused by differences in image acquisition times. Frame synchronization technology can also be adopted: professional frame synchronization devices (such as Genlock) can be used to synchronize the frame rates and timestamps of all cameras, enabling the images to be completely aligned at the physical level.

[0070] In addition, the present invention can also adopt calibration tools and algorithms: in addition to the checkerboard calibration method, circular pattern calibration or self-calibration methods can also be used. The self-calibration method does not rely on checkerboard patterns but calculates the internal parameters by the camera independently capturing a certain image sequence, which is suitable for dynamic or scenarios where patterns cannot be prepared in advance. Deep learning-assisted calibration: In some complex environments, calibration methods based on deep learning may be required, using neural network models to automatically learn the internal parameters of the camera from data at multiple perspectives.

[0071] Step S20: Collect the target human body video stream through the target multi-view camera, and perform key point extraction processing on the target human body video stream to obtain initial human key points.

[0072] As Figure 2 shown, for multi-view key point reading: In a multi-camera system, key point extraction is one of the core tasks of pose estimation and 3D reconstruction. To ensure efficient and accurate pose estimation, the present invention focuses on extracting five key joint points, namely the left shoulder, right shoulder, right elbow, right wrist, and right palm center. These key points are the basic data for controlling the robotic arm and performing motion analysis. Each camera captures different perspectives of the human body through independent video streams. The goal is to extract these five joint points from these video streams and provide support for subsequent data processing and analysis.

[0073] Specifically, video stream acquisition is performed through the target multi-view camera to obtain an initial human body video stream, and size adjustment processing and format conversion processing are performed on the initial human body video stream to obtain a target human body video stream; the MediaPipe pose estimation model is used to perform first key point extraction processing on the target human body video stream to obtain human joint key points; the MediaPipe hand estimation model is used to perform second key point extraction processing on the target human body video stream to obtain hand key points; absolute coordinate conversion processing is performed on the human joint key points and the hand key points to obtain initial human body key points.

[0074] To achieve efficient key point extraction, the present invention uses the Pose model and the Hands model in the MediaPipe framework. The Pose model can identify multiple joints of the human body, and the Hands model focuses on the key point detection of the hands. Although these models can detect more key points, in order to improve the calculation efficiency and system response speed, the present invention only focuses on these five specific joint points, thereby reducing unnecessary computational volume.

[0075] First, the system reads the video stream through OpenCV and adjusts the size of each frame of the image uniformly (to ensure good recognition effect, the image size is uniformly adjusted to 640*480) to ensure consistent image input from all cameras. Each frame of the video is converted to the RGB format, which is convenient for subsequent compatibility with the MediaPipe model. Then, the MediaPipePose model is used to extract the joint key points of the human body (MediaPipe Pose is based on a lightweight CNN architecture and is implemented through a two-stage process: first, detect the human body bounding box, and then regress the 3D coordinates of 33 joint points. The model uses heatmap and offset prediction, combines post-processing to optimize the spatial consistency of the key points, and outputs information including position and confidence. Its lightweight design supports real-time detection and is suitable for scenarios such as motion analysis and pose estimation, and has a certain robustness to occlusion and complex backgrounds), and the MediaPipe Hands model is used to detect the key points of the hand (MediaPipe Hands locates the hand by segmenting the palm area and then regresses the 3D coordinates of 21 key points. The model combines an encoder-decoder network with an attention mechanism to accurately capture details such as fingertips and joints, supports single-hand or two-hand detection, and the output key points include spatial position and visibility score, adapting to challenges such as multi-angle and occlusion, meeting the needs of gesture interaction and sign language recognition, and taking into account both real-time performance and accuracy). In the present invention, specifically, the five joint positions of the left shoulder, right shoulder, right elbow, right wrist and right palm center are extracted, which ensures efficient processing while guaranteeing sufficient pose information to meet the needs of robot control. To optimize the processing performance of the system, the system is set to perform key point detection every 5 frames. In this way, the calculation frequency and resource consumption are reduced, and while maintaining the real-time performance of the system, the overall efficiency is improved. Once the two-dimensional coordinates of the key points are extracted, the system converts these relative coordinates into absolute coordinates according to the width and height of the image to ensure the accuracy of the key point positions (the key point coordinates output by MediaPipe are: coordinates based on a coordinate system with a fixed coordinate point as the origin, and need to be combined with the image resolution to be converted into absolute coordinates. This process eliminates the influence of image scaling, ensures that the key points are aligned with the original pixels, provides a geometric consistency basis for subsequent 3D reconstruction or motion analysis, and avoids introducing errors due to resolution differences). After each valid data is detected, the system saves these coordinates as a JSON format file and stores it in the specified path to ensure the orderliness of data management and the convenience of subsequent processing).

[0076] In this way, the present invention can accurately extract the required key point data from the video stream of each camera and ensure its synchronization and accuracy, which is particularly suitable for subsequent 3D pose reconstruction and robot control tasks.

[0077] In addition, in addition to MediaPipe, algorithms such as OpenPose and AlphaPose can also effectively achieve human pose estimation. OpenPose is suitable for real-time pose recognition of multiple people, while AlphaPose has high accuracy and is suitable for accurate recognition in complex scenarios. Custom models based on deep learning: If higher accuracy or refined control of specific joints is required, a custom deep learning model can be trained to extract more key points or more complex pose information.

[0078] Step S30: Perform image undistortion processing on the initial human key points to obtain target human key points.

[0079] In a multi-camera system, due to the different lens characteristics and shooting angles, image distortion is inevitable, especially under wide-angle lenses, where the distortion phenomenon is particularly obvious. Image distortion can cause straight lines in the image to deform, thereby affecting the extraction of key points and the accuracy of 3D reconstruction. Therefore, image undistortion is one of the key steps to ensure the accuracy of the system.

[0080] Specifically, obtain the key point video where the initial human key points are located, and use an undistortion algorithm to perform radial distortion elimination and tangential distortion elimination on each frame of the image in the key point video according to the camera internal parameters to obtain target human key points.

[0081] As Figure 2 shown, the present invention uses the undistortion algorithm in OpenCV to correct the image through the internal parameters of the camera and eliminate the errors caused by lens distortion. Each camera has been accurately calibrated for internal parameters to obtain the corresponding distortion coefficients. Based on these internal parameters, each frame of the image can be accurately corrected for distortion to ensure the accurate positioning of key points in the image.

[0082] The specific process of de - distortion is as follows: First, the system extracts the internal parameters of each camera through the checkerboard calibration method, and uses the camera matrix and distortion coefficients obtained during the calibration process to perform de - distortion processing on each frame of video. The cv2.undistort method provided by OpenCV uses the camera matrix and distortion coefficients (by using multi - angle images of the checkerboard calibration board and calculating the internal parameters using cv2.calibrateCamera. First, extract the corner coordinates of each image, combine the known physical size information of the calibration board, and solve parameters such as the camera matrix and distortion coefficients through non - linear optimization. The internal parameter matrix contains core information such as focal length and principal point, and the reprojection error is used to evaluate the calibration accuracy) to perform perspective transformation on the image and eliminate the radial and tangential distortions in the image (cv2.undistort first constructs a projection relationship based on the camera internal parameter matrix, converting the image coordinate system to the normalized camera coordinate system. For radial distortion (k1, k2, k3) and tangential distortion (p1, p2), a non - linear model is used to calculate the corrected coordinates, and finally, bilinear interpolation resampling is used to generate an undistorted image. This process keeps the deformation of the central region of the image minimal and has a significant edge correction effect).

[0083] The de - distortion process can effectively correct the geometric distortion of the lens, avoiding the incorrect position of key points caused by distortion. After removing the image distortion, the present invention can accurately extract the two - dimensional coordinates of human key points and use them for subsequent three - dimensional pose reconstruction and action analysis. In this way, the accuracy of multi - camera data fusion is improved, ensuring the consistency and reliability of key - point data.

[0084] In addition, for the image de - distortion process, panoramic image de - distortion can also be used in the present invention: If a wide - angle camera or a panoramic camera (such as a 360° camera) is used, a more complex de - distortion algorithm is required to handle extreme distortion, which can be solved by the fisheye distortion model. Depth image correction: If a depth camera is used, distortion correction can be combined with the depth image to ensure that not only the RGB image but also the accuracy of the depth data is guaranteed.

[0085] Step S40: Perform pose recognition processing and three - dimensional reconstruction processing based on the target human key points to obtain the human three - dimensional pose estimation result.

[0086] After obtaining the coordinates of the target human body key points, these coordinates will be saved in their respective JSON files. After performing distortion removal processing on the images of each camera, triangulation is used to calculate the three-dimensional coordinates. The two-dimensional coordinates of the same key point captured by each camera from different perspectives can be used for triangulation. Combining the known relative positions and internal parameters of the cameras, the actual position of the key point in the three-dimensional space can be calculated. The process of triangulation is to convert the image coordinates of the camera into coordinates in the camera coordinate system, and then estimate the same key point based on multiple perspectives. Finally, the three-dimensional coordinates of the point are restored (using the time synchronization mechanism to match the same key point at the same moment in multiple cameras to ensure that the observations from different perspectives correspond to the points at the same position of the human body. Through the functions provided by OpenCV, the pixel coordinates are converted into unit direction vectors after undistortion to describe the position and direction of the key point relative to each camera. Finally, the linear least squares method is used to estimate the intersection position of multiple direction vectors to obtain the optimal intersection coordinates of the key point in the three-dimensional space, completing the triangulation process).

[0087] Specifically, obtain the two-dimensional coordinates of the key points corresponding to the target human body key points, and perform coordinate reconstruction processing and coordinate conversion processing on the two-dimensional coordinates of the key points to obtain the three-dimensional coordinates of the key points; perform three-dimensional reconstruction processing on the three-dimensional coordinates of the key points to obtain the three-dimensional human body pose estimation result.

[0088] As Figure 2 shown, for each key point, the present invention performs coordinate reconstruction through the two-dimensional coordinates of multiple cameras to calculate the corresponding three-dimensional space coordinates. These calculation results reflect the position and pose of the human body in the three-dimensional space. Specifically, the three-dimensional coordinates of key joints such as the right wrist, right elbow, and right shoulder can be obtained through the fusion calculation of two-dimensional coordinates from multiple perspectives, and then through coordinate conversion, they are transformed into a three-dimensional coordinate system with the left shoulder of the person as the origin. In this way, the skeleton structure of the human body in the three-dimensional space can be accurately restored. Then, the three-dimensional coordinates of the five key points are saved as a JSON file for the subsequent control of the robotic arm.

[0089] As Figure 2 shown, the calculated three-dimensional coordinates are then used for three-dimensional visualization. These three-dimensional coordinates are displayed as an interactive three-dimensional skeleton model through the Plotly library. Each key point in the model is represented by a three-dimensional scatter plot, and the human skeleton is formed by connecting with line segments, which is convenient for users to view and analyze the pose changes. During this process, the three-dimensional model can be updated in real time to reflect the changes in the human body pose, providing an intuitive interactive display.

[0090] At regular time intervals, the system will regularly read the latest key point data of each camera, perform 3D reconstruction, and update the skeleton model. In this way, the system can track human movements in real time in a dynamic environment and provide accurate 3D pose information. This process not only provides data support for robot control but also provides an important technical foundation for other applications such as virtual reality and action recognition.

[0091] In addition, for the part of pose recognition and 3D reconstruction, the present invention can also adopt deep learning-assisted 3D reconstruction: If it is necessary to further improve the 3D reconstruction accuracy, deep learning methods can be combined, and convolutional neural networks (CNNs) or graph neural networks (GNNs) can be used to optimize 3D reconstruction. Such methods can provide more accurate 3D reconstruction results in complex scenarios. Traditional 3D reconstruction methods can also be used: In addition to the triangulation method, traditional computer vision methods such as stereo vision or multi-view geometry can also be used for 3D reconstruction.

[0092] Furthermore, obtain the spatial vectors corresponding to each key point in the human 3D pose estimation result, and calculate the vector angles between the spatial vectors; determine the target type angles according to the vector angles, where the target type angles include 3D angles, 2D plane angles, and direction angles; determine the target robotic arm, and map the target type angles to the target robotic arm so that the target robotic arm completes the specified actions of the target human body.

[0093] As Figure 2 shown, the core objective of action control mapping is to map the calculated joint angles to the control requirements of the robotic arm, enabling the system to adapt to different robotic arms. Through this mapping, the system can flexibly control the movement of the robotic arm based on the angle data of the joints without having to recalculate and adjust for each robotic arm.

[0094] First, the present invention automatically selects a suitable angle type (including 3D angles, 2D plane angles, and direction angles) to control each degree of freedom of the robotic arm according to the calculated angle information (by calling the spatial position points of each joint point calculated previously, calculating the spatial vectors, and then obtaining the angle information by calculating the angles between the spatial vectors). By mapping the calculated 3D angles, 2D plane angles, and direction angles (expressing the detected 3D angles and 2D angles of the arm with the rotations of the six axes of the robotic arm and mapping the actions of the human arm to the actions of the robotic arm) to the specific joints of the robotic arm, the system can flexibly control the movement of the robotic arm.

[0095] After obtaining the required angle data (i.e., the "3D angles, 2D plane angles, and direction angles" mentioned earlier), the system will transmit the angle data to the robotic arm control program via TCP communication. Through the router, the angle information can be wirelessly transmitted to the remote robotic arm control system. Using the TCP protocol, the system can ensure the stable transmission of angle data and real-time update of the robotic arm's posture.

[0096] Each time the angle data read from the JSON file will be sent to the robotic arm control system, and the control system will calculate and adjust the joint angles of the robotic arm according to the received data. This process ensures that the system can accurately perform the conversion from human movements to robotic arm movements.

[0097] As Figure 2 shown, after the control program receives the angle data, it will parse and adjust the joint angles of the robotic arm. The angle calculation of each joint is based on the required angle information. The system calls the corresponding angle data according to the task requirements to ensure that the joints of the robotic arm can accurately rotate and complete the specified actions.

[0098] During the execution process, the system will dynamically update the angle data to ensure that the robotic arm can respond to real-time changing requirements. By continuously updating and transmitting the angle data, the system can achieve precise control of the robotic arm and complete the simulation and execution of complex actions.

[0099] Through action control mapping, the system can flexibly map the calculated angle data to each degree of freedom of the robotic arm. Through the transmission of angle data, the design of a standardized control interface, and the real-time transmission of TCP communication, the system can precisely control the actions of the robotic arm and achieve synchronization with human movements. This mapping mechanism ensures the high adaptability of the system, enabling precise action control between different robotic arms, thus expanding the application scope and flexibility of the system.

[0100] For the part of action control mapping, the present invention can also adopt kinematics-based control: in addition to direct control through angle data, the inverse kinematics (IK) algorithm can also be used for precise control of the robotic arm, which is particularly effective in the complex control of multi-degree-of-freedom robotic arms. It can also adopt action imitation-based control: the system can be combined with the action imitation (Imitation Learning) method to control the robotic arm by imitation learning and simulate human actions.

[0101] The main innovation points of the present invention are as follows:

[0102] 1. Multi - perspective camera configuration and layout: The present invention uses three cameras, which are respectively installed in the front, right, and front - right of the human body. By means of the multi - perspective layout, the blind - spot problem of a single - perspective camera is solved. Through the data fusion of multiple cameras, the movement of each joint can be accurately captured.

[0103] 2. Efficient key - point data synchronization: The present invention optimizes the synchronous reading of video streams through multi - thread technology and frame - rate control, enabling each camera to capture video streams in parallel and perform key - point extraction, thereby effectively reducing calculation latency and resource consumption, and ensuring the real - time performance and stability of the system.

[0104] 3. Improvement in the accuracy of 3D pose reconstruction: Using the triangulation method and combining data from multiple perspectives for 3D reconstruction, the system can calculate the actual position of the same key - point in 3D space based on the 2D coordinates captured by different cameras. This method improves the accuracy of 3D reconstruction, especially in low - cost RGB camera systems where traditional depth - perception technologies cannot be realized.

[0105] 4. Adaptability of motion control mapping and robotic - arm control: By mapping the calculated angle data (including 3D angles, 2D included angles, and direction angles) to the degrees of freedom of the robotic arm, the system can flexibly adapt to different types of robotic arms, ensuring accuracy under various robotic - arm control requirements. Using a standardized control interface, the system can automatically select appropriate angle data for transmission.

[0106] 5. Low - cost solution: Without using a depth camera, the present invention combines RGB cameras with the triangulation method and multi - perspective data fusion to achieve high - precision 3D reconstruction and pose estimation, greatly reducing the hardware cost and being applicable to application scenarios with limited budgets.

[0107] Possible design change directions or deformation schemes of the present invention include:

[0108] 1. Increasing the number of cameras: If higher system accuracy is required, the number of cameras can be increased to further expand the view - coverage range. For example, cameras can be added to the back, left, or above the human body to improve the ability to capture complex movements.

[0109] 2. Combining with depth cameras or lidar: If higher requirements for depth perception exist, depth cameras (such as Microsoft Kinect or Intel RealSense) or lidar devices can be introduced. By providing more accurate depth information, the accuracy of 3D reconstruction can be improved, especially in cases with more occlusions or longer distances.

[0110] 3. Hardware Synchronization Technology: In addition to multi-threaded software synchronization, precise synchronization between cameras can be achieved through hardware synchronization devices (such as Genlock), ensuring that each camera captures images simultaneously at the physical level and avoiding deviations caused by time misalignment.

[0111] 4. Pose Estimation and Reconstruction Based on Deep Learning: Currently, traditional methods such as MediaPipe and OpenCV are used in this invention for key point extraction and pose reconstruction. However, the estimation accuracy can be further improved through deep learning techniques (such as Convolutional Neural Network (CNN) or Graph Neural Network (GNN)), especially the ability to capture pose changes in complex environments.

[0112] 5. Deep Learning-Assisted 3D Reconstruction: In addition to the traditional triangulation method, deep learning models (such as 3D CNN or PointNet, etc.) can be combined for 3D reconstruction, thereby further improving the accuracy. Especially in dynamic and complex scenes, deep learning can help optimize the results of 3D reconstruction.

[0113] 6. Manipulator Control Based on Kinematic Algorithms: When controlling a manipulator, in addition to using simple angle mapping methods, inverse kinematics (IK) algorithms can be combined. The precise joint angles of the manipulator can be calculated through optimized algorithms. Especially when the manipulator has more degrees of freedom or the task is more complex, the inverse kinematics method can provide more precise joint control.

[0114] 7. Combining Action Imitation Technology: Action Imitation (Imitation Learning) technology can be combined with the system of this invention to enable the manipulator to be controlled by imitating human actions. Through reinforcement learning, the manipulator can autonomously learn how to adjust its actions according to human postures without having to calculate precise joint angles every time, thus improving the flexibility of control.

[0115] This invention also considers some optimization and extension directions, including introducing GPU acceleration to improve the calculation speed, especially for computationally intensive tasks such as key point detection and 3D reconstruction. The system can also adaptively optimize data processing and control strategies, dynamically adjusting resource allocation and calculation frequency according to different application scenarios.

[0116] To improve the adaptability and flexibility of the system, the system can achieve cross-platform adaptation, supporting different types of hardware devices such as depth cameras, panoramic cameras, and motion capture devices, thereby expanding the application scope, including fields such as virtual reality, augmented reality, and intelligent robots. By introducing multi-modal data fusion, the system can enhance the intelligent perception ability, especially providing more accurate pose estimation in complex environments.

[0117] In addition, the system can also be extended to large-scale application scenarios, such as sports training analysis, medical rehabilitation, and security monitoring. Through multi-view data fusion and 3D pose reconstruction, it can monitor human movements in real time and provide personalized feedback. In terms of sustainability, the system reduces energy consumption through low-power design, combined with hardware and software optimization, ensuring high efficiency and reduced environmental burden during long-term operation. These supplementary viewpoints and ideas not only bring innovation to the fields of pose estimation and robot control but also provide more possibilities for future technology expansion and multi-field applications.

[0118] Furthermore, as Figure 4 shown, based on the above-mentioned human 3D pose estimation method in a multi-view scenario, the present invention also correspondingly provides a human 3D pose estimation system in a multi-view scenario. Among them, the human 3D pose estimation system in a multi-view scenario includes:

[0119] A multi-view camera setting module 51, configured to determine multi-view cameras, and perform video stream synchronization reading settings and camera calibration processing on the multi-view cameras to obtain target multi-view cameras;

[0120] A key point extraction module 52, configured to collect a target human video stream through the target multi-view cameras, and perform key point extraction processing on the target human video stream to obtain initial human key points;

[0121] An image distortion removal module 53, configured to perform image distortion removal processing on the initial human key points to obtain target human key points;

[0122] A pose estimation result generation module 54, configured to perform pose recognition processing and 3D reconstruction processing according to the target human key points to obtain a human 3D pose estimation result.

[0123] Furthermore, as Figure 5 shown, based on the above-mentioned human 3D pose estimation method and system in a multi-view scenario, the present invention also correspondingly provides a terminal, and the terminal includes a processor 10, a memory 20, and a display 30. Figure 5 Only some components of the terminal are shown, but it should be understood that it is not required to implement all the shown components, and more or fewer components can be alternatively implemented.

[0124] In some embodiments, the memory 20 may be an internal storage unit of the terminal, such as the hard disk or memory of the terminal. In some other embodiments, the memory 20 may also be an external storage device of the terminal, such as a plug-in hard disk equipped on the terminal, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc. Further, the memory 20 may also include both the internal storage unit and the external storage device of the terminal. The memory 20 is used to store the application software installed on the terminal and various types of data, such as the program code for installing the terminal. The memory 20 may also be used to temporarily store the data that has been output or will be output. In one embodiment, a human body three-dimensional pose estimation program 40 based on a multi-view scenario is stored on the memory 20, and the human body three-dimensional pose estimation program 40 based on the multi-view scenario can be executed by the processor 10, so as to implement the human body three-dimensional pose estimation method based on the multi-view scenario in the present application.

[0125] In some embodiments, the processor 10 may be a central processing unit (CPU), a microprocessor or other data processing chips, and is used to run the program code stored in the memory 20 or process data, such as executing the human body three-dimensional pose estimation method based on the multi-view scenario, etc.

[0126] In some embodiments, the display 30 may be an LED display, a liquid crystal display, a touch liquid crystal display, and an OLED (Organic Light-Emitting Diode) toucher, etc. The display 30 is used to display the information on the terminal and to display a visual user interface.

[0127] In one embodiment, when the processor 10 executes the human body three-dimensional pose estimation program 40 based on the multi-view scenario in the memory 20, the steps of the human body three-dimensional pose estimation method based on the multi-view scenario are implemented.

[0128] In summary, the present invention provides a method, system and terminal for human three-dimensional pose estimation based on multi-view scenarios. The method includes: determining multi-view cameras, and performing video stream synchronous reading settings and camera calibration processing on the multi-view cameras to obtain target multi-view cameras; collecting a target human video stream through the target multi-view cameras, and performing key point extraction processing on the target human video stream to obtain initial human key points; performing image distortion correction processing on the initial human key points to obtain target human key points; performing pose recognition processing and three-dimensional reconstruction processing based on the target human key points to obtain a human three-dimensional pose estimation result. By using the target multi-view cameras after video stream synchronous reading settings and camera calibration processing to collect the target human video stream, and performing key point extraction, image distortion correction, pose recognition and three-dimensional reconstruction processing on the target human video stream, the present invention can obtain an accurate and real-time human three-dimensional pose estimation result, effectively improving the accuracy of pose estimation by multi-cameras in dynamic scenarios.

[0129] It should be noted that in this article, the terms "including", "comprising" or any other variant thereof are intended to cover non-exclusive inclusion, so that a process, method, article or terminal including a series of elements not only includes those elements, but also includes other elements not expressly listed, or further includes elements inherent to such process, method, article or terminal. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article or terminal including the element.

[0130] Of course, those of ordinary skill in the art can understand that all or part of the processes of implementing the above method embodiments can be completed by instructing relevant hardware (such as a processor, a controller, etc.) through a computer program. The program can be stored in a computer-readable storage medium readable by a computer. When the program is executed, it can include the processes of the above method embodiments. The computer-readable storage medium can be a memory, a magnetic disk, an optical disc, etc.

[0131] It should be understood that the application of the present invention is not limited to the above examples. For those of ordinary skill in the art, improvements or transformations can be made according to the above description. All such improvements and transformations should fall within the protection scope of the appended claims of the present invention.

Claims

1. A method for human three-dimensional pose estimation based on multi-view scenarios, characterized in that The human body three-dimensional pose estimation method based on a multi-view scenario includes: Determine multi-view cameras, and perform video stream synchronous reading settings and camera calibration processing on the multi-view cameras to obtain target multi-view cameras; Collect a target human body video stream through the target multi-view cameras, and perform key point extraction processing on the target human body video stream to obtain initial human body key points; Perform image undistortion processing on the initial human body key points to obtain target human body key points; Perform pose recognition processing and three-dimensional reconstruction processing based on the target human body key points to obtain a human body three-dimensional pose estimation result.

2. The method for human three-dimensional pose estimation based on multi-view scenarios according to claim 1, wherein, The step of determining multi-view cameras, and performing video stream synchronous reading settings and camera calibration processing on the multi-view cameras to obtain target multi-view cameras specifically includes: Set multiple preset positions, and set cameras at the multiple preset positions to obtain multi-view cameras, where the preset positions include directly in front of, diagonally in front of the right, and directly to the right of the target human body; Use a multi-threaded method to perform video stream synchronous reading settings on the multi-view cameras, and use the checkerboard calibration method to perform camera calibration processing on the multi-view cameras to obtain camera internal parameters; When the multi-view cameras complete the video stream synchronous reading settings and the camera calibration processing, target multi-view cameras are obtained.

3. The method for human three-dimensional pose estimation based on multi-view scenarios according to claim 2, characterized in that, The step of using the checkerboard calibration method to perform camera calibration processing on the multi-view cameras to obtain camera internal parameters specifically includes: Set a preset checkerboard, and obtain multi-angle checkerboard images of the preset checkerboard; Extract the checkerboard corner points of the multi-angle checkerboard images, and perform quality optimization processing on the checkerboard corner points to obtain target checkerboard corner point coordinates; Obtain the calibration size information of the preset checkerboard, and use the non-linear optimization solution method to calculate based on the calibration size information and the target checkerboard corner point coordinates to obtain camera internal parameters, where the camera internal parameters include focal length, principal point position, and distortion coefficients.

4. The method for estimating the three-dimensional human body posture based on multi-view scenarios according to claim 1, characterized in that The step of collecting a target human body video stream through the target multi-view cameras, and performing key point extraction processing on the target human body video stream to obtain initial human body key points specifically includes: Collect a video stream through the target multi-view cameras to obtain an initial human body video stream, and perform size adjustment processing and format conversion processing on the initial human body video stream to obtain a target human body video stream; Perform first key point extraction processing on the target human body video stream through a MediaPipe pose estimation model to obtain human joint key points; Perform second key point extraction processing on the target human body video stream through a MediaPipe hand estimation model to obtain hand key points; Perform absolute coordinate conversion processing on the human joint key points and the hand key points to obtain initial human body key points.

5. The method for human three-dimensional pose estimation based on multi-view scenarios according to claim 3, wherein The step of performing image undistortion processing on the initial human body key points to obtain target human body key points is specifically: Obtain the key point video where the initial human body key points are located, and use an undistortion algorithm to eliminate radial distortion and tangential distortion for each frame image in the key point video according to the camera internal parameters to obtain target human body key points.

6. The method for human three-dimensional pose estimation based on a multi-view scenario according to claim 1, wherein, Performing pose recognition processing and three-dimensional reconstruction processing according to the target human body key points to obtain a human body three-dimensional pose estimation result, specifically including: Obtaining the two-dimensional coordinates of the key points corresponding to the target human body key points, and performing coordinate reconstruction processing and coordinate conversion processing on the two-dimensional coordinates of the key points to obtain the three-dimensional coordinates of the key points; Performing three-dimensional reconstruction processing on the three-dimensional coordinates of the key points to obtain a human body three-dimensional pose estimation result.

7. The method for human three-dimensional pose estimation based on multi-view scenarios according to claim 2, wherein After performing pose recognition processing and three-dimensional reconstruction processing according to the target human body key points to obtain a human body three-dimensional pose estimation result, it further includes: Obtaining the spatial vectors corresponding to each key point in the human body three-dimensional pose estimation result, and calculating the vector angles between the spatial vectors; Determining the target type angles according to the vector angles, where the target type angles include three-dimensional angles, two-dimensional plane angles, and direction angles; Determining the target robotic arm, and mapping the target type angles to the target robotic arm to enable the target robotic arm to complete the specified actions of the target human body.

8. A human body three-dimensional pose estimation system based on a multi-view scenario, characterized in that, The human body three-dimensional pose estimation system based on a multi-view scenario includes: A multi-view camera setting module, configured to determine multi-view cameras, and perform video stream synchronous reading setting and camera calibration processing on the multi-view cameras to obtain target multi-view cameras; A key point extraction module, configured to collect a target human body video stream through the target multi-view cameras, and perform key point extraction processing on the target human body video stream to obtain initial human body key points; An image distortion removal module, configured to perform image distortion removal processing on the initial human body key points to obtain target human body key points; A pose estimation result generation module, configured to perform pose recognition processing and three-dimensional reconstruction processing according to the target human body key points to obtain a human body three-dimensional pose estimation result.

9. A terminal, characterized in that, The terminal includes: a memory, a processor, and a human body three-dimensional pose estimation program based on a multi-view scenario stored on the memory and executable on the processor. When the human body three-dimensional pose estimation program based on a multi-view scenario is executed by the processor, it implements the steps of the human body three-dimensional pose estimation method according to any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a human body three-dimensional pose estimation program based on a multi-view scenario. When the human body three-dimensional pose estimation program based on a multi-view scenario is executed by a processor, it implements the steps of the human body three-dimensional pose estimation method according to any one of claims 1-7.

Images