Human body joint point positioning method, device, equipment and product based on binocular vision

Abstract

The embodiment of the invention discloses a human body joint point positioning method and device based on binocular vision, equipment and a product. The method comprises the steps that a first joint point set in a first human body image and a second joint point set in a second human body image are extracted through a posture estimation algorithm; the first human body image and the second human body image are acquired through a binocular camera; matching first articulation points in the first articulation point set with second articulation points in the second articulation point set to obtain a plurality of matching pairs; for each matching pair, constructing a triangulation equation based on the matching pair and a projection matrix of the binocular camera, and solving the triangulation equation to obtain a three-dimensional coordinate of the matching pair; the projection matrix is constructed through an internal parameter matrix and an external parameter matrix of the binocular camera; and determining a human body joint point positioning result according to the three-dimensional coordinates of each matching pair. According to the technical scheme, the first articulation point in the first human body image is matched with the second articulation point in the second human body image, so that the articulation point positioning efficiency and accuracy are improved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image recognition technology, and in particular to a method, device, equipment and product for locating human joints based on binocular vision. Background Art

[0002] Human joint detection and positioning is an important research direction in the field of computer vision, and is widely used in intelligent monitoring, motion analysis, medical rehabilitation, occupational health and other fields. The core purpose of this technology is to accurately identify and locate the key points of the human body, such as the head, shoulders, wrists, knees, etc., from images or videos. Traditionally, this task mainly relies on monocular image deep learning processing technology. However, monocular vision systems often have difficulty accurately identifying human key points when processing complex posture changes. In addition, they cannot provide three-dimensional spatial positioning information of human key points.

[0003] The existing technology often matches the joint points of the human body under the binocular perspective through template matching methods, feature matching methods or deep learning methods, and then calculates the three-dimensional positioning information through the matching results. The accuracy of the matching results greatly affects the final positioning results. The template matching method is sensitive to changes in the scale, features, color, brightness, etc. of the image, and the template matching is sensitive to the interference of the background. If the background is complex, it is easy to mismatch. The feature matching method extracts the key points in the image (such as Scale-Invariant Feature Transform (SIFT), Speeded Up Robust Features (SURF), etc.) and matches them to identify the target. The feature matching method is sensitive to changes in illumination and has high computational complexity. When the target part has few features or is occluded, the feature matching effect will be significantly reduced because the key points may not be reliably extracted for matching. The deep learning method can cope with the challenges of image scale, features, color, etc. very well, and can also show good prediction ability after the target part is slightly occluded. However, deep learning methods have large data requirements, high computing resource requirements, and long training times. In addition, due to the high complexity and structure of the model, the design and debugging require rich experience and expertise, and the learning cost is high. Summary of the invention

[0004] The embodiments of the present disclosure provide a method, device, equipment and product for locating human joints based on binocular vision, which improves the efficiency and accuracy of joint positioning.

[0005] In a first aspect, a method for locating human joints based on binocular vision is provided, comprising:

[0006] Extracting a first set of joint points in a first human body image and a second set of joint points in a second human body image by using a posture estimation algorithm; the first human body image and the second human body image are acquired by a binocular camera;

[0007] Matching a first joint point in the first joint point set with a second joint point in the second joint point set to obtain a plurality of matching pairs;

[0008] For each of the matching pairs, a triangulation equation is constructed based on the projection matrix of the matching pair and the binocular camera, and the three-dimensional coordinates of the matching pair are obtained by solving the triangulation equation; the projection matrix is ​​constructed by the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera;

[0009] The human body joint point positioning results are determined according to the three-dimensional coordinates of each matching pair.

[0010] In a second aspect, a human joint positioning device based on binocular vision is provided, comprising:

[0011] A joint point set extraction module, used to extract a first joint point set in a first human body image and a second joint point set in a second human body image by using a posture estimation algorithm; the first human body image and the second human body image are acquired by a binocular camera;

[0012] a matching module, configured to match a first joint point in the first joint point set with a second joint point in the second joint point set to obtain a plurality of matching pairs;

[0013] A solution module, for each of the matching pairs, constructing a triangulation equation based on the matching pair and a projection matrix of a binocular camera, and solving the triangulation equation to obtain the three-dimensional coordinates of the matching pair; the projection matrix is ​​constructed by an intrinsic parameter matrix and an extrinsic parameter matrix of the binocular camera;

[0014] The result determination module is used to determine the human body joint point positioning result according to the three-dimensional coordinates of each matching pair.

[0015] In a third aspect, an electronic device is provided, including:

[0016] one or more processors;

[0017] a storage device for storing one or more programs,

[0018] When the one or more programs are executed by the one or more processors, the one or more processors implement the human joint positioning method based on binocular vision as described in the first aspect above.

[0019] In a fourth aspect, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method for locating human joints based on binocular vision as described in the first aspect above is implemented.

[0020] In a fifth aspect, a computer program product is provided, the computer program product comprising a computer program, and when the computer program is executed by a processor, the method for locating human joints based on binocular vision as described in the first aspect above is implemented.

[0021] The disclosed embodiment discloses a method, device, equipment and product for locating human joints based on binocular vision, including: extracting a first joint point set in a first human image and a second joint point set in a second human image by using a posture estimation algorithm; the first human image and the second human image are obtained by a binocular camera; matching the first joint point in the first joint point set with the second joint point in the second joint point set to obtain multiple matching pairs; for each matching pair, constructing a triangulation equation based on the projection matrix of the matching pair and the binocular camera, and solving the triangulation equation to obtain the three-dimensional coordinates of the matching pair; the projection matrix is ​​constructed by the internal parameter matrix and the external parameter matrix of the binocular camera; determining the human joint point positioning result according to the three-dimensional coordinates of each matching pair. This technical solution solves the problems of posture, illumination and background interference in the single image matching method by matching the first joint point and the second joint point in the image taken by the binocular camera, realizes high-precision binocular image matching and human motion function evaluation, improves the efficiency and accuracy of joint point positioning, and reduces the complexity of calculation, providing reliable data support and analysis tools for the intelligent human motion function evaluation system.

[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the embodiments of the present disclosure. Other features of the embodiments of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0024] Figure 1 It is a flow chart of a method for locating human joints based on binocular vision provided in the first embodiment of the present disclosure;

[0025] Figure 2The first disclosed embodiment provides a process schematic diagram of a method for locating human joints based on binocular vision;

[0026] Figure 3 is a schematic diagram of a joint point matching process provided by the first embodiment of the present disclosure;

[0027] Figure 4 It is a structural schematic diagram of a human joint positioning device based on binocular vision provided in the second embodiment of the present disclosure;

[0028] Figure 5 It is a structural diagram of an electronic device provided in Embodiment 3 of the present disclosure. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the solutions of the embodiments of the present disclosure, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are only embodiments of a part of the embodiments of the present disclosure, rather than all of the embodiments. Based on the embodiments in the embodiments of the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work should fall within the scope of protection of the embodiments of the present disclosure.

[0030] It should be noted that the terms "first", "second", etc. in the specification and claims of the embodiments of the present disclosure and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present disclosure described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions, for example, a process, method, system, product, or device that includes a series of steps or units is not necessarily limited to those steps or units that are clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products, or devices.

[0031] Embodiment 1

[0032] Figure 1 The flowchart of a method for locating human joints based on binocular vision is provided in the first embodiment of the present disclosure. This embodiment is applicable to the situation of locating human joints. The method can be performed by a human joint positioning device based on binocular vision. The human joint positioning device based on binocular vision can be implemented in the form of hardware and / or software. The human joint positioning based on binocular vision can be configured in electronic devices, including but not limited to computers, computers, terminals, servers and other devices with data processing capabilities. Figure 1 As shown, the method includes:

[0033] S110, extracting a first joint point set in the first human body image and a second joint point set in the second human body image by using a posture estimation algorithm; the first human body image and the second human body image are acquired by a binocular camera.

[0034] In this embodiment, the posture estimation algorithm can be used to help computer vision understand the movement or posture of the human body in an image or video, and the posture estimation algorithm can be used to extract key point features of the human body posture. Exemplarily, the posture estimation algorithm can be a MediaPipe human body posture estimation algorithm or a ViTPose posture estimation algorithm.

[0035] Specifically, a binocular camera can be used to capture images of a human body at different angles. The captured images of a human body may include a first image of a human body and a second image of a human body. It can be understood that the first image of a human body and the second image of a human body are images of a human body captured by a binocular camera at different viewing angles. Among them, a binocular camera is a device that uses two cameras to capture the same scene from different viewing angles. This setting can provide depth information, thereby achieving more accurate spatial perception and object positioning. The binocular camera can obtain images of a human body from two viewing angles, and can distinguish the images of a human body from two viewing angles of the same human body according to the left and right positions. The first image of a human body may be an image of a human body from the left viewing angle, and the second image of a human body may be an image of a human body from the right viewing angle.

[0036] The joint points in the first human body image can be extracted using a posture estimation algorithm to obtain a first joint point set, wherein the first joint point set includes a fixed number of first joint points; the joint points in the second human body image can be extracted using the same posture estimation algorithm as that of the first human body image to obtain a second joint point set, wherein the second joint point set has the same number of joint points as the first joint point set.

[0037] S120. Match the first joint point in the first joint point set with the second joint point in the second joint point set to obtain a plurality of matching pairs.

[0038] Specifically, after obtaining the first joint point set and the second joint point set, for any first joint point in the first joint point set, the first joint point can be matched with the second joint point in the second joint point set. Specifically, the polar line of the first joint point in the second human body image can be used to filter out the joint point closest to the polar line, and the joint point can be used as the second joint point, and the filtered second joint point can be matched with the first joint point; a preset threshold can also be given, and the joint point in the second joint point set whose distance from the polar line meets the preset threshold can be used as the second joint point.

[0039] Continuing with the above description, by matching each first joint point in the first joint point set, multiple matching pairs can be obtained.

[0040] S130. For each matching pair, a triangulation equation is constructed based on the matching pair and the projection matrix of the binocular camera, and the triangulation equation is solved to obtain the three-dimensional coordinates of the matching pair; the projection matrix is ​​constructed by the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera.

[0041] In this embodiment, after obtaining multiple matching pairs, for any matching pair, a triangulation equation can be constructed based on the matching pair and the projection matrix of the binocular camera, wherein the triangulation equation is a basic equation used in stereo vision to recover three-dimensional space points from two-dimensional image coordinates. These equations are based on the principle of perspective projection and determine the position of a space point by analyzing corresponding points in two or more views. The projection matrix of the binocular camera is a matrix used to map three-dimensional world coordinates to two camera image planes. In stereo vision, each camera has a projection matrix, which describes the imaging geometry and perspective transformation of the camera.

[0042] Continuing with the above description, after the triangulation equation is obtained, the triangulation equation can be solved, such as using a linear triangulation method (such as a direct linear transformation algorithm) or a nonlinear optimization method to solve the triangulation equation and obtain the three-dimensional coordinates corresponding to the matching points.

[0043] Specifically, the coordinates of the matching pair can be expressed as (x l ,y l ) and (x r ,y r ), through the internal parameter matrix K and the external parameter matrix [R|t], the projection matrices of the two cameras can be expressed as p l and p r . Normalize the coordinates of the matching pairs and Then the triangulation equations based on the projection matrix of the matching pair and the binocular camera can be expressed as: Among them, X is the three-dimensional coordinate, λ l and λ r is the corresponding scale factor of the two coordinates, R in the external parameter matrix [R|t] can be a rotation matrix, and t can be a translation vector. After obtaining the triangulation equations, the obtained triangulation equations can be solved. Specifically, the solution process of the triangulation equations can include: setting a linear system AX=B and solving it using numerical methods (such as singular value decomposition (SVD) and other methods). The cv2.triangulatePoints function in OpenCV can be used for triangulation calculations.

[0044] S140, determining the human body joint location results according to the three-dimensional coordinates of each matching pair.

[0045] Specifically, after obtaining the three-dimensional space coordinates corresponding to each matching pair, the positioning results of the human joints can be determined according to the three-dimensional coordinates. It should be noted that after obtaining the positioning results of the human joints, the posture and joint position of the human body can be estimated according to the positioning results, and the movement trajectory and posture change trend of the human body can also be analyzed according to the matching results in continuous image frames.

[0046] The present embodiment provides a method for locating human joints based on binocular vision, comprising: extracting a first joint set in a first human image and a second joint set in a second human image by using a posture estimation algorithm; acquiring the first human image and the second human image by using a binocular camera; matching the first joint in the first joint set with the second joint in the second joint set to obtain a plurality of matching pairs; for each matching pair, constructing a triangulation equation based on the projection matrix of the matching pair and the binocular camera, and solving the triangulation equation to obtain the three-dimensional coordinates of the matching pair; the projection matrix is ​​constructed by using the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera; determining the human joint positioning result according to the three-dimensional coordinates of each matching pair, thereby realizing high-precision binocular image matching and human motion function evaluation, improving matching efficiency and accuracy, and providing reliable data support and analysis tools for an intelligent human motion function evaluation system.

[0047] As an optional implementation of this embodiment, matching the first joint point in the first joint point set with the second joint point in the second joint point set to obtain a plurality of matching pairs includes:

[0048] a1) for each of the first joint points in the first joint point set, calculating the epipolar line of the first joint point in the second human body image using a fundamental matrix;

[0049] In this embodiment, the epipolar line can be a description of how a point in one camera image is represented in another camera image in a binocular camera system. The epipolar line is a straight line parallel to the optical axes of the two cameras. If a point is in the image of the left camera, its corresponding point (if any) must be located on the corresponding epipolar line in the right camera image. For each first joint point in the first joint point set, the epipolar line of the first joint point in the second human body image can be calculated using the basic matrix.

[0050] Specifically, the epipolar line can be calculated through the first joint point and the basic matrix, the basic matrix is ​​used to describe the geometric relationship between the corresponding points in the first human body image and the second human body image, and the basic matrix can be obtained through stereo calibration of the binocular camera.

[0051] For example, the basic matrix can be represented as F, and the nth first joint point in the first joint point set can be represented as x n , then the polar line l corresponding to the node can be expressed as: l = Fx n . By utilizing the geometric relationship of the binocular cameras and using epipolar constraints, the matching search range is reduced, thus improving the matching efficiency and accuracy.

[0052] b1) determining a second joint point in the second joint point set corresponding to the first joint point based on the polar line, and the first joint point and the second joint point are regarded as a matching pair; wherein the basic matrix is ​​used to describe the geometric relationship between corresponding points in the first human body image and the second human body image.

[0053] Specifically, after obtaining the epipolar line, the second joint point corresponding to the first joint point in the second joint point set can be determined according to the epipolar line, and the first joint point and its corresponding second joint point are used as a matching pair. The basic matrix in this embodiment can be used to describe the geometric relationship between corresponding points in the first human body image and the second human body image.

[0054] Optionally, determining, according to the polar line, a second joint point in the second joint point set corresponding to the first joint point includes:

[0055] a2) respectively calculating the distances from the second joint points in the second joint point set to the polar lines;

[0056] Specifically, for each joint point in the first human image, its corresponding matching point should be located on the corresponding epipolar line in the right image, narrowing the search range. After the epipolar line is obtained by calculating the basic matrix, the distance between the second joint point in the second joint point set and the epipolar line can be directly calculated;

[0057] The distance between the joint points in the second joint point set located around the polar line and the polar line can also be calculated, wherein when the distance between the joint point and the polar line is within a preset threshold range, the joint point can be considered to be located around the polar line, thereby realizing the screening of the second joint points in the second joint point set to reduce the amount of calculation.

[0058] b2) taking the second joint point which is closest to the polar line as the second joint point corresponding to the first joint point.

[0059] Continuing from the above description, the joint point with the shortest distance from the polar line (i.e., the joint point with the shortest vertical distance from the polar line) can be used as the second joint point corresponding to the first joint point. By using the polar line constraint, the range of the searched joint points in the second joint point set is narrowed, the amount of calculation is reduced, the robustness and accuracy of the matching process can be improved, and the calculation complexity can be reduced.

[0060] As an optional implementation of this embodiment, the method further includes:

[0061] a3) Calibrate the binocular camera based on the camera calibration program of the open source computer vision library to obtain the intrinsic parameter matrix and extrinsic parameter matrix corresponding to the binocular camera;

[0062] Specifically, camera calibration is a key step in achieving accurate human joint detection and positioning. The purpose of camera calibration is to determine the internal parameters of the camera (such as focal length, optical center, distortion coefficient) and external parameters (relative position and orientation of the camera), etc. In this embodiment, the binocular camera can be calibrated based on the camera calibration program of the open source computer vision library in conjunction with a regular checkerboard calibration plate to determine the internal parameter matrix and external parameter matrix corresponding to the binocular camera. The checkerboard pattern can be composed of black and white or different colored squares, providing clear and accurate corner features, which is convenient for computer vision algorithm detection and recognition, thereby performing image acquisition and calculation of the camera calibration program. Exemplarily, the checkerboard calibration plate can be 12×9 squares, and the size of each square can be 60mm×60mm).

[0063] Among them, the intrinsic parameter matrix can describe the conversion relationship from the three-dimensional world coordinate system to the camera coordinate system, and can be used to project points in the three-dimensional space onto the two-dimensional image plane. The intrinsic parameter matrix can be a 3×3 matrix; the extrinsic parameter matrix can describe the conversion relationship from the camera coordinate system to the world coordinate system, and can be used to determine the actual position and orientation of the camera in space. The intrinsic parameter matrix can include focal length, principal point coordinates and / or distortion coefficients, etc. The extrinsic parameter matrix can be a 4×4 matrix, which contains the position and orientation of the camera relative to the world coordinate system, that is, the rotation matrix and the translation vector. The extrinsic parameter matrix can include the rotation matrix and / or the translation vector. Among them, the rotation matrix can be used to represent the rotation of one coordinate system relative to another coordinate system, and can describe the change in direction from one coordinate system to another coordinate system; the translation vector can be used to represent the translation distance of one coordinate system relative to another coordinate system, and can describe the change in the spatial position of the origin of the coordinate system. It should be noted that the camera calibration program based on the open source computer vision library can calibrate the binocular camera, and the stereo calibration parameters corresponding to the binocular camera can also be obtained. The stereo calibration parameters can include the basic matrix and the intrinsic matrix. The stereo calibration parameters can be various parameters used to describe the relative position and direction of the two cameras in the stereo vision system. The basic matrix can be used to describe the correspondence between the midpoints of the two camera images. The intrinsic matrix can contain the relative rotation and translation information between the cameras. The intrinsic matrix can also contain the intrinsic parameter information of the camera. The intrinsic matrix can be understood as a combination of the basic matrix and the intrinsic parameter matrix.

[0064] b3) Using the intrinsic parameter matrix and the extrinsic parameter matrix to obtain the corresponding projection matrix.

[0065] Specifically, after obtaining the intrinsic parameter matrix and the extrinsic parameter matrix, the projection matrix corresponding to the binocular camera can be calculated according to the intrinsic parameter matrix and the extrinsic parameter matrix. For example, the intrinsic parameter matrix can be expressed as A, the extrinsic parameter matrix can be expressed as B, and the projection matrix can be expressed as C, then C = A × B.

[0066] As an optional implementation of this embodiment, before extracting the first joint point set in the first human image and the second joint point set in the second human image using a posture estimation algorithm, the method further includes:

[0067] a4) using a binocular camera to collect a human body image, wherein the human body image includes a first initial human body image and a second initial human body image;

[0068] In this embodiment, a binocular camera may also be used to collect human images, and the human images include a first initial human image and a second initial human image, wherein the first initial human image and the second initial human image are images that have not been preprocessed.

[0069] b4) Preprocessing the first initial human body image to obtain the first human body image, and preprocessing the second initial human body image to obtain the second human body image; the preprocessing includes: grayscale, denoising and histogram equalization.

[0070] Specifically, after obtaining the first initial human body image and the second initial human body image, the first initial human body image can be preprocessed to obtain the first human body image; the second initial human body image can be preprocessed to obtain the second human body image, and the processing operations may include grayscale, denoising and histogram equalization.

[0071] As an optional implementation of this embodiment, the method of this embodiment further includes:

[0072] The human body joint point positioning results are displayed using a visual display device.

[0073] Specifically, after obtaining the three-dimensional coordinates of the matching pairs and determining the human joint positioning results based on the three-dimensional coordinates, the three-dimensional coordinates of each matching pair can be combined into a 3D point cloud data set, and the human joint positioning device results can be displayed using a visualization device, wherein the visualization device can be a device that can display data, information or images in an intuitive and graphical manner, such as: a display, a projector, a touch screen and / or a mobile device.

[0074] Figure 2 This embodiment provides a process schematic diagram of a method for locating human joint points based on binocular vision, such as Figure 2 The following is the detailed implementation process:

[0075] 1. Binocular camera calibration

[0076] The purpose of stereo camera calibration is to obtain the internal and external parameters of the stereo camera, including the internal parameter matrix, distortion coefficient, rotation matrix and translation vector. The steps of stereo camera calibration include: shooting with a checkerboard pattern to obtain multiple sets of images; calibrating with the camera calibration function in OpenCV and other libraries to calculate the internal and external parameters of the camera.

[0077] 2. Image preprocessing

[0078] Image preprocessing is used to improve the quality of images and the effect of feature extraction. The specific implementation steps include: grayscale: converting color images into grayscale images; denoising: using Gaussian filtering or other filters to denoise images; histogram equalization: enhancing image contrast and highlighting features.

[0079] 3. Posture estimation algorithm extracts joint points

[0080] The pose estimation algorithm is used to extract pose estimation joint points in the binocular image, and these joint points will be used for matching. Specifically, a pose estimation algorithm such as MediaPipe or ViTPose can be used to extract a first set of joint points in the first human image and a second set of joint points in the second human image.

[0081] 4. Polar geometry constraints

[0082] By using the geometric relationship of the binocular camera, the matching search range is reduced through epipolar constraints to improve matching efficiency and accuracy. For each joint point in the first human image, its corresponding matching point in the second human image should be located on the corresponding epipolar line, narrowing the search range.

[0083] 5. Joint point matching

[0084] Calculate the distance between the joint points identified by the posture estimation algorithm in the first human image and the epipolar line obtained from the second human image in the binocular camera. Confirm the matching point with the point where the epipolar line is located by finding the point closest to the epipolar line.

[0085] 6. Determine the positioning results of human joints

[0086] The projection matrix and matching pair of the binocular camera are used to determine the triangulation equation, and the triangulation equation is solved to obtain the three-dimensional coordinates of the matching pair, thereby determining the positioning results of the human body joints.

[0087] The three-dimensional coordinates of the matching pairs can constitute three-dimensional (3D) point cloud data. Through the 3D point cloud data, the posture and joint position of the human body can be estimated; the motion trajectory and posture changes of the human body can be analyzed based on the matching results in continuous image frames; and based on the motion trajectory and posture changes, human motion function evaluation can be performed, such as gait analysis, motion range evaluation, etc.

[0088] 7. Visual feedback

[0089] The evaluation results are presented to the user in an intuitive way. Based on the 3D point cloud data, the human body model is reconstructed for visual display. For real-time application scenarios, a detailed motion function evaluation report is generated, including indicators such as joint motion range and motion trajectory, providing instant motion status feedback and suggestions.

[0090] The binocular image human matching method based on epipolar geometric constraints achieves high-precision binocular image matching and human motion function evaluation through a series of image processing, key point (joint point) extraction, epipolar constraints, feature matching and triangulation steps. This method not only improves the efficiency and accuracy of matching, but also provides reliable data support and analysis tools for the intelligent human motion function evaluation system.

[0091] Figure 3 A schematic diagram of a joint point matching process provided in this embodiment, such as Figure 2 As shown, the joint point A identified by the posture estimation algorithm in the first human image is used to calculate the epipolar line corresponding to the joint point A in the second human image, and the distance between the joint point and the epipolar line in the second human image is calculated. The matching point of the joint point A is confirmed by finding the point closest to the epipolar line.

[0092] The beneficial effects of this technical solution include:

[0093] 1. Improve matching accuracy and robustness

[0094] Through epipolar geometry constraints and robust feature extraction algorithms, the matching accuracy and robustness are significantly improved, especially in the case of illumination changes, noise interference and partial occlusion.

[0095] 2. Reduce computational complexity

[0096] The polar line constraint significantly reduces the search space for feature matching, greatly reduces the computational complexity, improves the matching efficiency, and is suitable for real-time application scenarios.

[0097] 3. Provide high-precision 3D information

[0098] By using triangulation technology, high-precision 3D position information can be recovered from binocular images, accurate 3D reconstruction can be achieved, and more complex spatial analysis and applications can be supported.

[0099] 4. Enhance applicability in complex environments

[0100] Robust feature extraction and matching algorithms are used to improve matching stability and reliability in complex environments (such as noise, lighting changes, occlusion, etc.), making it suitable for various complex application scenarios.

[0101] 5.Support multiple application scenarios

[0102] This method can be widely used in sports rehabilitation, sports training, fitness guidance, labor protection, elderly activity monitoring, clinical research, special education, and virtual reality (VR) and / or augmented reality (AR) experience, etc., to provide high-quality solutions.

[0103] The human joint positioning method based on binocular vision provided in this embodiment effectively solves many shortcomings in the prior art, significantly improves matching accuracy, robustness and efficiency, and can provide reliable and efficient solutions in multiple application scenarios.

[0104] Embodiment 2

[0105] Figure 4 is a structural schematic diagram of a human joint positioning device based on binocular vision provided in the second embodiment of the present disclosure; Figure 4 As shown, the device includes: a joint point set extraction module 210, a matching module 220, a solution module 230, and a result determination module 240.

[0106] The joint point set extraction module 210 is used to extract a first joint point set in a first human image and a second joint point set in a second human image by using a posture estimation algorithm; the first human image and the second human image are acquired by a binocular camera;

[0107] A matching module 220, configured to match a first joint point in the first joint point set with a second joint point in the second joint point set to obtain a plurality of matching pairs;

[0108] A solution module 230 is used to construct a triangulation equation for each matching pair based on the matching pair and the projection matrix of the binocular camera, and solve the triangulation equation to obtain the three-dimensional coordinates of the matching pair; the projection matrix is ​​constructed by the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera;

[0109] The result determination module 240 is used to determine the human body joint point positioning result according to the three-dimensional coordinates of each matching pair.

[0110] Embodiment 2 of the present disclosure provides a human joint positioning device based on binocular vision, which improves the efficiency and accuracy of joint positioning.

[0111] Furthermore, the matching module 220 further includes:

[0112] an epipolar line calculation unit, configured to calculate, for each of the first joint points in the first joint point set, an epipolar line of the first joint point in the second human body image using a basic matrix;

[0113] A matching pair determination unit is used to determine a second joint point in the second joint point set corresponding to the first joint point based on the polar line, and the first joint point and the second joint point are regarded as a matching pair; wherein the basic matrix is ​​used to describe the geometric relationship between corresponding points in the first human body image and the second human body image.

[0114] Furthermore, the matching pair determination unit is further configured to:

[0115] respectively calculating the distances from the second joint points in the second joint point set to the polar lines;

[0116] The second joint point that is closest to the polar line is used as the second joint point corresponding to the first joint point.

[0117] Furthermore, the device also includes:

[0118] A calibration module is used to calibrate the binocular camera based on the camera calibration function of the open source computer vision library to obtain the intrinsic parameter matrix and extrinsic parameter matrix corresponding to the binocular camera;

[0119] The projection matrix determination module is used to obtain the corresponding projection matrix using the intrinsic parameter matrix and the extrinsic parameter matrix.

[0120] Furthermore, the device also includes:

[0121] An image acquisition module, used for acquiring a human body image using a binocular camera, wherein the human body image includes a first initial human body image and a second initial human body image;

[0122] A preprocessing module, configured to preprocess the first initial human body image to obtain the first human body image, and preprocess the second initial human body image to obtain the second human body image;

[0123] The preprocessing includes: grayscale, denoising and histogram equalization.

[0124] Furthermore, the device also includes:

[0125] The display module is used to display the human body joint point positioning results using a visual display device.

[0126] The human joint positioning device based on binocular vision provided in the embodiments of the present disclosure can execute the human joint positioning method based on binocular vision provided in any embodiment of the embodiments of the present disclosure, and has the corresponding functional modules and beneficial effects of the execution method.

[0127] Embodiment 3

[0128] Figure 5 A schematic diagram of the structure of an electronic device 10 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the embodiments of the present disclosure described and / or claimed herein.

[0129] like Figure 5 As shown, the electronic device 10 includes at least one processor 11, and a memory connected to the at least one processor 11, such as a read-only memory (ROM) 12, a random access memory (RAM) 13, etc., wherein the memory stores a computer program that can be executed by at least one processor, and the processor 11 can perform various appropriate actions and processes according to the computer program stored in the read-only memory (ROM) 12 or the computer program loaded from the storage unit 18 to the random access memory (RAM) 13. In the RAM 13, various programs and data required for the operation of the electronic device 10 can also be stored. The processor 11, the ROM 12, and the RAM 13 are connected to each other through a bus 14. An input / output (I / O) interface 15 is also connected to the bus 14.

[0130] A number of components in the electronic device 10 are connected to the I / O interface 15, including: an input unit 16, such as a keyboard, a mouse, etc.; an output unit 17, such as various types of displays, speakers, etc.; a storage unit 18, such as a disk, an optical disk, etc.; and a communication unit 19, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 19 allows the electronic device 10 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0131] The processor 11 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the processor 11 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various processors running machine learning model algorithms, a digital signal processor (DSP), and any appropriate processor, controller, microprocessor, etc. The processor 11 executes the various methods and processes described above, such as a human joint point positioning method based on binocular vision.

[0132] In some embodiments, the human joint point positioning method based on binocular vision can be implemented as a computer program, which is tangibly contained in a computer-readable storage medium, such as a storage unit 18. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 10 via the ROM 12 and / or the communication unit 19. When the computer program is loaded into the RAM 13 and executed by the processor 11, one or more steps of the human joint point positioning method based on binocular vision described above can be performed. Alternatively, in other embodiments, the processor 11 can be configured to perform the human joint point positioning method based on binocular vision by any other appropriate means (e.g., by means of firmware).

[0133] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0134] The computer programs for implementing the methods of the embodiments of the present disclosure may be written in any combination of one or more programming languages. These computer programs may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that when the computer program is executed by the processor, the functions / operations specified in the flow chart and / or block diagram are implemented. The computer program may be executed entirely on the machine, partially on the machine, partially on the machine as a stand-alone software package and partially on a remote machine, or entirely on a remote machine or server.

[0135] In the context of the disclosed embodiments, a computer-readable storage medium may be a tangible medium that may contain or store a computer program for use by or in conjunction with an instruction execution system, device, or equipment. A computer-readable storage medium may include, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or equipment, or any suitable combination of the foregoing. Alternatively, a computer-readable storage medium may be a machine-readable signal medium. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0136] To provide interaction with a user, the systems and techniques described herein may be implemented on an electronic device having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and a pointing device (e.g., a mouse or trackball) through which the user can provide input to the electronic device. Other types of devices may also be used to provide interaction with the user; for example, the feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form (including acoustic input, voice input, or tactile input).

[0137] The systems and techniques described herein may be implemented in a computing system that includes backend components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes frontend components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such backend components, middleware components, or frontend components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), a blockchain network, and the Internet.

[0138] A computing system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The client and server relationship is generated by computer programs running on the corresponding computers and having a client-server relationship with each other. The server may be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services.

[0139] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in the embodiments of the present disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions of the embodiments of the present disclosure can be achieved, and this document does not limit this.

[0140] The above specific implementations do not constitute a limitation on the protection scope of the embodiments of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the embodiments of the present disclosure shall be included in the protection scope of the embodiments of the present disclosure.

[0141] The embodiments of the present disclosure also provide a computer program product, including a computer program and / or instructions, which, when executed by a processor, implements the method for locating human joints based on binocular vision as provided in any embodiment of the present application.

[0142] In the process of implementation, the computer program product can be written in one or more programming languages ​​or a combination thereof to write computer program codes for performing the operation of the disclosed embodiments, the programming languages ​​including object-oriented programming languages, such as Java, Smalltalk, C++, and also including conventional procedural programming languages, such as "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., using an Internet service provider to connect through the Internet).

[0143] Note that the above are only preferred embodiments of the embodiments of the present disclosure and the technical principles used. Those skilled in the art will understand that the embodiments of the present disclosure are not limited to the specific embodiments herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the protection scope of the embodiments of the present disclosure. Therefore, although the embodiments of the present disclosure are described in more detail through the above embodiments, the embodiments of the present disclosure are not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the embodiments of the present disclosure, and the scope of the embodiments of the present disclosure is determined by the scope of the attached claims.

Claims

1. A method for locating human joints based on binocular vision, characterized in that: include: Extracting a first set of joint points in the first human body image and a second set of joint points in the second human body image using a posture estimation algorithm; The first human body image and the second human body image are acquired by a binocular camera; Matching a first joint point in the first joint point set with a second joint point in the second joint point set to obtain a plurality of matching pairs; For each of the matching pairs, a triangulation equation is constructed based on the projection matrix of the matching pair and the binocular camera, and the three-dimensional coordinates of the matching pair are obtained by solving the triangulation equation; the projection matrix is ​​constructed by the intrinsic parameter matrix and the extrinsic parameter matrix of the binocular camera; The human body joint point positioning result is determined according to the three-dimensional coordinates of each matching pair.

2. The method according to claim 1, characterized in that Matching a first joint point in the first joint point set with a second joint point in the second joint point set to obtain a plurality of matching pairs, including: For each of the first joint points in the first joint point set, using a basic matrix to calculate an epipolar line of the first joint point in the second human body image; A second joint point in the second joint point set corresponding to the first joint point is determined according to the polar line, and the first joint point and the second joint point are regarded as a matching pair; wherein the basic matrix is ​​used to describe the geometric relationship between corresponding points in the first human body image and the second human body image.

3. The method according to claim 2, characterized in that Determining a second joint point in the second joint point set corresponding to the first joint point according to the polar line includes: respectively calculating the distances from the second joint points in the second joint point set to the polar lines; The second joint point that is closest to the polar line is used as the second joint point corresponding to the first joint point.

4. The method according to claim 1, characterized in that: The method further comprises: The binocular camera is calibrated based on the camera calibration program of the open source computer vision library to obtain the intrinsic parameter matrix and extrinsic parameter matrix corresponding to the binocular camera; The corresponding projection matrix is ​​obtained using the intrinsic parameter matrix and the extrinsic parameter matrix.

5. The method according to claim 1, characterized in that Before extracting the first joint point set in the first human image and the second joint point set in the second human image by using a posture estimation algorithm, the method further includes: Using a binocular camera to collect a human body image, wherein the human body image includes a first initial human body image and a second initial human body image; Preprocessing the first initial human body image to obtain the first human body image, and preprocessing the second initial human body image to obtain the second human body image; The preprocessing includes: grayscale, denoising and histogram equalization.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: The human body joint point positioning results are displayed using a visual display device.

7. A human joint positioning device based on binocular vision, characterized in that: include: A joint point set extraction module, used to extract a first joint point set in the first human body image and a second joint point set in the second human body image by using a posture estimation algorithm; The first human body image and the second human body image are acquired by a binocular camera; a matching module, configured to match a first joint point in the first joint point set with a second joint point in the second joint point set to obtain a plurality of matching pairs; A solution module, for each of the matching pairs, constructing a triangulation equation based on the matching pair and a projection matrix of a binocular camera, and solving the triangulation equation to obtain the three-dimensional coordinates of the matching pair; the projection matrix is ​​constructed by an intrinsic parameter matrix and an extrinsic parameter matrix of the binocular camera; The result determination module is used to determine the human body joint point positioning result according to the three-dimensional coordinates of each matching pair.

8. An electronic device, characterized in that: include: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the human joint positioning method based on binocular vision as described in any one of claims 1-6.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, a method for locating human joints based on binocular vision as described in any one of claims 1 to 6 is implemented.

10. A computer program product, characterized in that The computer program product comprises a computer program, and when the computer program is executed by a processor, the computer program implements the method for locating human joints based on binocular vision as described in any one of claims 1 to 6.

Images