A real-time three-dimensional key point estimation method based on binocular vision

Abstract

The application provides a real-time three-dimensional key point estimation method and system based on binocular vision, left and right eye images are synchronously collected by a calibrated binocular camera, and de-distortion and polar line correction are performed; a preset coordinate offset mapping of one eye image relative to another eye image is established, and key points are transformed to an offset coordinate system along a parallax direction; left and right eye images are spliced along a channel dimension to form a binocular fusion image, which is input into a key point estimation model, multi-scale features are extracted, and key point categories and square encoding boxes are output; during training, key points with the same number that can be seen in left and right eyes are respectively used as midpoints of opposite sides of the encoding box after coordinate transformation, and the distance between the two midpoints is used as the side length; when any eye cannot see, the corresponding encoding box is not generated. According to the center of the encoding box, the side length and the offset mapping, the two-dimensional coordinates of the left and right eye key points are recovered and the parallax is determined, and then the three-dimensional coordinates are calculated in combination with the calibration parameters, so that the synchronous estimation of the two-dimensional position, the parallax and the three-dimensional coordinates of the key points and the real-time detection of the front and side are realized.

Description

Technical Field

[0001] This invention relates to the fields of computer vision and binocular stereo vision, and in particular to a real-time 3D keypoint estimation method and system based on binocular vision. Background Technology

[0002] 3D keypoint estimation is widely used in fields such as human pose analysis, human-computer interaction, robot navigation, and industrial inspection. Binocular vision acquires images of the same scene simultaneously using left and right cameras and calculates 3D coordinates using the disparity between corresponding keypoints. It has advantages such as low power consumption, simple hardware structure, and easy edge deployment.

[0003] Existing binocular 3D keypoint estimation typically involves first detecting 2D keypoints in the left and right eye images separately, followed by keypoint matching and triangulation. This multi-stage processing approach is prone to the propagation of detection errors, matching errors, and depth calculation errors at each stage, while also increasing inference latency. Some methods improve matching accuracy by constructing a cost volume or predicting the entire dense disparity map, but these methods have high computational and memory requirements. In contrast, keypoint estimation only needs to obtain the disparity of a small number of keypoints, resulting in significant redundant computations and making it difficult to meet the real-time operational requirements of edge devices.

[0004] Therefore, how to utilize the epipolar geometry of binocular images to directly obtain the two-dimensional position and disparity of key points with the same number during unified processing, thereby reducing the errors and computational burden caused by independent matching and dense disparity calculation, is a problem that existing technologies need to solve. Summary of the Invention

[0005] To address the aforementioned shortcomings of the existing technology, the purpose of this invention is to provide a real-time 3D keypoint estimation method and system based on binocular vision, aiming to solve the technical problems of stepwise error accumulation, complex matching process, and high computational burden on the end side in binocular 3D keypoint estimation.

[0006] To achieve the above objectives, this invention provides a real-time 3D keypoint estimation method based on binocular vision, comprising:

[0007] The left and right eye images are acquired simultaneously using a calibrated binocular camera, and distortion correction and epipolar correction are performed.

[0008] Establish a preset coordinate offset mapping between one eye-corrected image and another eye-corrected image, which is used to transform the coordinates of key points in the eye image to the offset coordinate system along the parallax direction;

[0009] The left and right eye images are stitched together along the channel dimension to obtain a binocular fused image; the binocular fused image is input into a trained keypoint estimation model to extract multi-scale image features and output the keypoint category and square coding box corresponding to each visible keypoint;

[0010] The key point category represents the key point number. The key point estimation model uses the following training annotation: For key points with the same number that are visible in both the left and right eye images, one of the key points is transformed according to the preset coordinate offset mapping. The two key points with the same number after transformation are respectively used as the midpoints of a pair of opposite sides of the square coding box, and the distance between the two midpoints is used as the side length of the square coding box.

[0011] For key points that are not visible in any image, no corresponding square bounding box is generated;

[0012] Based on the center position and side length of the square encoding frame and the preset coordinate offset mapping, the two-dimensional coordinates of the same numbered key points in the left and right eye images are recovered and the parallax is determined.

[0013] The three-dimensional coordinates of each key point are calculated based on the two-dimensional coordinates, the parallax, and the calibration parameters of the binocular camera.

[0014] As a further improvement to the above technical solution, the establishment of the preset coordinate offset mapping includes:

[0015] Traverse the binocular image pairs in the training set and the key points with the same number that are visible in both the left and right images of each binocular image pair. Determine the preset coordinate offset based on the coordinate difference between two key points with the same number along the disparity direction, so that after one key point is transformed, the two key points with the same number are arranged in a preset position order, and the distance between them along the disparity direction is not less than the preset minimum encoding size.

[0016] As a further improvement to the above technical solution, restoring the two-dimensional coordinates and determining the disparity includes:

[0017] Based on the center position and side length of the square encoding frame, determine the midpoints of a pair of opposite sides of the square encoding frame;

[0018] Take one of the midpoints as the coordinates of the key point without coordinate transformation, and perform a coordinate transformation on the other midpoint that is opposite to the preset coordinate offset mapping to obtain the coordinates of the other key point;

[0019] The coordinate difference between two key points along the parallax direction is determined as the parallax of the corresponding key point.

[0020] As a further improvement to the above technical solution, the key point estimation model includes a lightweight backbone network, a feature pyramid network, and a key point prediction module connected in sequence.

[0021] The lightweight backbone network is used to extract image features of different resolutions from the binocular fused image;

[0022] The feature pyramid network is used to upsample, stitch together, and fused convolutional features of images at different resolutions to generate fused features at multiple scales.

[0023] The keypoint prediction module is used to predict the keypoint category and square coding box of each keypoint based on the fusion features.

[0024] As a further improvement to the above technical solution, the lightweight backbone network includes multiple convolutional layers, C3K2 feature extraction layers, spatial pyramid pooling layers and C2PSA feature extraction layers arranged sequentially.

[0025] The feature pyramid network outputs three sets of fusion features with resolutions of 1 / 8, 1 / 16, and 1 / 32 of the resolution of the binocular fused image, respectively.

[0026] As a further improvement to the above technical solution, the key point prediction module includes a one-to-many prediction branch and a one-to-one prediction branch during the training phase.

[0027] The one-to-many prediction branch and the one-to-one prediction branch share the lightweight backbone network and the feature pyramid network, and respectively set the key point category prediction structure and the square coding box regression structure;

[0028] The one-to-many prediction branch assigns multiple positive samples to each square bounding box label, and the one-to-one prediction branch assigns one prediction result to each square bounding box label.

[0029] As a further improvement to the above technical solution, during the training process of the key point estimation model, the classification loss and bounding box regression loss of the one-to-many prediction branch and the one-to-one prediction branch are calculated respectively, and the losses are weighted and summed.

[0030] The classification loss uses binary cross-entropy loss, the bounding box regression loss uses full intersection-union loss, and the loss weight corresponding to the one-to-many prediction branch gradually decreases from the first weight in the early stage of training to the second weight in the later stage of training.

[0031] As a further improvement to the above technical solution, the first weight is 0.8 and the second weight is 0.1; in the inference stage of the key point estimation model, the one-to-many prediction branch is disabled, and the key point category and the square coding box are output by the one-to-one prediction branch, and the non-maximum suppression processing is omitted.

[0032] As a further improvement to the above technical solution, when the epipolar correction ensures that key points with the same number in the left and right eye images are located in the same image row, the three-dimensional coordinates of the k-th key point are determined according to the following formula:

[0033] ;

[0034] ;

[0035] ;

[0036] ;

[0037] in, Let the disparity be the value of the k-th key point. , These are the x-coordinates of the k-th keypoint in the left and right eye images, respectively. Let be the ordinate of the k-th key point in the left eye image. , , Let K be the three-dimensional coordinates of the k-th key point. Baseline distance, , For focal length parameters, , The coordinates of the main point.

[0038] As a further improvement to the above technical solution, the trained keypoint estimation model is exported as an intermediate model format. The intermediate model format is then compiled into an inference engine using a hardware acceleration engine corresponding to the edge device. Integer fixed-point quantization is performed on the keypoint estimation model, and operator fusion is performed on adjacent convolution operations, batch normalization operations, and activation operations.

[0039] As a further improvement to the above technical solution, the binocular camera includes a left-eye camera and a right-eye camera, the baseline distance between the left-eye camera and the right-eye camera is 60-120mm, and the left-eye image and the right-eye image are acquired synchronously through hardware triggering.

[0040] Secondly, the present invention also provides a real-time 3D keypoint estimation system based on binocular vision, comprising:

[0041] The binocular image acquisition module is used to simultaneously acquire left and right eye images through the calibrated binocular camera; the image preprocessing module is used to perform distortion correction and epipolar correction on the two images, establish a preset coordinate offset mapping between the corrected image of one eye and the corrected image of the other eye, and stitch the two corrected images along the channel dimension to obtain a binocular fused image.

[0042] The keypoint estimation module is used to input the binocular fused image into a trained keypoint estimation model, extract multi-scale image features, and output the keypoint category and square coding box corresponding to each visible keypoint. The keypoint category represents the keypoint number. The keypoint estimation model is trained and labeled as follows: for keypoints with the same number that are visible in both the left and right eye images, one of the keypoints is transformed to the offset coordinate system along the disparity direction according to the preset coordinate offset mapping. The two keypoints with the same number after transformation are respectively used as the midpoints of a pair of opposite sides of the square coding box, and the distance between the two midpoints is used as the side length. For keypoints that are not visible in either eye image, no corresponding square coding box is generated.

[0043] The coordinate recovery module is used to recover the two-dimensional coordinates of key points with the same number in the left and right eye images and determine the parallax based on the center position, side length and preset coordinate offset mapping of the square encoding frame.

[0044] The three-dimensional coordinate calculation module is used to calculate the three-dimensional coordinates of each key point based on the two-dimensional coordinates, the parallax, and the calibration parameters of the binocular camera.

[0045] As a further improvement to the above technical solution, the key point estimation module includes a lightweight backbone network, a feature pyramid network, and a key point prediction module; the lightweight backbone network is used to extract image features of different resolutions from the binocular fused image, the feature pyramid network is used to fuse the image features of different resolutions, and the key point prediction module is used to output the key point category and the square coding box based on the fused image features.

[0046] As a further improvement to the above technical solution, the key point prediction module includes a one-to-many prediction branch and a one-to-one prediction branch during the training phase. The one-to-many prediction branch and the one-to-one prediction branch share the lightweight backbone network and the feature pyramid network. During the inference phase, the one-to-many prediction branch is disabled, and the key point category and the square coding box are output through the one-to-one prediction branch.

[0047] As a further improvement to the above technical solution, it also includes an edge deployment module, which is used to compile the key point estimation model into an inference engine adapted to the edge device, perform integer fixed-point quantization on the key point estimation model, and perform operator fusion on adjacent convolution operations, batch normalization operations and activation operations.

[0048] Because the present invention adopts the above technical solutions, the beneficial effects of the present invention are as follows:

[0049] This invention provides a real-time 3D keypoint estimation method based on binocular vision. It leverages the characteristic that keypoints with the same number satisfy corresponding epipolar constraints after epipolar correction in binocular images. Through a preset coordinate offset mapping, keypoints with the same number in both eyes are encoded into square bounding boxes. The fused binocular image is used as input to the keypoint estimation model, enabling the keypoint category, 2D positions of both eyes, and disparity to be obtained in a unified processing flow. This reduces the processing steps required for separate detection and independent matching of keypoints in both eyes, as well as dense disparity calculation, thus reducing error propagation and computational burden, and providing a foundation for real-time 3D keypoint estimation on the edge. Specifically:

[0050] This invention performs distortion correction and epipolar correction on synchronously acquired left and right eye images, ensuring that key points with the same number in both eyes satisfy the corresponding epipolar geometric relationship. Based on this, by using a preset coordinate offset mapping, one of the key points is transformed to the offset coordinate system along the disparity direction, which can establish a definite positional relationship between key points with the same number in both eyes, providing a geometric basis for subsequent unified encoding, position recovery, and disparity calculation.

[0051] This invention uses two keypoints with the same number after coordinate transformation as the midpoints of a pair of opposite sides of a square encoding frame, and uses the distance between the two midpoints as the side length of the square encoding frame. This allows a single encoding frame to simultaneously represent the keypoint number and the positional relationship between keypoints with the same number in the left and right eyes. Therefore, it eliminates the need to detect keypoints separately in the left and right eye images and then perform independent matching, reducing the impact of matching errors on disparity and 3D coordinate calculation results.

[0052] This invention stitches the left and right eye images along the channel dimension to form a binocular fused image, and inputs this binocular fused image into the same keypoint estimation model. This allows the model to directly output the category and square bounding box of visible keypoints within a unified feature space, utilizing information from both the left and right eye images. Compared to processing the left and right eye images separately, this reduces redundant feature extraction and multi-stage serial processing, lowering processing overhead and error accumulation caused by intermediate data transfer.

[0053] This invention uses the center position and side length of a square encoding frame, combined with a preset coordinate offset mapping, to recover the two-dimensional coordinates of key points with the same number for both left and right eyes and determine the disparity, thus establishing a reversible geometric relationship between the encoding result and the positions of the binocular key points. Therefore, after obtaining the key point prediction results, the two-dimensional position and disparity information required for three-dimensional coordinate calculation can be obtained directly, without the need for separate key point matching and disparity search processes.

[0054] This invention predicts key point categories and square coding boxes for only a limited number of visible key points. It does not require constructing a cost volume containing multiple candidate disparities across the entire image, nor does it require generating a dense disparity map and then extracting the disparity at the key point location. This reduces pixel-level computation and intermediate feature storage that are unrelated to sparse key point estimation, thereby reducing the computational load and memory usage of the model.

[0055] This invention employs a lightweight backbone network to extract image features and uses multi-scale feature fusion to take into account both local location features and high-level semantic features of key points. During the training phase, it utilizes both one-to-many and one-to-one prediction branches for optimization, while during the inference phase, it only uses one-to-one prediction branches to output results. This reduces the inference overhead caused by additional prediction branches and non-maximum suppression, which is beneficial to improving the processing efficiency of edge devices.

[0056] For keypoints that are not visible in either eye image, this invention does not generate corresponding square bounding box annotations. This avoids keypoints lacking effective binocular correspondence information from participating in bounding box regression, reducing interference from invalid annotations on model training. Based on the 2D coordinates and disparity recovered from the visible keypoints, combined with the calibration parameters of the binocular camera, the 3D coordinates of the corresponding keypoints can be further calculated, achieving continuous output of the 2D position, disparity, and 3D position of the binocular keypoints. Attached Figure Description

[0057] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the structures shown in these drawings without creative effort.

[0058] Figure 1 This is a flowchart illustrating a real-time 3D keypoint estimation method based on binocular vision disclosed in this invention.

[0059] Figure 2 This is a schematic diagram of the structure of the C3K2 feature extraction layer disclosed in an embodiment of the present invention;

[0060] Figure 3 This is a schematic diagram of the structure of the C2PSA feature extraction layer disclosed in an embodiment of the present invention;

[0061] Figure 4 This is a schematic diagram of the lightweight backbone network and feature pyramid network disclosed in the embodiments of the present invention;

[0062] Figure 5 This is a schematic diagram of the structure of one-to-many prediction branches and one-to-one prediction branches disclosed in the embodiments of the present invention;

[0063] Figure 6 This is a schematic diagram of establishing a preset coordinate offset mapping as disclosed in an embodiment of the present invention;

[0064] Figure 7 This is a schematic diagram of constructing a square coding frame based on key points with the same number on the left and right eyes, as disclosed in an embodiment of the present invention.

[0065] Figure 8 This is a schematic diagram of the training process of the key point estimation model disclosed in an embodiment of the present invention;

[0066] Figure 9 This is a schematic diagram of the reasoning process for three-dimensional keypoint estimation disclosed in an embodiment of the present invention;

[0067] Figure 10 This is a schematic diagram of extracting disparity features using a strip convolution kernel along the disparity direction, as disclosed in an embodiment of the present invention.

[0068] The realization of the objective, functional characteristics and advantages of the present invention will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0069] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of the present invention, and not all of them. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0070] It should be noted that the technical solutions of the various embodiments of the present invention can be combined with each other, but only if they are based on the ability of those skilled in the art to implement them. When the combination of technical solutions is contradictory or cannot be implemented, it should be considered that such combination of technical solutions does not exist and is not within the scope of protection claimed by the present invention.

[0071] Example 1

[0072] See Figures 1-9 This invention provides a real-time 3D keypoint estimation method based on binocular vision, comprising the following steps: binocular image acquisition and correction, keypoint coordinate encoding, model training, keypoint prediction, coordinate decoding, and 3D coordinate calculation. The specific implementation process is as follows:

[0073] First, left and right eye images are simultaneously acquired using a calibrated binocular camera. The calibration parameters for the binocular camera include the intrinsic parameters, distortion parameters, relative rotation matrix, relative translation vector, and baseline distance of the left and right cameras. Based on these calibration parameters, distortion correction and epipolar correction are performed on the two images, ensuring that keypoints with the same number in the left and right eye images lie on their corresponding epipolar lines. The epipolar extension direction is defined as the disparity direction, and the direction perpendicular to the disparity direction is defined as the epipolar orthogonal direction. In the case of horizontal epipolar correction, the disparity direction is horizontal, and the epipolar orthogonal direction is vertical.

[0074] Before deploying the keypoint estimation model, binocular training data is pre-constructed. Multiple keypoints are predefined based on the target to be estimated, and the epipolar-corrected left and right eye training images are labeled according to the keypoint numbers. For the k-th keypoint visible in both the left and right eye images, its coordinates along the disparity direction are recorded. , and polar orthogonal direction coordinates After polar line correction, the polar line orthogonal coordinates of key points with the same number on the left and right sides are the same or within the allowable correction error range.

[0075] To enable binocular keypoints with the same number to be converted into geometrically sized annotations suitable for detection, a preset coordinate offset mapping is established between the coordinates of one keypoint and the other. In this embodiment, the right keypoint is used as the offset object, and its coordinates are transformed along the parallax direction as follows:

[0076] ;

[0077] Where D is the preset coordinate offset. The coordinates of the right eye key point in the offset coordinate system.

[0078] For the same numbered key points in the binoculars after offset, use point ( , ) and point ( , These two points are respectively the midpoints of a pair of opposite sides of a square encoding frame. The center coordinates of this square encoding frame are:

[0079] ;

[0080] Its side length is: ;

[0081] In = - When the disparity of the k-th keypoint is represented, the side length satisfies:

[0082] ;

[0083] Therefore, a square bounding box can simultaneously contain the positional relationship and disparity information of keypoints with the same number in both the left and right eyes. By using the keypoint number as the category of the square bounding box, the model can determine the corresponding keypoint number based on the predicted category, eliminating the need to perform independent matching after the keypoints in both eyes have been detected.

[0084] For keypoints that are not visible in either eye image, a square bounding box corresponding to the keypoint is not constructed, nor is the monocular keypoint position used to replace the binocular correspondence annotation. This process can avoid keypoints lacking effective binocular correspondence from participating in bounding box regression and reduce the interference of invalid annotations on model training, but it is not used to infer or recover occluded keypoints.

[0085] After training and annotation are completed, the left and right eye images are stitched together according to the channel dimension. For a three-channel left eye image and a three-channel right eye image, the stitched image forms a six-channel binocular fusion image. The binocular fusion image is input into the same keypoint estimation model, allowing the model to utilize information from both the left and right eye images simultaneously during a single feature extraction process, avoiding redundant calculations caused by using two separate models to extract features from the left and right eyes.

[0086] The keypoint prediction module outputs the keypoint category and the center position and side length of the square bounding box based on multi-scale fused features. During training, both a one-to-many prediction branch and a one-to-one prediction branch can be configured. These two branches share a lightweight backbone network and a feature pyramid network, and calculate classification loss and bounding box regression loss respectively; the classification loss can use binary cross-entropy loss, and the bounding box regression loss can use full intersection-union loss. Initially, the loss weight of the one-to-many prediction branch is increased to provide sufficient supervision signals; as training progresses, the loss weight of this branch is gradually decreased. During the inference phase, the one-to-many prediction branch is disabled, and only the one-to-one prediction branch is used to output the prediction results, thereby reducing the inference overhead caused by additional prediction branches and non-maximum suppression processing.

[0087] During model inference, the simultaneously acquired left and right eye images undergo the same distortion correction, epipolar correction, normalization, and channel stitching processes as in the training phase. The resulting binocular fused image is then input into the trained keypoint estimation model. The model outputs the keypoint category and bounding box center coordinates for each visible keypoint. , and side length .

[0088] Based on the predicted square encoding frame, the coordinates of the midpoints of the two opposite sides are recovered. Under the positional order set in this embodiment, the coordinates of the left and right visual keypoints along the parallax direction are determined according to the following formulas:

[0089] ;

[0090] ;

[0091] The polar orthogonal coordinates of the key points in both eyes are taken as follows: The corresponding parallax is:

[0092] ;

[0093] Therefore, the two-dimensional coordinates and disparities of keypoints with the same number for both eyes can be directly recovered from a single square bounding box. This eliminates the need to construct a cost volume across multiple candidate disparity ranges in the entire image, and also eliminates the need to first generate a dense disparity map before extracting the disparity at the keypoint location. Since the model outputs categories and bounding boxes for only a limited number of visible keypoints, it reduces pixel-level computations and intermediate feature storage that are irrelevant to sparse keypoint estimation.

[0094] With horizontal epipolar correction, let q=u and r=v, then the three-dimensional coordinates of the kth keypoint can be calculated using the following formula:

[0095] ;

[0096] ;

[0097] ;

[0098] in, , , Let K be the 3D coordinates of the k-th keypoint in the stereo camera coordinate system. , Let B be the two-dimensional coordinates of the key point in the left eye image, and let B be the baseline distance of the binocular cameras. , These are the camera's focal length parameters in both the horizontal and vertical directions. , These are the coordinates of the camera's principal point.

[0099] Through the above processing, keypoints with the same number in both eyes are converted into single detection objects with definite geometric meanings. The keypoint category, 2D position of both eyes, and disparity can be obtained from the prediction results of the same model. This approach reduces the intermediate steps required for separate detection of keypoints in both eyes, result matching, and dense disparity estimation, and also reduces the corresponding repetitive feature extraction, pixel-level disparity calculation, and intermediate data storage, providing a foundation for real-time 3D keypoint estimation on edge devices.

[0100] In a preferred embodiment, a preset coordinate offset mapping is established during the training sample construction phase. Specifically, all binocular image pairs in the training set are traversed, and keypoints with the same number that are visible in both the left and right images of each binocular image pair are read. The coordinate direction of the disparity generated after epipolar correction is defined as the disparity direction; depending on the installation direction of the binocular camera and the epipolar correction result, the disparity direction can be horizontal or vertical.

[0101] For the Each key point with the same number is respectively... and This represents its coordinates along the parallax direction in the left and right eye images. In this embodiment, the right eye keypoint is used as the offset object, and it is transformed to the offset coordinate system along a preset direction:

[0102] ;

[0103] in, These are the coordinates of the right eye keypoint in the offset coordinate system. This is a preset coordinate offset. The preset coordinate offset is a fixed value shared by all binocular image pairs in the training set, and is configured to ensure that all valid keypoints with the same number satisfy:

[0104] ;

[0105] in, A preset minimum encoding size is used. As a result, the offset right eye key point is located on the same preset side of the left eye key point along the parallax direction, the two key points maintain a fixed positional order, and there is a distance between them that is not less than the preset minimum encoding size.

[0106] In one possible implementation, the preset coordinate offset can be selected according to the following formula:

[0107] ;

[0108] in, This indicates that the training focuses on a set of keypoints with the same number that are visible in both the left and right eye images. When the calculation result is less than zero, the preset coordinate offset can be zero or a preset non-negative offset value. The preset minimum encoding size can be determined based on the input image resolution, the feature map downsampling factor, and the keypoint detection scale; in a specific example, the preset minimum encoding size is 16 pixels.

[0109] After the coordinate offset is completed, the left-eye keypoint and the offset right-eye keypoint are respectively used as the midpoints of a pair of opposite sides of the square encoding frame, and the distance between them along the parallax direction is used as the side length of the square encoding frame. Since all encoding frames are constructed according to the same keypoint order, after the model predicts the square encoding frame, the left-eye keypoint and the right-eye keypoint can be directly distinguished based on the position of the midpoint of the opposite side, without having to determine the correspondence between the two midpoints again. At the same time, by limiting the minimum encoding size, the missed detection caused by the encoding frame being too small and the confusion of the keypoint order during decoding can be reduced.

[0110] In this embodiment, the coordinate offset mapping is used for training annotation encoding and decoding prediction results without changing the pixel arrangement of the left and right eye images. The same offset direction and preset coordinate offset are used in both the training and inference phases. After determining the coordinates of the right eye keypoint in the offset coordinate system during the inference phase, its original coordinates are restored by performing the opposite operation to the coordinate transformation described above. The fixed offset mapping maintains a definite and reversible geometric relationship between the encoding box and the coordinates of the left and right eye keypoints, thus providing a unified coordinate reference for subsequent recovery of 2D coordinates and disparity based on the encoding box.

[0111] In a preferred embodiment, the keypoint estimation model includes a lightweight backbone network, a feature pyramid network, and a keypoint prediction module connected in sequence. After the aforementioned image preprocessing, the three-channel left-eye image and the three-channel right-eye image are stitched together along the channel dimension to form a binocular fused image of size H×W×6, which is then input into the lightweight backbone network. Thus, the left and right-eye images share a single feature extraction process, eliminating the need for separate backbone networks and reducing redundant convolution calculations and intermediate feature storage.

[0112] In one specific embodiment, the lightweight backbone network comprises, in sequence:

[0113] Layer 0 is a convolutional layer that converts the number of input channels from 6 to 16 with a stride of 2.

[0114] The first layer is a convolutional layer, which converts the number of channels from 16 to 32 with a stride of 2.

[0115] The second layer is the C3K2 feature extraction layer, which converts the number of channels from 32 to 64 and repeats this setting twice.

[0116] The third layer is a convolutional layer with 64 input channels and 64 output channels, and a stride of 1.

[0117] The fourth layer is the C3K2 feature extraction layer, which converts the number of channels from 64 to 128 and repeats this setting twice.

[0118] The fifth layer is a convolutional layer with 128 input and 128 output channels and a stride of 2.

[0119] The 6th layer is the C3K2 feature extraction layer, with 128 input channels and 128 output channels, and this setting is repeated twice.

[0120] The 7th layer is a convolutional layer, which converts the number of channels from 128 to 256 with a stride of 2.

[0121] The 8th layer is the C3K2 feature extraction layer, with 256 input channels and 256 output channels, and this setting is repeated twice.

[0122] The 9th layer is a spatial pyramid pooling layer with 256 input channels and 256 output channels.

[0123] The 10th layer is the C2PSA feature extraction layer, with 256 input and 256 output channels, and this setting is repeated twice.

[0124] The backbone network employs a multi-layered architecture, consisting of convolutional layers for channel adjustment and feature map downsampling; C3K2 feature extraction layers for extracting local structural features while controlling the number of network parameters; spatial pyramid pooling layers for pooling contextual information from different receptive fields; and C2PSA feature extraction layers for further filtering and fusion of deep features. Through this hierarchical structure, the backbone network can progressively reduce feature map resolution and increase the number of feature channels, thereby extracting higher-level semantic information while preserving the local location information of key points.

[0125] The lightweight backbone network inputs features extracted at different stages into the feature pyramid network. To avoid confusion between different level names, this embodiment refers to the features output by the feature pyramid network as the first-scale fusion feature, the second-scale fusion feature, and the third-scale fusion feature, respectively, according to the output resolution:

[0126] The first-scale fusion feature has 64 channels and a resolution of H / 8×W / 8.

[0127] The second-scale fusion feature has 128 channels and a resolution of H / 16×W / 16.

[0128] The third-scale fusion feature has 256 channels and a resolution of H / 32×W / 32.

[0129] The feature pyramid network upsamples deep features and concatenates them with shallow features of similar or the same resolution in the backbone network. Channel adjustment and feature fusion are then achieved through convolution operations. Simultaneously, downsampling and feature concatenation further transmit information from different levels. Thus, the first-scale fused features retain more spatial details, suitable for representing small or closely spaced keypoints; the second-scale fused features consider both location details and semantic information; and the third-scale fused features contain a wider range of contextual information. These three sets of fused features are used together for keypoint prediction, reducing the insufficient keypoint localization information caused by relying solely on single-resolution features.

[0130] The keypoint prediction module receives three sets of fused features and outputs category prediction results and square bounding box prediction results for each visible keypoint according to predefined keypoint categories. The category prediction results are used to determine the keypoint number, and the square bounding box prediction results include at least the center position and side length of the bounding box. The prediction results are decoded according to the aforementioned two-dimensional coordinate recovery and disparity determination steps to obtain the two-dimensional coordinates and corresponding disparities of keypoints with the same number for both left and right eyes.

[0131] This embodiment employs a lightweight backbone network to uniformly process binocular fused images and uses three sets of multi-scale fusion features for keypoint-level sparse prediction. It eliminates the need to extract features from the left and right eye images separately, and avoids constructing a cost volume covering the entire image and multiple candidate disparities. This reduces repetitive feature extraction, dense disparity calculation, and large-size intermediate feature storage. While maintaining the ability to extract keypoint location features and contextual information, it reduces the model's computational load and memory consumption, providing a foundation for real-time inference on edge devices.

[0132] In a preferred embodiment, during the training phase of the keypoint estimation model, the keypoint prediction module is configured with a one-to-many prediction branch and a one-to-one prediction branch. The two prediction branches share the aforementioned lightweight backbone network and feature pyramid network to jointly utilize the multi-scale fusion features of the binocular fusion image; each prediction branch includes a keypoint category prediction structure and a square bounding box regression structure, used to output the keypoint category and the center position and side length of the corresponding square bounding box.

[0133] After training samples are input into the keypoint estimation model, labels are assigned to two prediction branches. The one-to-many prediction branch assigns multiple positive samples that meet the matching criteria for the same square bounding box label, ensuring that candidate predictions at different scales and locations receive supervision information. The one-to-one prediction branch, on the other hand, assigns a matching prediction result to each square bounding box label from the candidate prediction results and uses it as the corresponding positive sample. Thus, the one-to-many prediction branch provides sufficient supervision during the training phase, while the one-to-one prediction branch gradually develops an output method where each keypoint corresponds to a single prediction result.

[0134] For each of the two prediction branches, the classification loss and bounding box regression loss are calculated separately. The classification loss uses binary cross-entropy loss to constrain the difference between the predicted keypoint category and the labeled keypoint number; the bounding box regression loss uses perfect intersection-over-union (MIU) loss to constrain the difference between the predicted square bounding box and the labeled square bounding box in terms of center position, side length, and overlap. The classification loss and bounding box regression loss of the two prediction branches are weighted and summed, and the network parameters of the lightweight backbone network, feature pyramid network, and the two prediction branches are updated based on the total loss.

[0135] To balance sufficient supervision during training with the learning effectiveness of the one-to-one prediction branch, a progressive loss weight is adopted for the one-to-many prediction branch. In the early stages of training, the loss weight corresponding to the one-to-many prediction branch is set as the first weight to utilize multiple positive samples to promote parameter updates in the shared feature network. As training progresses, this loss weight is gradually reduced according to a preset decay strategy, allowing the model to better adapt to the output method of one-to-one prediction results in later training stages. In one specific embodiment, the first weight is 0.8, and the loss weight of the one-to-many prediction branch in the later stages of training is reduced to a second weight of 0.1. These weights represent the relative role of the one-to-many prediction branch loss in the total loss, and the specific decay process can be divided into continuous decay or phased decay depending on the training rounds.

[0136] After the keypoint estimation model is trained, the one-to-many prediction branch is disabled during the inference phase, and only the one-to-one prediction branch is retained. The aforementioned three sets of multi-scale fused features are input into the one-to-one prediction branch, which directly outputs the keypoint category and square bounding boxes for each visible keypoint. Since each square bounding box corresponds to a matching prediction result during the training phase, there is no need to filter the final result from multiple overlapping candidate boxes through non-maximum suppression during inference.

[0137] By using both one-to-many and one-to-one prediction branches in the training phase, the one-to-many label assignment can supplement the relatively limited supervision information under the one-to-one training method, while the one-to-one prediction branch learns the output relationship of each key point corresponding to a single prediction result. In the inference phase, only the one-to-one prediction branch is retained, which can reduce the computation and post-processing steps required for one-to-many prediction branches and non-maximum suppression, which is beneficial to reducing edge inference latency.

[0138] In a preferred embodiment, the square bounding box regression structure of the keypoint prediction module includes a disparity feature extraction layer disposed before the bounding box regression output layer, wherein the disparity feature extraction layer employs a strip convolutional kernel extending along the disparity direction. See also... Figure 10 The length of the strip convolution kernel along the parallax direction is 5, and the width along the epipolar orthogonal direction is 1.

[0139] After the aforementioned epipolar correction, keypoints with the same number in the left and right eye images are located on the corresponding epipolar lines, meaning their coordinates along the orthogonal direction of the epipolar lines are the same or their difference is within a preset error range. The coordinate difference between the two keypoints along the disparity direction is used to characterize the disparity. Based on this geometric constraint, the disparity feature extraction layer performs convolutional aggregation on the fused features of adjacent positions along the disparity direction to extract local features related to the positional difference of keypoints with the same number in the left and right eyes. The convolution result is then input into the square bounding box regression output layer to predict the center position and side length of the square bounding box.

[0140] In the implementation of horizontal epipolar correction, the disparity direction is horizontal, and the bar convolution kernel is set along the horizontal direction of the image; in the implementation of vertical epipolar correction, the disparity direction is vertical, and the bar convolution kernel is set along the vertical direction of the image. Thus, the convolution kernel direction is always consistent with the actual disparity direction, avoiding the introduction of irrelevant region features due to the inconsistency between the convolution direction and the epipolar correction direction.

[0141] Compared to a square convolution kernel of the same length, the strip convolution kernel expands the receptive field only in the disparity direction, reducing the introduction of background and adjacent keypoint features along the epipolar orthogonal direction and lowering the computational cost of convolution. Placing this disparity feature extraction layer before the bounding box regression output layer allows the regression structure to determine the bounding box side length using local features along the disparity direction. The bounding box side length has the aforementioned geometric relationship with the disparity of the same-numbered keypoints in the left and right eyes, thus providing more targeted feature input for subsequent 2D coordinate recovery and disparity calculation.

[0142] In a preferred embodiment, after the keypoint estimation model has completed training and reached the preset convergence condition, the model is exported and optimized for edge deployment. This embodiment uses a Jetson edge device as an example, exporting the trained keypoint estimation model from the training framework format to the ONNX intermediate model format, and then using the TensorRT hardware acceleration engine to compile the ONNX model into an inference engine adapted to the target device.

[0143] Before exporting the model, the network parameters of the lightweight backbone network, feature pyramid network, and keypoint prediction module are fixed. Following the aforementioned inference method, the one-to-many prediction branch is disabled, and only the one-to-one prediction branch is retained to construct the inference computation graph. The exported intermediate model maintains the input format of the binocular fused image and the output format of the keypoint category and square encoding box, ensuring consistency in the input, output, and subsequent decoding relationships before and after model deployment.

[0144] During model export, the network operations in the training framework are converted into operator representations supported by the intermediate model, and the weights of each layer and the network connection relationships are saved. Subsequently, the intermediate model is input into TensorRT, which performs network parsing and inference engine compilation based on the computing resources and operator support of the edge device. The compiled inference engine is used to receive the binocular fused image obtained through the aforementioned processing on the edge device and output the keypoint categories and square bounding boxes of visible keypoints.

[0145] In one optional implementation, INT8 integer fixed-point quantization is performed on the keypoint estimation model. This integer fixed-point quantization can be performed using post-training quantization or quantization-aware training, converting some floating-point weights and intermediate features in the model into integer fixed-point representations. Whether to use integer fixed-point quantization depends on the computing power, storage conditions, and keypoint prediction accuracy requirements of the edge device. Integer fixed-point quantization reduces the storage space occupied by model parameters and intermediate features, and lowers the data processing volume of corresponding operations.

[0146] Furthermore, adjacent convolution operations, batch normalization operations, and activation operations in the inference computation graph are fused, combining multiple operations that were originally executed sequentially into a single fusion operation. Since the batch normalization parameters remain fixed after model training, they can be merged into adjacent convolution operations, and the corresponding activation processing can be performed after the convolution result is output. This approach does not change the functional relationships between network layers or the meaning of model output, reduces the repeated writing and reading of intermediate results, and lowers the processing overhead caused by scheduling multiple operators separately.

[0147] After completing the above processing, the inference engine is loaded into the edge device. The edge device performs the aforementioned distortion correction, epipolar correction, normalization, and channel stitching processing on the synchronously acquired left and right eye images, inputs the resulting binocular fused image into the inference engine, obtains the key point category and square coding box, and then obtains the two-dimensional coordinates and disparity of the same numbered key points in the left and right eyes according to the aforementioned coordinate recovery method, and calculates the three-dimensional coordinates in combination with the calibration parameters of the binocular camera.

[0148] By exporting intermediate models, compiling with hardware acceleration engines, using optional integer fixed-point quantization and operator fusion, the amount of data processing, intermediate feature storage and operator scheduling in the edge inference process can be reduced without changing the relationship between keypoint encoding, model output and coordinate decoding. This helps to reduce the inference time and storage burden of the keypoint estimation model on the edge device.

[0149] In a preferred embodiment, the binocular camera includes a fixedly mounted left and right eye camera, with the optical center distance between them forming a baseline distance B, which is 60–120 mm. Before use, the left and right eye cameras are binocularly aligned to obtain camera intrinsic parameters, distortion parameters, and relative pose parameters between them, and their relative mounting positions are kept constant during image acquisition.

[0150] The left and right cameras are connected to the same hardware trigger signal source and synchronously acquire left and right eye images in response to the same trigger signal. Based on the same frame number or acquisition time identifier, the corresponding left and right eye images are combined into a binocular image pair. Distortion correction and epipolar correction are then performed according to the binocular calibration parameters, and a binocular fused image is formed in the aforementioned manner.

[0151] By fixing the baseline distance of the binocular cameras and using hardware-triggered synchronous acquisition, the positional deviation of key points caused by the inconsistency in the acquisition time of the left and right eye images can be reduced, providing binocular images with definite temporal and spatial geometric relationships for subsequent determination of key point correspondence, disparity calculation, and 3D coordinate recovery.

[0152] To verify the accuracy of 3D keypoint estimation and edge-side performance of the method described in this invention, comparative experiments were conducted on the THU-Bi-Hand dataset. The THU-Bi-Hand dataset, short for Tsinghua Binocular Hand Pose Dataset, is a publicly available dataset for binocular stereo vision hand pose estimation. This dataset contains 446,620 binocular image pairs from 10 subjects, with 356,522 frames in the training set and 90,098 frames in the test set.

[0153] Each frame in the dataset includes simultaneously acquired left and right eye infrared images, both with a resolution of 640×480 pixels. Each frame is labeled with six hand keypoints: wrist, thumb tip, index finger tip, middle finger tip, ring finger tip, and little finger tip. The annotation information for each keypoint includes its three-dimensional coordinates in the binocular camera coordinate system, as well as its two-dimensional pixel coordinates in the left and right eye images. The three-dimensional coordinates are in millimeters.

[0154] The ground truth values ​​of the 3D keypoints in the dataset were acquired using the TrakSTAR six-DOF magnetic field tracking system. During the acquisition process, six magnetic field sensors were respectively positioned at the wrist and five fingertips on the back of the subject's hand to obtain the 3D positions of the corresponding hand keypoints. The nominal positioning accuracy of the magnetic field tracking system is 1.4 mm.

[0155] This experiment uses the Bi-Pose-REN and YOLO11Pose methods as comparison methods. Bi-Pose-REN is the benchmark method for binocular hand pose estimation used in this dataset; the YOLO11Pose method performs 2D keypoint detection on both the left and right images, calculates disparity based on the lateral coordinate difference of keypoints with the same number in the left and right images, and obtains the 3D keypoint coordinates by combining the binocular camera calibration parameters.

[0156] The method of this invention is denoted as YOLO26-Stereo. This method performs distortion correction, epipolar correction, and channel stitching on the left and right eye images according to the aforementioned embodiments. The resulting binocular fused image is then input into a keypoint estimation model to directly predict keypoint categories and square bounding boxes. Subsequently, the two-dimensional coordinates of keypoints with the same number in both eyes are recovered based on the square bounding boxes, and disparity is determined. Finally, the three-dimensional coordinates of each keypoint are calculated.

[0157] All three methods are evaluated using the Euclidean distance between the predicted 3D coordinates and the corresponding ground truth coordinates of each hand keypoint in the test set, and the average 3D distance of all test samples and all keypoints is calculated. The smaller the average 3D distance, the closer the 3D keypoint estimation result is to the ground truth. The running speed of the YOLO11Pose method and the method of this invention was tested on a Jetson Nano edge device, and the running speed is expressed as the number of binocular image pairs processed per second.

[0158] The experimental results are shown in Table 1 below.

[0159] Table 1 shows the comparative experimental results of the three methods.

[0160]

[0161] The results in the table show that the average 3D distance of the method of the present invention on the test set is 9.05 mm, which is lower than the 9.17 mm of the Bi-Pose-REN method and the 9.29 mm of the YOLO11Pose method. On the Jetson Nano edge device, the running speed of the method of the present invention is 22 frames / second, which is higher than the 13 frames / second of the YOLO11Pose method.

[0162] The above results demonstrate that, compared to the method of separately detecting keypoints in the left and right eye images before performing corresponding matching and disparity calculation, this invention extracts left and right eye features uniformly from binocular fused images and uses square coding boxes to jointly represent keypoints with the same number in both eyes, thus reducing redundant feature extraction in two branches and subsequent keypoint matching steps. Under the experimental conditions, this method improves edge-side running speed while reducing the average 3D distance. It should be noted that the edge-side running speed of the Bi-Pose-REN method was not measured in this experiment, therefore its running speed is not listed in the table.

[0163] Example 2

[0164] This invention also provides a real-time 3D keypoint estimation system based on binocular vision. The system includes a binocular image acquisition module, an image preprocessing module, a keypoint estimation module, a coordinate restoration module, a 3D coordinate calculation module, and an edge deployment module. Each module can be implemented by a processor in the edge device calling corresponding program instructions, and data is transmitted in the order of binocular image acquisition, image preprocessing, keypoint prediction, coordinate restoration, and 3D coordinate calculation.

[0165] The binocular image acquisition module is connected to a calibrated binocular camera, used to control the left and right cameras to simultaneously acquire left and right eye images, and to combine two images with the same frame number or acquisition time identifier into a binocular image pair. The calibration parameters of the binocular camera include the intrinsic parameters, distortion parameters, relative rotation matrix, relative translation vector, and baseline distance of the left and right cameras. Synchronous acquisition reduces the time difference in acquisition of moving targets in the left and right eye images, providing time-corresponding image input for subsequent disparity determination of key points with the same number.

[0166] The image preprocessing module receives a pair of binocular images and performs distortion correction and epipolar correction on the left and right images respectively according to the calibration parameters of the binocular camera, so that keypoints with the same number in the left and right images are located on the corresponding epipolar lines. The image preprocessing module also performs size unification and normalization on the two corrected images and stitches them along the channel dimension to obtain a binocular fused image. Taking two images that are both three-channel images as an example, the stitched binocular fused image has six channels, which allows the subsequent keypoint estimation model to jointly extract features from the left and right images in the same network, reducing the redundant calculations caused by setting up separate feature extraction networks for the left and right eyes.

[0167] The image preprocessing module also stores a preset coordinate offset map, which includes the offset direction and the preset coordinate offset amount. It should be noted that the preset coordinate offset map is used for keypoint coordinate transformation in training annotation construction and prediction result decoding, and does not involve pixel translation of the entire left or right eye image. Therefore, the square encoding box construction method in the training phase and the coordinate recovery method in the inference phase can use the same coordinate reference, avoiding decoding errors caused by inconsistent offset objects or offset directions.

[0168] The keypoint estimation module is used to input the binocular fused image into the trained keypoint estimation model and output the keypoint category and square bounding box corresponding to each visible keypoint. The keypoint category represents a predefined keypoint number, and the square bounding box includes at least the center position and side length.

[0169] The training annotation of the keypoint estimation model is constructed as follows: For the k-th keypoint visible in both the left and right eye training images, one of the keypoints is transformed to the offset coordinate system along the disparity direction according to a preset coordinate offset mapping. The keypoint without coordinate transformation and the transformed keypoint are respectively used as the midpoints of a pair of opposite sides of a square coding box. The distance between the two midpoints is used as the side length of the square coding box, and the keypoint number is used as the category of the coding box. When the k-th keypoint is not visible in either eye training image, no square coding box is generated for that keypoint. The above annotation method converts the correspondence between the left and right eye keypoints with the same number into a detection target with category and geometric size, so that the model does not need to perform left and right eye matching separately after outputting the two-dimensional keypoints.

[0170] In one specific embodiment, the keypoint estimation module includes a lightweight backbone network, a feature pyramid network, and a keypoint prediction module connected in sequence. The lightweight backbone network extracts image features of different resolutions from the binocular fused image; the feature pyramid network upsamples, concatenates, and fuseds the image features of different resolutions through convolution to generate multi-scale fused features; the keypoint prediction module outputs keypoint categories and square bounding boxes based on the multi-scale fused features. The multi-scale fused features take into account both the local positional details of the keypoints and a larger range of contextual information, which can reduce the problem of insufficient representation of keypoints of different sizes when only a single-scale feature is used.

[0171] During the model training phase, the keypoint prediction module includes a one-to-many prediction branch and a one-to-one prediction branch. The two prediction branches share a lightweight backbone network and a feature pyramid network, and respectively set up a keypoint category prediction structure and a square bounding box regression structure. The one-to-many prediction branch assigns multiple positive samples to the same square bounding box annotation to increase effective supervision information during the training phase; the one-to-one prediction branch assigns a matching prediction result to each square bounding box annotation, enabling the model to form a prediction relationship of one keypoint corresponding to one output result.

[0172] During training, the classification loss and bounding box regression loss for both prediction branches are calculated separately, and the model parameters are updated after summing them according to their respective weights. After training, the one-to-many prediction branch is disabled during the inference phase, and only the one-to-one prediction branch outputs the keypoint category and square bounding box. Since the one-to-one prediction branch has already established a one-to-one correspondence between the annotation and the prediction result, the non-maximum suppression processing for screening overlapping candidate boxes can be omitted, thereby reducing the post-processing steps in the inference phase.

[0173] The coordinate recovery module receives the keypoint category and its corresponding square bounding box, and determines the midpoints of a pair of opposite sides based on the center position and side length of the square bounding box. Following the midpoint arrangement order defined in the training annotation phase, one midpoint is designated as the keypoint without coordinate transformation, and the other midpoint is designated as the keypoint in the offset coordinate system. A coordinate transformation opposite to the preset coordinate offset mapping is then performed on the latter to restore its position in the original image coordinate system. Subsequently, the coordinate difference along the disparity direction between the recovered left and right eye keypoints with the same number is determined as the corresponding disparity. This process employs a deterministic transformation opposite to that constructed during training annotation, eliminating the need to re-search for the corresponding positions of the left and right eye keypoints.

[0174] The 3D coordinate calculation module receives the 2D coordinates and disparity of key points with the same number from both the left and right cameras, and calls upon the intrinsic parameters and baseline distance of the binocular cameras to calculate the 3D coordinates of each key point according to the binocular triangulation relationship. Since the input 2D coordinates, disparity, and calibration parameters have clear data sources, key point prediction, coordinate decoding, and 3D reconstruction can be seamlessly integrated.

[0175] The edge deployment module is used to export the keypoint estimation model to an intermediate model format after training, and then compile it into an inference engine using a hardware acceleration engine adapted to the edge device. In one specific embodiment, the trained model is exported to ONNX format, and then compiled into an inference engine adapted to Jetson edge devices using TensorRT. The edge deployment module also performs INT8 integer fixed-point quantization on the model and fuses adjacent convolution operations, batch normalization operations, and activation operations to reduce the data storage of model parameters and intermediate features, as well as the scheduling and data read / write overhead caused by executing multiple operators separately.

[0176] During inference, the edge device loads the inference engine, inputs the binocular fused image generated by the image preprocessing module into the inference engine, obtains the keypoint category and square encoding box through the keypoint estimation module, and then outputs the 3D coordinates of each keypoint through the coordinate recovery module and the 3D coordinate calculation module. The system completes binocular image feature extraction and joint encoding of left and right eye keypoints in one model, without the need to construct a dense disparity cost volume for the entire image, nor to perform a separate keypoint matching process after the model output. Therefore, it can reduce the computing resources occupied by repeated feature extraction, dense disparity calculation and post-processing, and provide a foundation for real-time 3D keypoint estimation on the edge.

[0177] The above are merely preferred embodiments of the present invention and do not limit the patent scope of the present invention. Any equivalent structural transformations made using the contents of the present invention's specification and drawings under the inventive concept of the present invention, or direct or indirect applications in other related technical fields, are within the patent protection scope of the present invention.

Claims

1. A real-time 3D keypoint estimation method based on binocular vision, characterized in that, include: The left and right eye images are acquired simultaneously using a calibrated binocular camera, and distortion correction and epipolar correction are performed. Establish a preset coordinate offset mapping between one eye-corrected image and another eye-corrected image, which is used to transform the coordinates of key points in the eye image to the offset coordinate system along the parallax direction; The left and right eye images are stitched together along the channel dimension to obtain a binocular fused image; the binocular fused image is input into a trained keypoint estimation model to extract multi-scale image features and output the keypoint category and square coding box corresponding to each visible keypoint; The key point category represents the key point number. The key point estimation model uses the following training annotation: For key points with the same number that are visible in both the left and right eye images, one of the key points is transformed according to the preset coordinate offset mapping. The two key points with the same number after transformation are respectively used as the midpoints of a pair of opposite sides of the square coding box, and the distance between the two midpoints is used as the side length of the square coding box. For key points that are not visible in any image, no corresponding square bounding box is generated; Based on the center position and side length of the square encoding frame and the preset coordinate offset mapping, the two-dimensional coordinates of the same numbered key points in the left and right eye images are recovered and the parallax is determined. The three-dimensional coordinates of each key point are calculated based on the two-dimensional coordinates, the parallax, and the calibration parameters of the binocular camera.

2. The real-time 3D keypoint estimation method according to claim 1, characterized in that, The establishment of the preset coordinate offset mapping includes: Traverse the binocular image pairs in the training set and the key points with the same number that are visible in both the left and right images of each binocular image pair. Determine the preset coordinate offset based on the coordinate difference between two key points with the same number along the disparity direction, so that after one key point is transformed, the two key points with the same number are arranged in a preset position order, and the distance between them along the disparity direction is not less than the preset minimum encoding size.

3. The real-time 3D keypoint estimation method according to claim 1, characterized in that, Recovering the two-dimensional coordinates and determining the disparity includes: Based on the center position and side length of the square encoding frame, determine the midpoints of a pair of opposite sides of the square encoding frame; Take one of the midpoints as the coordinates of the key point without coordinate transformation, and perform a coordinate transformation on the other midpoint that is opposite to the preset coordinate offset mapping to obtain the coordinates of the other key point; The coordinate difference between two key points along the parallax direction is determined as the parallax of the corresponding key point.

4. The real-time 3D keypoint estimation method according to claim 1, characterized in that, The keypoint estimation model includes a lightweight backbone network, a feature pyramid network, and a keypoint prediction module connected in sequence. The lightweight backbone network is used to extract image features of different resolutions from the binocular fused image; the lightweight backbone network includes multiple convolutional layers, C3K2 feature extraction layers, spatial pyramid pooling layers and C2PSA feature extraction layers arranged in sequence; The feature pyramid network is used to upsample, stitch together, and fused convolutional features of images at different resolutions to generate fused features at multiple scales. The feature pyramid network outputs three sets of fusion features with resolutions of 1 / 8, 1 / 16, and 1 / 32 of the resolution of the binocular fused image, respectively. The keypoint prediction module is used to predict the keypoint category and square coding box of each keypoint based on the fusion features.

5. The real-time 3D keypoint estimation method according to claim 4, characterized in that, The keypoint prediction module includes a one-to-many prediction branch and a one-to-one prediction branch during the training phase. The one-to-many prediction branch and the one-to-one prediction branch share the lightweight backbone network and the feature pyramid network, and respectively set the key point category prediction structure and the square coding box regression structure; The one-to-many prediction branch assigns multiple positive samples to each square bounding box label, and the one-to-one prediction branch assigns one prediction result to each square bounding box label.

6. The real-time 3D keypoint estimation method according to claim 5, characterized in that, During the training process of the key point estimation model, the classification loss and bounding box regression loss of the one-to-many prediction branch and the one-to-one prediction branch are calculated respectively, and the losses are weighted and summed. The classification loss uses binary cross-entropy loss, the bounding box regression loss uses full intersection-union loss, and the loss weight corresponding to the one-to-many prediction branch gradually decreases from the first weight in the early stage of training to the second weight in the later stage of training.

7. The real-time 3D keypoint estimation method according to claim 6, characterized in that, The first weight is 0.8, and the second weight is 0.1; During the inference phase of the keypoint estimation model, the one-to-many prediction branch is disabled, and the keypoint category and the square coding box are output by the one-to-one prediction branch, omitting non-maximum suppression processing.

8. The real-time 3D keypoint estimation method according to claim 1, characterized in that, When the epipolar correction ensures that keypoints with the same number in the left and right eye images are located in the same image row, the three-dimensional coordinates of the k-th keypoint are determined according to the following formula: ； ； ； ； in, Let the disparity be the value of the k-th key point. , These are the x-coordinates of the k-th keypoint in the left and right eye images, respectively. Let be the ordinate of the k-th key point in the left eye image. , , Let K be the three-dimensional coordinates of the k-th key point. Baseline distance, , For focal length parameters, , The coordinates of the main point.

9. A real-time 3D keypoint estimation system based on binocular vision, characterized in that, include: The binocular image acquisition module is used to simultaneously acquire left and right eye images using a calibrated binocular camera. The image preprocessing module is used to perform distortion correction and epipolar correction on two images, establish a preset coordinate offset mapping between one corrected image and the other corrected image, and stitch the two corrected images along the channel dimension to obtain a binocular fused image. The keypoint estimation module is used to input the binocular fused image into a trained keypoint estimation model, extract multi-scale image features, and output the keypoint category and square coding box corresponding to each visible keypoint. The keypoint category represents the keypoint number. The keypoint estimation model is trained and labeled as follows: for keypoints with the same number that are visible in both the left and right eye images, one of the keypoints is transformed to the offset coordinate system along the disparity direction according to the preset coordinate offset mapping. The two keypoints with the same number after transformation are respectively used as the midpoints of a pair of opposite sides of the square coding box, and the distance between the two midpoints is used as the side length. For keypoints that are not visible in either eye image, no corresponding square coding box is generated. The coordinate recovery module is used to recover the two-dimensional coordinates of key points with the same number in the left and right eye images and determine the parallax based on the center position, side length and preset coordinate offset mapping of the square encoding frame. The three-dimensional coordinate calculation module is used to calculate the three-dimensional coordinates of each key point based on the two-dimensional coordinates, the parallax, and the calibration parameters of the binocular camera.

10. The real-time 3D keypoint estimation system according to claim 1, characterized in that, The keypoint estimation module includes a lightweight backbone network, a feature pyramid network, and a keypoint prediction module. The lightweight backbone network is used to extract image features of different resolutions from the binocular fused image, the feature pyramid network is used to fuse the image features of different resolutions, and the key point prediction module is used to output the key point category and the square coding box based on the fused image features. The key point prediction module includes a one-to-many prediction branch and a one-to-one prediction branch during the training phase. The one-to-many prediction branch and the one-to-one prediction branch share the lightweight backbone network and the feature pyramid network. During the inference phase, the one-to-many prediction branch is deactivated, and the key point category and the square coding box are output through the one-to-one prediction branch.

Images