A robot binocular vision-based position sensing method and device and robot

Abstract

The present disclosure relates to a robot binocular vision-based position perception method and device and robot. The method comprises: acquiring a binocular image containing a second robot based on a binocular sensor, and detecting an image position of a target region of the second robot in the binocular image; determining first three-dimensional position information of the target region, an uncertainty parameter of the first three-dimensional position information, and a re-projection error of the target region in the binocular image according to the binocular image and the image position; calibrating the uncertainty parameter according to the re-projection error to obtain a calibrated uncertainty parameter; conservatively correcting the first three-dimensional position information according to the calibrated uncertainty parameter to obtain second three-dimensional position information; and taking the second three-dimensional position information as the three-dimensional position of the target region of the second robot, thereby outputting a three-dimensional position with a safety margin only relying on the binocular image of the robot, and improving the accuracy of target three-dimensional position perception.

Description

Technical Field

[0001] This disclosure relates to the field of artificial intelligence technology, and more specifically, to a position perception method, device, and robot based on binocular vision. Background Technology

[0002] In the robot boxing competition scenario, the participating robot relies on a head-mounted binocular camera to obtain the opponent's spatial pose information to support defensive (blocking, dodging, retreating) and safety control strategies; the system calculates the three-dimensional coordinates (including position and depth) of the opponent's key points through stereo matching of left and right views.

[0003] However, stereo matching is prone to ambiguity when faced with target occlusion, low-texture areas, or high-speed camera movement, leading to "false confidence" in the output depth values—that is, the system reports results with significant deviations in a deterministic form, but lacks a quantitative assessment of its own reliability. Therefore, improving the accuracy and reliability of target 3D perception in dynamic adversarial scenarios, while relying solely on stereo vision, has become an urgent technical challenge. Summary of the Invention

[0004] To overcome the problems existing in related technologies, this disclosure provides a position perception method, device and robot based on robot binocular vision.

[0005] In a first aspect, this disclosure provides a position perception method based on binocular vision for a robot, applied to a first robot. The method includes: acquiring a binocular image including a second robot based on a binocular sensor, and detecting the image position of a target region of the second robot in the binocular image; determining, based on the binocular image and the image position, first three-dimensional position information of the target region, an uncertainty parameter of the first three-dimensional position information, and a reprojection error of the target region in the binocular image, wherein the reprojection error is the deviation between the reprojection position obtained by reprojecting the first three-dimensional position information onto the binocular image and the image position; calibrating the uncertainty parameter based on the reprojection error to obtain a calibrated uncertainty parameter; conservatively correcting the first three-dimensional position information based on the calibrated uncertainty parameter to obtain second three-dimensional position information, wherein the conservative correction includes: translating the first three-dimensional position information along the line of sight of the first robot by a safe offset, the safe offset being positively correlated with the calibrated uncertainty parameter; and using the second three-dimensional position information as the three-dimensional position of the target region of the second robot.

[0006] Secondly, this disclosure provides a position perception device based on robot binocular vision, applied to a first robot. The device includes: a binocular image acquisition module, an uncertainty parameter determination module, an uncertainty parameter calibration module, a conservative correction module, and a three-dimensional position determination module. The system includes a binocular image acquisition module configured to acquire binocular images including the second robot based on a binocular sensor, and detect the image position of the target region of the second robot in the binocular images; an uncertainty parameter determination module configured to determine, based on the binocular images and the image position, a first three-dimensional position information of the target region, an uncertainty parameter of the first three-dimensional position information, and a reprojection error of the target region in the binocular images, wherein the reprojection error is the deviation between the reprojection position obtained by reprojecting the first three-dimensional position information onto the binocular images and the image position; an uncertainty parameter calibration module configured to calibrate the uncertainty parameter based on the reprojection error to obtain a calibrated uncertainty parameter; a conservative correction module configured to conservatively correct the first three-dimensional position information based on the calibrated uncertainty parameter to obtain second three-dimensional position information, wherein the conservative correction includes: translating the first three-dimensional position information by a safe offset along the line of sight of the first robot, the safe offset being positively correlated with the calibrated uncertainty parameter; and a three-dimensional position determination module using the second three-dimensional position information as the three-dimensional position of the target region of the second robot.

[0007] Thirdly, this disclosure provides a robot comprising: one or more processors; a memory; and one or more programs, wherein the one or more programs are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to implement the steps of the method described in the first aspect.

[0008] Through the above technical solution, the first robot acquires binocular images including those of the second robot based on a binocular sensor, and detects the image position of the target region of the second robot in the binocular images; based on the binocular images and image positions, it determines the first three-dimensional position information of the target region, the uncertainty parameter of the first three-dimensional position information, and the reprojection error of the target region in the binocular images, wherein the reprojection error is the deviation between the reprojected position obtained by reprojecting the first three-dimensional position information onto the binocular images and the image position; the uncertainty parameter is calibrated based on the reprojection error to obtain the calibrated uncertainty parameter; based on the calibrated uncertainty parameter, the first three-dimensional position information is conservatively corrected to obtain... The second three-dimensional position information is obtained, wherein the conservative correction includes: translating the first three-dimensional position information by a safe offset along the line of sight of the first robot, the safe offset being positively correlated with the calibrated uncertainty parameter; using the second three-dimensional position information as the three-dimensional position of the target area of ​​the second robot, thereby perceiving the target's three-dimensional position based solely on the binocular images acquired by the robot, obtaining the uncertainty of the perceived three-dimensional position, and conservatively correcting the perceived three-dimensional position according to the calibrated uncertainty, outputting a three-dimensional position with a safety margin, improving the accuracy of the target's three-dimensional position perception, and ensuring the safety of driving the robot to work based on the three-dimensional position.

[0009] Other features and advantages of this disclosure will be described in detail in the following detailed description section. Attached Figure Description

[0010] The accompanying drawings are provided to further illustrate the present disclosure and form part of the specification. They are used together with the following detailed description to explain the present disclosure, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart illustrating an exemplary embodiment of a position perception method based on robot binocular vision.

[0011] Figure 2 This is a flowchart illustrating an exemplary embodiment of a position perception method based on robot binocular vision.

[0012] Figure 3 This is a block diagram illustrating a position perception device based on binocular vision for robots, as shown in an exemplary embodiment.

[0013] Figure 4 This is a block diagram illustrating an exemplary embodiment of a robot. Detailed Implementation

[0014] The specific embodiments of this disclosure will be described in detail below with reference to the accompanying drawings. It should be understood that the specific embodiments described herein are for illustration and explanation only and are not intended to limit this disclosure.

[0015] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0016] It should be understood that the steps described in the method embodiments of this disclosure may be performed in different orders and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of this disclosure is not limited in this respect.

[0017] The term "comprising" and its variations as used herein are open-ended inclusions, meaning "including but not limited to". The term "based on" means "at least partially based on". The term "one embodiment" means "at least one embodiment"; the term "another embodiment" means "at least one additional embodiment"; the term "some embodiments" means "at least some embodiments". Definitions of other terms will be given in the description below.

[0018] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0019] In robot boxing, the robot relies on binocular cameras for 3D localization to drive defensive and control strategies. The system outputs deterministic estimates of the 3D coordinates of key points through stereo matching. However, target occlusion, missing textures, and rapid movement can lead to matching distortion, and the system cannot self-assess the reliability of this result. If the distortion value is directly used for control, it may trigger misjudged and dangerous actions (such as reckless advances). Therefore, existing technologies lack mechanisms for the downstream transmission of uncertainty and safety margin corrections, resulting in severely insufficient accuracy of binocular 3D perception in dynamic combat scenarios.

[0020] To address the aforementioned problems, this disclosure proposes a position perception method, apparatus, and robot based on binocular vision. The position perception method based on binocular vision provided by the embodiments of this disclosure will be described in detail below.

[0021] The position perception method based on robot binocular vision provided in this embodiment can be used for a first robot. The first robot refers to a machine device capable of performing tasks autonomously or semi-autonomously, including but not limited to industrial robots, service robots, and special-purpose robots; no specific limitation is made to the type of the first robot herein.

[0022] Figure 1 This is a flowchart illustrating an exemplary embodiment of a position perception method based on robot binocular vision, such as... Figure 1 As shown, it includes the following steps: Step S110: Acquire a binocular image including the second robot based on the binocular sensor, and detect the image position of the target area of ​​the second robot in the binocular image.

[0023] In some implementations, a binocular vision sensor mounted on the head of the first robot can acquire real-time images of the left and right eyes, including those of the second robot. The binocular vision sensor can include a first camera and a second camera horizontally arranged at a preset baseline distance, allowing implementation solely based on the head's binocular vision without requiring an external third-party perspective. This enables implementation even in scenarios where the robot's field of view is limited, improving the practicality of positional awareness.

[0024] The binocular images can be acquired using either a global shutter or a rolling shutter. These images can include acquisition timestamps and calibration version information, ensuring time synchronization and traceable calibration, and guaranteeing consistency in triangulation and reprojection error calculations.

[0025] The left and right images can be input into a pre-trained target detection network (such as YOLO or Faster R-CNN) to detect the target region of the second robot in the left image and the image location of the target region of the second robot in the right image, respectively. The target region may include at least one of the following: the main outline of the second robot, the head region, the torso region, or a pre-defined key component region.

[0026] Specifically, the first pixel coordinate range of the target region in the left-eye image and the second pixel coordinate range of the target region in the right-eye image can be extracted. These first and second pixel coordinate ranges are then used as the image position of the target region of the second robot in the binocular images for subsequent 3D coordinate calculation. Optionally, the result of detecting the image position of the target region in the binocular images can be output in the form of rectangular bounding boxes; each bounding box can contain the pixel coordinate information of the target region in the corresponding image.

[0027] Step S120: Based on the stereo image and the image position, determine the first three-dimensional position information of the target region, the uncertainty parameter of the first three-dimensional position information, and the reprojection error of the target region in the stereo image, wherein the reprojection error is the deviation between the reprojection position obtained by reprojecting the first three-dimensional position information onto the stereo image and the image position.

[0028] In some implementations, after obtaining the image position of the target region of the second robot in the binocular image, the three-dimensional position of the target region of the second robot can be obtained based on the calibration parameters of the binocular vision sensor of the first robot and the image position, and used as the first three-dimensional position information. The uncertainty parameter of this first three-dimensional position information can also be obtained. The uncertainty parameter of the three-dimensional position information can be used to characterize the depth and / or triangulation uncertainty of the three-dimensional position information, and can be determined through heteroscedasticity networks, stereo matching cost volume, or geometric residual statistics, without limitation here. For example, the uncertainty parameter of the three-dimensional position information can be determined by the linear propagation equation of the disparity error; alternatively, the uncertainty parameter of the three-dimensional position information can be calculated by constructing a Jacobian matrix of the three-dimensional coordinates with respect to the pixel coordinates.

[0029] As an feasible approach, a target model can be pre-set. This target model can be used to detect and output the three-dimensional position information of the target region in the binocular image and the corresponding uncertainty parameters. Specifically, the binocular image of the second robot can be input into the target model to obtain the first three-dimensional position information of the target region of the second robot and the uncertainty parameters of the first three-dimensional position information output by the target model.

[0030] In this embodiment, the first three-dimensional position information can be reprojected onto the reprojection position obtained from the binocular image, and the deviation between the reprojection position and the image position can be obtained as the reprojection error of the target region in the binocular image.

[0031] Step S130: Calibrate the uncertainty parameter according to the reprojection error to obtain the calibrated uncertainty parameter.

[0032] In some implementations, after obtaining the first three-dimensional position information of the target region, the uncertainty parameter of the first three-dimensional position information, and the reprojection error of the target region in the binocular image, the uncertainty parameter can be calibrated based on the reprojection error to obtain the calibrated uncertainty parameter.

[0033] The system can pre-set a monotonic mapping relationship between reprojection error and uncertainty parameter, and can calibrate the uncertainty parameter of the first three-dimensional position information based on this monotonic mapping relationship and the reprojection error of the target area in the binocular image to obtain the calibrated uncertainty parameter.

[0034] For example, the uncertainty parameter σ of the first three-dimensional position information and the reprojection error e_reproj of the target region in the stereo image are obtained. The uncertainty parameter σ can be logarithmically transformed and then input together with the reprojection error e_reproj into a mapping function g(·) with a preset monotonic mapping relationship. The output of the mapping function is then exponentially transformed to obtain the calibrated uncertainty parameter σ'.

[0035] Based on this, the reprojection error of the target region in the binocular image and the uncertainty parameter of the first three-dimensional position information can be substituted into the mapping function of the monotonic mapping relationship, so as to calibrate the uncertainty parameter according to the reprojection error and obtain the calibrated uncertainty parameter.

[0036] Step S140: Based on the calibrated uncertainty parameter, conservatively correct the first three-dimensional position information to obtain the second three-dimensional position information. The conservative correction includes: shifting the first three-dimensional position information by a safe offset along the line of sight of the first robot. The safe offset is positively correlated with the calibrated uncertainty parameter.

[0037] In some implementations, after obtaining the calibrated uncertainty parameter corresponding to the first three-dimensional position information, the first three-dimensional position information can be conservatively corrected based on the calibrated uncertainty parameter to obtain the second three-dimensional position information.

[0038] Specifically, the gaze direction of the binocular sensors can be determined as the gaze direction of the first robot during the acquisition of binocular images. After obtaining the calibrated uncertainty parameters, a safety offset can be acquired, and the first 3D position information can be corrected based on this safety offset to obtain the second 3D position information. The safety offset can be positively correlated with the calibrated uncertainty parameters, thus ensuring the robot's safety by adjusting the target's position based on a larger safety offset when the uncertainty in determining the target's location is high.

[0039] As an feasible approach, in the process of conservatively correcting the first three-dimensional position information based on the calibrated uncertainty parameters to obtain the second three-dimensional position information, the first three-dimensional position information can be translated by a safe offset along the line of sight of the first robot to obtain the second three-dimensional position information.

[0040] As another feasible approach, in the process of conservatively correcting the first three-dimensional position information based on the calibrated uncertainty parameters to obtain the second three-dimensional position information, the first three-dimensional position information can be translated by a safe offset along a preset safe extrapolation direction to obtain the second three-dimensional position information. The preset safe extrapolation direction can be set manually or determined through third-party experimental data, and is not limited here. For example, the preset safe extrapolation direction includes, but is not limited to, directions away from the second robot.

[0041] For example, the second three-dimensional position information can be obtained by conservatively correcting the first three-dimensional position information based on the calibrated uncertainty parameter using a conservative position correction formula. The conservative position correction formula can be: X' = ​​X + Δ ×

[0042] Where X' can represent the second three-dimensional position information obtained after conservative correction. Where X can represent the first three-dimensional position information. This can represent a unit vector along the line-of-sight direction or the safe extrapolation direction, used to specify the spatial direction of the offset correction. Here, Δ represents the safe offset, a scalar value that can be used to represent the distance of translation along the line-of-sight direction or the safe extrapolation direction. Based on this, the first 3D position information can be conservatively corrected according to the calibrated uncertainty parameters to obtain the second 3D position information, thereby improving the accuracy of 3D position perception and enhancing the accuracy and safety of robot control.

[0043] Optionally, the second three-dimensional position information may include the pixel coordinates of the target region in the corresponding image, the coordinates of the target region in the world coordinate system, and the uncertainty parameters corresponding to the second three-dimensional position information, etc., which are not limited here. The uncertainty parameters corresponding to the second three-dimensional position information can be understood as the calibrated uncertainty parameters corresponding to the first three-dimensional position information.

[0044] Step S150: Use the second three-dimensional position information as the three-dimensional position of the target area of ​​the second robot.

[0045] In some implementations, after determining the second three-dimensional position information corresponding to the target area of ​​the second robot, the second three-dimensional position information can be used as the three-dimensional position of the target area of ​​the second robot. Thus, when the depth / triangulation result of the three-dimensional position is uncertain, the original three-dimensional position is no longer blindly published, but a conservative three-dimensional position with a safety margin is output. This improves the accuracy of the target three-dimensional position perception based on binocular images, avoids dangerous actions driven by high uncertainty observations, and improves the safety of robot control.

[0046] After acquiring the three-dimensional position of the target area, the actions of the first robot against the second robot can be controlled based on this three-dimensional position. For example, the first robot can be controlled to perform defensive or offensive actions based on the second three-dimensional position information. Optionally, the actions of the first robot against the second robot can also be controlled based on the relationship between the uncertainty parameter in the second three-dimensional position information and a preset uncertainty parameter, including but not limited to controlling the joint twisting and translation distance of the first robot. For example, if the uncertainty parameter of the second three-dimensional position information is greater than the preset uncertainty parameter, the first robot can be controlled to retreat; if the uncertainty parameter of the second three-dimensional position information is less than or equal to the preset uncertainty parameter, the first robot can be controlled to perform defensive actions. This avoids the robot from performing risky actions when there is significant uncertainty in position perception, and outputs precise defensive actions when observation is clear, thus balancing the stability of robot operation and the safety of the confrontation.

[0047] In this scheme, the first robot acquires binocular images including those of the second robot based on a binocular sensor, and detects the image position of the target region of the second robot in the binocular images. Based on the binocular images and the image positions, the first three-dimensional position information of the target region, the uncertainty parameter of the first three-dimensional position information, and the reprojection error of the target region in the binocular images are determined. The reprojection error is the deviation between the reprojected position obtained by reprojecting the first three-dimensional position information onto the binocular images and the image position. The uncertainty parameter is calibrated based on the reprojection error to obtain the calibrated uncertainty parameter. Based on the calibrated uncertainty parameter, the first three-dimensional position information is conservatively corrected to obtain the... The system provides two-dimensional and three-dimensional position information. The conservative correction includes: translating the first three-dimensional position information by a safe offset along the line of sight of the first robot; the safe offset is positively correlated with the calibrated uncertainty parameter; using the second three-dimensional position information as the three-dimensional position of the target area for the second robot. This allows for the perception of the target's three-dimensional position based solely on the binocular images acquired by the robot, obtaining the uncertainty of the perceived three-dimensional position, and conservatively correcting the perceived three-dimensional position according to the calibrated uncertainty. This outputs a three-dimensional position with a safety margin, improving the accuracy of the target's three-dimensional position perception and ensuring the safety of driving the robot based on its three-dimensional position.

[0048] Figure 2 This is a flowchart illustrating a position perception method based on binocular vision for robots, applied to a first robot, such as... Figure 2 As shown, it includes the following steps: Step S201: Acquire a binocular image containing the second robot based on the binocular sensor, and detect the image position of the target area of ​​the second robot in the binocular image.

[0049] For a detailed description of step S201, please refer to the previous description of step S110, which will not be repeated here.

[0050] Step S202: Input the binocular image into the target model, obtain the first three-dimensional position information of the target region output by the target model, and the uncertainty parameter corresponding to the first three-dimensional position information.

[0051] In some implementations, the target model can be pre-set in the first robot or obtained by the first robot from an associated cloud or electronic device, without limitation. The target model can be used to detect the three-dimensional position information of the target region in the binocular image and the corresponding uncertainty parameter. Specifically, the binocular image of the second robot, acquired by the binocular sensor of the first robot, can be input into the target model to obtain the first three-dimensional position information of the target region of the second robot, and the corresponding uncertainty parameter. Thus, the target model outputs the predicted three-dimensional position and uncertainty parameter in one step, avoiding multiple inferences and shortening the latency of position perception.

[0052] In some implementations, before inputting the binocular image into the target model and obtaining the first three-dimensional position information of the target region output by the target model, as well as the uncertainty parameter corresponding to the first three-dimensional position information, a training sample set can be obtained, and a preset model can be trained based on the training sample set to obtain the target model.

[0053] The training sample set may include multiple binocular sample images, as well as uncertain parameters of the third-dimensional position information of the region to be perceived included in each binocular sample image.

[0054] Optionally, during the acquisition of the training sample set, multiple stereo sample images and the ground truth of the three-dimensional position information of the region to be perceived included in each stereo sample image can be acquired; the third three-dimensional position information of the region to be perceived in each stereo sample image can be determined based on the calibration parameters of the sensor that acquired each stereo sample image and the image position of the region to be perceived in each stereo sample image; the uncertain parameters of the third three-dimensional position information of the region to be perceived in each stereo sample image can be determined based on the ground truth of the three-dimensional position information of the region to be perceived in each stereo sample image and the third three-dimensional position information of the region to be perceived in each stereo sample image; and the training sample set can be acquired based on each stereo sample image and the uncertain parameters of the third three-dimensional position information of the region to be perceived in each stereo sample image.

[0055] The true value of the three-dimensional position information of the area to be perceived can be obtained through multi-view images. For example, multi-view images can be obtained from historical playback videos of the first robot's punching, moving, and other actions, and can be used to obtain the true position of the area to be perceived. Thus, the playback data and the true value of the multi-view images are used to generate samples with uncertainty parameter labels, providing supervision signals for model training and improving the accuracy of the model.

[0056] The preset model may include a heteroscedastic network (HSW). The HSW can be used to predict the 3D location information of the region to be perceived in a binocular sample image, as well as the uncertainty parameter of this 3D location information. The target model can be obtained by training the HSW loss on a training sample set.

[0057] As an implementable approach, the output of a heteroscedastic network can include the results of depth or line-of-sight observations of the region to be perceived in a binocular sample image (which can also be understood as three-dimensional position information), as well as the uncertainty parameters of the depth or line-of-sight observation results.

[0058] For example, heteroscedastic networks can be used to model the depth information of the region to be perceived in a stereo image. Specifically, the heteroscedastic network can be implemented based on a deep neural network. For the i-th input stereo sample image, the heteroscedastic network can output the distribution parameter pairs (μ) of the depth observation values ​​of the region to be perceived in that sample image. i , σ i Specifically, heteroscedastic networks assume that the depth observation value d of the region to be perceived in the i-th sample image is... i Obey μ i The mean is σ i ² represents a normal distribution of variance. Where μ i It can be used to characterize the expectation estimate of depth observations by heteroscedastic networks; σ i It can be used to characterize the depth observation value d i The degree of deviation from the true depth value (which can be understood as the uncertainty parameter of depth estimation); where σ i The larger the value, the lower the confidence level of the heteroscedastic network in the depth estimation.

[0059] Among them, the output (μ) of the heteroscedastic network i , σ i Yes, this can be used for subsequent calibration of uncertainty parameters and conservative correction of 3D position (such as making safer and more conservative decisions in the control strategy when the uncertainty parameter is large). Based on this, the heteroscedastic network architecture enables the system to distinguish between reliable and unreliable estimates in downstream tasks by simultaneously estimating depth and the corresponding uncertainty, thereby effectively improving the accuracy and robustness of 3D position perception.

[0060] As an feasible approach, the training sample set can be classified according to the occlusion state of the region to be perceived included in each binocular sample image and the proximity state of the region to be perceived relative to the sensor that acquired the binocular sample images, to obtain different categories of training sample subsets. A pre-defined model can then be trained based on these categorized training sample subsets to obtain a target model. The target model can also be used to predict the cause information corresponding to the uncertainty parameters of the 3D position information of the region to be perceived. This cause information can include at least one of the occlusion state and / or proximity state of the region to be perceived.

[0061] Among them, the occlusion state can be used to characterize the range of the area to be perceived that is occluded, and the proximity state can be used to characterize the range of distance between the area to be perceived and the sensor that acquires the binocular sample images.

[0062] In this embodiment, considering that occlusion / close proximity are high-uncertainty scenarios, in order to ensure the proportion and coverage of high-risk samples in training, the training sample set can be classified according to the occlusion state and close proximity state of the area to be perceived. This allows for training the model to improve the accuracy of target location perception and the accuracy of uncertainty parameter prediction for different categories of training sample subsets, thereby improving the accuracy and efficiency of model training.

[0063] In some implementations, the target model can be used to predict and output the cause information corresponding to the uncertainty parameters of the three-dimensional position information of the area to be sensed. The methods for outputting the cause information include, but are not limited to, sending the cause information to a related cloud or electronic device, converting the cause information into voice output, or converting the cause information into indicator light output, etc., and are not limited here.

[0064] After outputting the cause information, the system can respond to control commands input based on this cause information to control the actions of the first robot against the second robot. For example, the cause information may include the occluded area of ​​the region to be sensed being larger than a preset size, or the distance between the region to be sensed and the sensor acquiring the binocular sample images being less than a preset distance. The cause information can be output via voice prompts, and in response to control commands input by the user based on this cause information (such as controlling the first robot to retreat a target distance), the first robot's action against the second robot can be controlled to be backward, thus avoiding occlusion of the region to be sensed and increasing the distance between the region to be sensed and the sensor acquiring the binocular sample images, ensuring the safety of the robot confrontation.

[0065] Step S203: Obtain the reprojection position obtained by reprojecting the first three-dimensional position information onto the stereo image, and the deviation between the reprojection position and the image position, as the reprojection error of the target region in the stereo image.

[0066] In some implementations, the deviation between the reprojection position obtained by reprojecting the first three-dimensional position information onto the binocular image and the image position can be used as the reprojection error of the target region in the binocular image.

[0067] Optionally, the first 3D position information corresponding to the target area of ​​the second robot can be the estimated spatial coordinates in a preset world coordinate system calculated based on binocular stereo matching. Specifically, based on the pre-calibrated camera intrinsic and extrinsic parameter matrices (including rotation and translation vectors) of the first robot's binocular vision system, the first 3D position information can be projected from the world coordinate system to the image pixel coordinate system, obtaining its reprojected pixel position in the left and / or right eye images. Furthermore, the true image detection position (i.e., image position) of the target area in the corresponding binocular image can be extracted, and the pixel distance (e.g., Euclidean distance) between the reprojected pixel position and the true image detection position can be calculated. This pixel distance is used as the reprojection error of the target area in the binocular image. Thus, the degree of geometric consistency between the current 3D localization result and the 2D observation data is quantified through the reprojection error; the smaller the error, the higher the 3D localization accuracy.

[0068] Understandably, binocular vision's 3D solution relies on precise matching of corresponding points in the left and right eye images. However, interference such as image noise, occlusion, and texture loss can lead to matching deviations, which ultimately manifest as errors in the 3D coordinates. By introducing reprojection error as a closed-loop verification metric, the 2D observation data (image position) can be used as an unbiased benchmark to verify the reliability of the 3D positioning results, thereby achieving self-monitoring of 3D positioning accuracy. Furthermore, this reprojection error can serve as an adaptive adjustment basis for the Kalman filter observation noise covariance matrix and can also be used to determine whether the current output 3D point meets the accuracy threshold required for safety control, improving the accuracy of 3D position perception.

[0069] Step S204: Based on the preset monotonic mapping relationship, determine the calibrated uncertainty parameter according to the reprojection error and the uncertainty parameter. The preset monotonic mapping relationship is obtained by fitting the reprojection error of the third-dimensional position information of the region to be perceived included in each binocular sample image in the training sample set to each binocular sample image and the true value of the three-dimensional position information of the region to be perceived included in each binocular sample image.

[0070] In some implementations, after obtaining the uncertainty parameters of the first three-dimensional position information and the reprojection error of the target region in the stereo image, the calibrated uncertainty parameters can be determined based on a preset monotonic mapping relationship, according to the reprojection error and the uncertainty parameters. The preset monotonic mapping relationship can be obtained by fitting the reprojection error of the third three-dimensional position information of the region to be perceived included in each stereo sample image in the training sample set to the ground truth value of the three-dimensional position information of the region to be perceived included in each stereo sample image.

[0071] For example, the uncertainty parameter σ of the first three-dimensional position information and the reprojection error e_reproj of the target region in the binocular image are obtained; wherein, the uncertainty parameter σ can be logarithmically transformed and then input together with the reprojection error e_reproj into the preset monotonic mapping function g(·), and the output of the mapping function is exponentially transformed to obtain the calibrated uncertainty parameter σ'.

[0072] Step S205: If the calibrated uncertainty parameter and the reprojection error do not meet the constraint conditions, then the first three-dimensional position information is conservatively corrected according to the calibrated uncertainty parameter to obtain the second three-dimensional position information. The constraint conditions include that the calibrated uncertainty parameter is less than or equal to a first threshold and the reprojection error is less than or equal to a second threshold. The second three-dimensional position information includes a confidence level indicator.

[0073] In some implementations, after obtaining the calibrated uncertainty parameters, the calibrated uncertainty parameters can be matched with the reprojection error and constraints. Based on the matching result, it is determined whether to perform a conservative correction on the first 3D position information based on the calibrated uncertainty parameters to obtain the second 3D position information. The constraints can serve as a threshold for the output of the position perception result. If the threshold is exceeded, the original 3D position information is not output; instead, conservatively corrected 3D position information with a confidence level is output. If the threshold is below the threshold, the original 3D position information is output, thereby ensuring the effectiveness of position perception and the safety of robot control.

[0074] For example, the constraints may include a calibrated uncertainty parameter less than or equal to a first threshold and a reprojection error less than or equal to a second threshold. Furthermore, if it is determined that the calibrated uncertainty parameter and reprojection error do not meet the constraints, the first 3D position information can be conservatively corrected based on the calibrated uncertainty parameter to obtain the second 3D position information. It can be understood that the uncertainty-driven output gating includes dual thresholds for both the uncertainty parameter and the reprojection error, improving the accuracy of the position output.

[0075] For example, the calibrated uncertainty parameter σ' and the reprojection error e_reproj can be substituted into a preset uncertainty-driven release gating formula to determine the final output three-dimensional position Xout of the target region.

[0076] The configuration for the uncertainty-driven release gating formula can be: g= 1[σ'≤ Tσ]× 1[e_reproj ≤ Te] Xout = g×X+ (1 - g) × X' Where σ' can be used to characterize the uncertainty parameter after calibration. Where e_reproj can be used to characterize the reprojection error. Where Tσ can be used to characterize the preset first threshold, used to determine whether the uncertainty is within an acceptable range. Where Te can be used to characterize the preset second threshold, used to determine whether the reprojection error is within an acceptable range. 1[·] is an indicator function, which takes a value of 1 when its internal condition is met, and a value of 0 when it is not met.

[0077] The value of the gating signal g determines the branch selection of the output strategy. When the uncertainty parameter σ' is less than or equal to the first threshold Tσ, and the reprojection error e_reproj is less than or equal to the second threshold Te, the gating signal g = 1, indicating that the accuracy and reliability of the current perception result meet the preset safety standards. In this case, the first 3D position information X (such as the 3D position information directly predicted by the heteroscedasticity network) can be directly output. When the uncertainty parameter σ' is greater than the first threshold Tσ, and / or the reprojection error e_reproj is greater than the second threshold Te, the gating signal g = 0, indicating that the reliability of the current perception result is insufficient. In this case, instead of directly outputting the original 3D position information X, a conservative correction is made to the first 3D position information X based on the calibrated uncertainty parameter σ' to obtain the second 3D position information X', and this conservatively corrected 3D position is output. Thus, through the gating mechanism, when perception is reliable, high-precision positioning data is fully utilized to improve response performance; when perception is unreliable, a conservative strategy is actively adopted to ensure motion safety, achieving a dynamic balance between robot control performance and safety.

[0078] In some implementations, during the process of conservatively correcting the first three-dimensional position information based on the calibrated uncertainty parameter to obtain the second three-dimensional position information, the calibrated uncertainty parameter can be substituted into the conservative position correction formula to obtain the second three-dimensional position information calculated based on the conservative position correction formula.

[0079] For example, the conservative position correction formula is configured as follows: X' = ​​X + Δ ×

[0080] Δ = k × σ' Where X' can be used to represent the second 3D position information output after conservative correction. Where X can be used to represent the first 3D position information. It can be used to characterize a unit vector along the line-of-sight direction or the safety extrapolation direction, and can be used to specify the spatial direction of the correction offset. Here, Δ can be used to characterize the safety offset, which is a scalar value, and can be used to characterize the distance translated along the line-of-sight direction or the safety extrapolation direction. The value of the safety offset Δ can be determined by multiplying the calibrated uncertainty parameter σ' by the safety level coefficient k. Therefore, there is a positive correlation between Δ and σ': the larger the calibrated uncertainty parameter σ', the lower the reliability of the current 3D positioning result, and the larger the safety offset Δ becomes, thus automatically generating a safety margin proportional to the perceived risk.

[0081] Optionally, the safety level coefficient k can be determined based on the budgeted robot combat distance. For example, the value of k can be positively correlated with the maximum robot combat distance; the larger the maximum combat distance, the larger the safety level coefficient, ensuring sufficient safety compensation even when long-distance perception errors are amplified. Wherein, unit vector... It can be the reverse of the line of sight from the second robot to the first robot, so that the corrected position is closer to the robot than the original position, and the downstream decision module makes defensive or evasive actions based on the conservative premise that the opponent is closer than actually perceived.

[0082] In some implementations, the second and first three-dimensional position information may include corresponding confidence level identifiers for their respective three-dimensional position information. These confidence level identifiers can be used to indicate the confidence level of the three-dimensional position information. Specifically, the confidence level identifiers included in the second three-dimensional position information can be low-confidence identifiers, and the low-confidence identifiers included in the first three-dimensional position information can be high-confidence identifiers. This allows for the simultaneous output of the three-dimensional position information and the annotation of its confidence level, enabling the determination of the robot control method based on the three-dimensional position information, thereby improving the safety and accuracy of robot control.

[0083] Step S206: Use the second three-dimensional position information as the three-dimensional position of the target area of ​​the second robot.

[0084] For a description of step S206, please refer to the previous description of step S150, which will not be repeated here.

[0085] Step S207: Based on the confidence level indicator, control the first robot to act against the second robot.

[0086] In some implementations, the second three-dimensional position information may include a confidence level identifier, which can be used to control the actions of the first robot against the second robot. Thus, by including a confidence level identifier in the output three-dimensional position information, the robot control can utilize it for safety constraints, conservative defense, risk field and distance control, thereby improving the safety of robot control.

[0087] Optionally, the actions of the first robot against the second robot can be downgraded based on the confidence level of the second three-dimensional position information. For example, the actions of the first robot against the second robot can be downgraded from offensive to defensive, thereby ensuring the safety of the robot.

[0088] Step S208: If the calibrated uncertainty parameter is less than or equal to the first threshold, and the reprojection error is less than or equal to the second threshold, then the first three-dimensional position information is used as the three-dimensional position of the target area of ​​the second robot.

[0089] In some implementations, if the calibrated uncertainty parameter is less than or equal to a first threshold and the reprojection error is less than or equal to a second threshold, it can be determined that the calibrated uncertainty parameter and the reprojection error satisfy the constraint conditions, and the first three-dimensional position information can be used as the three-dimensional position of the target area of ​​the second robot, so as to control the first robot based on the three-dimensional position and improve the efficiency of robot control.

[0090] In some implementations, after the first robot completes the competition, the data (such as binocular images, 3D position information of target position perception, uncertainty parameters, cause information, threshold version, calculation formula, etc.) of the first robot during the competition can be reviewed and verified. The target model parameters can be adjusted according to the verification results to form a verifiable improvement closed loop, thereby improving the safety and reliability of robot control.

[0091] Compared to Figure 1 In the illustrated embodiment, before conservatively correcting the first three-dimensional position information based on the calibrated uncertainty parameter to obtain the second three-dimensional position information, it is determined that the calibrated uncertainty parameter and the reprojection error do not satisfy the following constraints: the calibrated uncertainty parameter is less than or equal to a first threshold, and the reprojection error is less than or equal to a second threshold. Thus, by using the dual thresholds of uncertainty and reprojection error representing geometric consistency, the three-dimensional position of the target area is jointly determined, avoiding high-risk position information from entering the robot control. Furthermore, when the uncertainty and reprojection error representing geometric consistency do not satisfy the constraints, a corrected three-dimensional position with a safety margin is output, improving the safety of robot control.

[0092] Furthermore, if the calibrated uncertainty parameter is less than or equal to the first threshold and the reprojection error is less than or equal to the second threshold, the first three-dimensional position information can be used as the three-dimensional position of the target area of ​​the second robot. Thus, under the constraint conditions that the uncertainty and the reprojection error representing geometric consistency meet the constraints, the original perceived three-dimensional position can be output, which improves the efficiency of position perception and ensures the safety of robot control.

[0093] In addition, this embodiment can also input the binocular image into the target model to obtain the first three-dimensional position information of the target region output by the target model, as well as the uncertainty parameter corresponding to the first three-dimensional position information; obtain the reprojection position obtained by reprojecting the first three-dimensional position information onto the binocular image, and the deviation between the reprojection position and the image position, as the reprojection error of the target region in the binocular image, so that the three-dimensional position information and the uncertainty parameter of the three-dimensional position information are output by the model at one time, avoiding the cumbersome process of multiple inferences and improving the efficiency and accuracy of position perception.

[0094] In addition, this embodiment can also determine the calibrated uncertainty parameter based on a preset monotonic mapping relationship, according to the reprojection error and uncertainty parameter. The preset monotonic mapping relationship is obtained by fitting the reprojection error of the third-dimensional position information of the region to be perceived included in each binocular sample image in the training sample set to each binocular sample image and the true value of the three-dimensional position information of the region to be perceived included in each binocular sample image. In this way, the original uncertainty parameter is corrected to a scale consistent with the empirical error corresponding to the reprojection error, which improves the margin confidence of the three-dimensional position conservative correction and improves the accuracy of position perception.

[0095] In addition, the second three-dimensional position information in this embodiment may include a confidence level indicator. This embodiment can also control the first robot to resist the second robot's actions based on the confidence level indicator after using the second three-dimensional position information as the three-dimensional position of the target area of ​​the second robot. Thus, when the three-dimensional position is uncertain, the original three-dimensional position is no longer blindly published, but a conservative three-dimensional position with a safety margin is output with a confidence level indicator, and the first robot's actions to resist the second robot are controlled based on the confidence level indicator, avoiding dangerous actions driven by high uncertainty observations and improving the safety of robot control.

[0096] Please refer to Figure 3The diagram illustrates a structural block diagram of a position sensing device based on binocular vision for robots, according to an embodiment of this application. The position sensing device 300 can be applied to a first robot. The position sensing device 300 may include: a binocular image acquisition module 310, an uncertainty parameter determination module 320, an uncertainty parameter calibration module 330, a conservative correction module 340, and a three-dimensional position determination module 350. Wherein: The binocular image acquisition module 310 is used to acquire binocular images including the second robot based on a binocular sensor, and to detect the image position of the target area of ​​the second robot in the binocular images.

[0097] The uncertainty parameter determination module 320 is used to determine, based on the stereo image and the image position, the first three-dimensional position information of the target region, the uncertainty parameter of the first three-dimensional position information, and the reprojection error of the target region in the stereo image, wherein the reprojection error is the deviation between the reprojection position obtained by reprojecting the first three-dimensional position information onto the stereo image and the image position.

[0098] Uncertainty parameter calibration module 330 is used to calibrate the uncertainty parameter according to the reprojection error to obtain the calibrated uncertainty parameter.

[0099] The conservative correction module 340 is used to conservatively correct the first three-dimensional position information according to the calibrated uncertainty parameter to obtain the second three-dimensional position information. The conservative correction includes: translating the first three-dimensional position information by a safe offset along the line of sight of the first robot. The safe offset is positively correlated with the calibrated uncertainty parameter.

[0100] The three-dimensional position determination module 350 uses the second three-dimensional position information as the three-dimensional position of the target area of ​​the second robot.

[0101] Optionally, the position perception device 300 based on robot binocular vision may further include a constraint condition determination module. This constraint condition determination module can be used to determine, before conservatively correcting the first three-dimensional position information based on the calibrated uncertainty parameter to obtain the second three-dimensional position information, that the calibrated uncertainty parameter and the reprojection error do not satisfy the following constraint conditions: the calibrated uncertainty parameter is less than or equal to a first threshold, and the reprojection error is less than or equal to a second threshold.

[0102] Optionally, the position perception device 300 based on robot binocular vision may further include a three-dimensional position determination submodule. This three-dimensional position determination submodule can be used to determine the three-dimensional position of the target region of the second robot if the calibrated uncertainty parameter is less than or equal to the first threshold and the reprojection error is less than or equal to the second threshold.

[0103] Optionally, the uncertainty parameter determination module 320 may be specifically used to input the stereo image into the target model, obtain the first three-dimensional position information of the target region output by the target model, and the uncertainty parameter corresponding to the first three-dimensional position information; obtain the deviation between the reprojection position obtained by reprojecting the first three-dimensional position information onto the stereo image and the image position, as the reprojection error of the target region in the stereo image.

[0104] Optionally, the position perception device 300 based on robot binocular vision may further include a model training module. The model training module can be used to acquire a training sample set before inputting the binocular image into a target model and obtaining the first three-dimensional position information of the target region output by the target model, and the uncertainty parameter corresponding to the first three-dimensional position information. The training sample set includes multiple binocular sample images and uncertainty parameters of the third three-dimensional position information of the region to be perceived included in each binocular sample image. Based on the training sample set, a preset model is trained to obtain the target model. The preset model includes a heteroscedasticity network (HSM), which is used to predict the three-dimensional position information of the region to be perceived included in the binocular sample image, and the uncertainty parameter of the three-dimensional position information. The target model is obtained by training the loss of the HSM based on the training sample set.

[0105] Optionally, the model training module can also be specifically used to classify the training sample set according to the occlusion state of the region to be perceived included in each binocular sample image in the training sample set and the proximity state of the region to be perceived relative to the sensor that acquired the binocular sample image, to obtain training sample subsets of different categories. The occlusion state is used to characterize the range of the region to be perceived that is occluded, and the proximity state is used to characterize the range of distance between the region to be perceived and the sensor that acquired the binocular sample image. The classification is based on the training sample subsets to train the preset model to obtain the target model. The target model is further used to predict the cause information corresponding to the uncertainty parameter of the three-dimensional position information of the region to be perceived. The cause information includes at least one of the occlusion state and / or proximity state of the region to be perceived.

[0106] Optionally, the position perception device 300 based on robot binocular vision may further include a robot control first module. The robot control first module may be used to output the cause information; and in response to control commands input based on the cause information, control the first robot to resist the actions of the second robot.

[0107] Optionally, the model training module can also be specifically used to acquire multiple stereo sample images, and the ground truth value of the three-dimensional position information of the region to be perceived included in each stereo sample image; determine the third three-dimensional position information of the region to be perceived in each stereo sample image based on the calibration parameters of the sensor that acquired each stereo sample image and the image position of the region to be perceived in each stereo sample image; determine the uncertainty parameter of the third three-dimensional position information of the region to be perceived in each stereo sample image based on the ground truth value of the three-dimensional position information of the region to be perceived in each stereo sample image and the third three-dimensional position information of the region to be perceived in each stereo sample image; and acquire the training sample set based on each stereo sample image and the uncertainty parameter of the third three-dimensional position information of the region to be perceived in each stereo sample image.

[0108] Optionally, the uncertainty parameter calibration module 330 can be specifically used to determine the calibrated uncertainty parameter based on a preset monotonic mapping relationship, according to the reprojection error and the uncertainty parameter. The preset monotonic mapping relationship is obtained by fitting the reprojection error of the third-dimensional position information of the region to be perceived included in each binocular sample image in the training sample set to each binocular sample image and the true value of the three-dimensional position information of the region to be perceived included in each binocular sample image.

[0109] Optionally, the second three-dimensional position information includes a confidence level indicator, and the position perception device 300 based on robot binocular vision may further include a second robot control module. The second robot control module can be used to control the first robot's actions against the second robot based on the confidence level indicator after the second three-dimensional position information is used as the three-dimensional position of the target area of ​​the second robot.

[0110] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described device and module can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0111] In the several embodiments provided in this application, the coupling between modules can be electrical, mechanical, or other forms of coupling.

[0112] Furthermore, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0113] In summary, this disclosure improves the accuracy of target 3D position perception by relying solely on binocular images acquired by the robot for target 3D position perception, obtains the uncertainty of the perceived 3D position, and conservatively corrects the perceived 3D position based on the calibrated uncertainty, outputting a 3D position with a safety margin.

[0114] The following will combine Figure 4 This application describes a robot.

[0115] Reference Figure 4 , Figure 4 A structural block diagram of a robot according to an embodiment of this application is shown. The method described above in this embodiment can be executed by the robot 400. The robot 400 may include a processor 401 and a memory 402. The robot 400 may also include one or more of a multimedia component 403, an input / output (I / O) interface 404, and a communication component 405.

[0116] The processor 401 controls the overall operation of the robot 400 to complete all or part of the steps in the aforementioned position perception method based on robot binocular vision. The memory 402 stores various types of data to support the operation of the robot 400. This data may include, for example, instructions for any application or method used to operate on the robot 400, and application-related data such as contact data, sent and received messages, images, audio, video, etc. The memory 402 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Read-Only Memory (PROM), Read-Only Memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. Multimedia component 403 may include a screen and an audio component. The screen may be, for example, a touchscreen, and the audio component is used to output and / or input audio signals. For example, the audio component may include a microphone for receiving external audio signals. The received audio signals may be further stored in memory 402 or transmitted via communication component 405. The audio component also includes at least one speaker for outputting audio signals. I / O interface 404 provides an interface between processor 401 and other interface modules, such as a keyboard, mouse, buttons, etc. These buttons may be virtual or physical buttons. Communication component 405 is used for wired or wireless communication between the robot 400 and other devices. Wireless communication includes, for example, Wi-Fi, Bluetooth, Near Field Communication (NFC), 2G, 3G, or 4G, or a combination of these. Therefore, the corresponding communication component 405 may include a Wi-Fi module, a Bluetooth module, or an NFC module.

[0117] In an exemplary embodiment, the robot 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the above-described position perception method based on robot binocular vision.

[0118] In another exemplary embodiment, a computer-readable storage medium including program instructions is also provided, which, when executed by a processor, implement the steps of the above-described position perception method based on robot binocular vision. For example, the computer-readable storage medium may be the memory 402 including the program instructions, which may be executed by the processor 401 of the robot 400 to complete the above-described position perception method based on robot binocular vision.

[0119] In another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a processor, which, when executed by the processor, implements the steps of the above-described position perception method based on robot binocular vision.

[0120] In yet another exemplary embodiment, a computer program product is also provided, which includes a computer program executable by a processor, which, when executed by the processor, implements the steps of the above-described position perception method based on robot binocular vision.

[0121] The preferred embodiments of this disclosure have been described in detail above with reference to the accompanying drawings. However, this disclosure is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this disclosure, various simple modifications can be made to the technical solutions of this disclosure, and these simple modifications all fall within the protection scope of this disclosure.

[0122] It should also be noted that the various specific technical features described in the above embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this disclosure will not describe the various possible combinations separately.

[0123] Furthermore, various different embodiments of this disclosure can be combined in any way, as long as they do not violate the spirit of this disclosure, they should also be regarded as the content disclosed in this disclosure.

Claims

1. A position perception method based on binocular vision for robots, applied to a first robot, characterized in that, The method includes: The binocular image, including the second robot, is acquired based on the binocular sensor, and the image position of the target area of ​​the second robot in the binocular image is detected. Based on the binocular image and the image position, the first three-dimensional position information of the target region, the uncertainty parameter of the first three-dimensional position information, and the reprojection error of the target region in the binocular image are determined. The reprojection error is the deviation between the reprojection position obtained by reprojecting the first three-dimensional position information onto the binocular image and the image position. The uncertainty parameter is calibrated based on the reprojection error to obtain the calibrated uncertainty parameter; Based on the calibrated uncertainty parameter, the first three-dimensional position information is conservatively corrected to obtain the second three-dimensional position information. The conservative correction includes: shifting the first three-dimensional position information by a safe offset along the line of sight of the first robot. The safe offset is positively correlated with the calibrated uncertainty parameter. The second three-dimensional position information is used as the three-dimensional position of the target area of ​​the second robot.

2. The method according to claim 1, characterized in that, Before performing a conservative correction on the first three-dimensional position information based on the calibrated uncertainty parameter to obtain the second three-dimensional position information, the method further includes: The calibrated uncertainty parameter and the reprojection error do not satisfy the following constraints: The calibrated uncertainty parameter is less than or equal to the first threshold, and the reprojection error is less than or equal to the second threshold.

3. The method according to claim 2, characterized in that, The method further includes: If the calibrated uncertainty parameter is less than or equal to the first threshold, and the reprojection error is less than or equal to the second threshold, then the first three-dimensional position information is used as the three-dimensional position of the target area of ​​the second robot.

4. The method according to claim 1, characterized in that, The step of determining the first three-dimensional position information of the target region, the uncertainty parameter of the first three-dimensional position information, and the reprojection error of the target region in the stereo image based on the stereo image and the image position includes: The binocular image is input into the target model to obtain the first three-dimensional position information of the target region output by the target model, and the uncertainty parameter corresponding to the first three-dimensional position information; The deviation between the reprojection position obtained by reprojecting the first three-dimensional position information onto the stereo image and the image position is used as the reprojection error of the target region in the stereo image.

5. The method according to claim 4, characterized in that, Before inputting the binocular image into the target model and obtaining the first three-dimensional position information of the target region output by the target model, and the uncertainty parameter corresponding to the first three-dimensional position information, the method further includes: Obtain a training sample set, wherein the training sample set includes multiple binocular sample images, and uncertain parameters of the third-dimensional position information of the region to be perceived included in each binocular sample image; The target model is obtained by training a preset model based on the training sample set. The preset model includes a heteroscedastic network, which is used to predict the three-dimensional position information of the region to be perceived in the binocular sample image and the uncertainty parameter of the three-dimensional position information. The target model is obtained by training the heteroscedastic network loss based on the training sample set.

6. The method according to claim 5, characterized in that, The step of training a preset model based on the training sample set to obtain the target model includes: Based on the occlusion state of the region to be perceived included in each binocular sample image in the training sample set and the proximity state of the region to be perceived relative to the sensor that acquired the binocular sample image, the training sample set is classified to obtain different categories of training sample subsets. The occlusion state is used to characterize the range of the region to be perceived that is occluded, and the proximity state is used to characterize the range of the distance between the region to be perceived and the sensor that acquired the binocular sample image. The classification is based on the training sample subset to train the preset model to obtain the target model. The target model is also used to predict the cause information corresponding to the uncertainty parameter of the three-dimensional position information of the region to be perceived. The cause information includes at least one of the occlusion state and / or close proximity state of the region to be perceived.

7. The method according to claim 6, characterized in that, The method further includes: Output the aforementioned cause information; In response to a control command input based on the cause information, the first robot is controlled to resist the action of the second robot.

8. The method according to claim 5, characterized in that, The acquisition of the training sample set includes: Acquire multiple binocular sample images, and the ground truth of the three-dimensional position information of the region to be perceived included in each binocular sample image; Based on the calibration parameters of the sensors that acquire each binocular sample image and the image position of the region to be perceived in each binocular sample image, the third and third-dimensional position information of the region to be perceived in each binocular sample image is determined. Based on the true value of the three-dimensional position information of the region to be perceived included in each binocular sample image and the third three-dimensional position information of the region to be perceived in each binocular sample image, the uncertain parameters of the third three-dimensional position information of the region to be perceived included in each binocular sample image are determined. The training sample set is obtained based on the uncertain parameters of the third-dimensional position information of the region to be perceived included in each binocular sample image and the binocular sample image. The step of calibrating the uncertainty parameter based on the reprojection error to obtain the calibrated uncertainty parameter includes: Based on a preset monotonic mapping relationship, the calibrated uncertainty parameter is determined according to the reprojection error and the uncertainty parameter. The preset monotonic mapping relationship is obtained by fitting the reprojection error of the third-dimensional position information of the region to be perceived included in each binocular sample image in the training sample set to each binocular sample image and the true value of the three-dimensional position information of the region to be perceived included in each binocular sample image.

9. The method according to any one of claims 1-8, characterized in that, The second three-dimensional position information includes a confidence level indicator. After using the second three-dimensional position information as the three-dimensional position of the target region of the second robot, the method further includes: Based on the confidence level indicator, control the actions of the first robot against the second robot.

10. A robot, characterized in that, include: processor; Memory used to store processor-executable instructions; The processor is configured to perform the steps of the method according to any one of claims 1-9.

Images