Variable-focus binocular ranging method and system fusing instance segmentation and self-calibration

Abstract

The invention belongs to the technical field of computer vision and three-dimensional perception, and particularly relates to a variable-focus binocular ranging method and system fusing instance segmentation and self-calibration, and the method comprises the specific steps: carrying out the self-calibration through employing an image collected by a binocular camera through a self-calibration method based on deep learning, and obtaining the internal and external parameters of the camera; acquiring a mask area, a category label and confidence of each detection instance in the image; the method comprises the following steps: preprocessing left and right images collected by a binocular camera, inputting the preprocessed left and right images into a stereo matching network to obtain a dense disparity map, and adjusting the dense disparity map to the size of an original image; calculating a dense depth map by using the focal length obtained by self-calibration, and converting the dense depth map into a pseudo-color depth map; extracting a depth value corresponding to a mask area pixel from the dense depth map, taking a depth median of the area as a distance estimation value of the instance, and superposing the distance estimation value in an original image collected by a camera to obtain an instance depth superposition map; and respectively converting the pseudo-color depth map and the instance depth overlay map into image messages and publishing the image messages.

Description

Technical Field

[0001] This invention belongs to the field of computer vision and 3D perception technology, specifically relating to a variable-focus binocular ranging method and system that integrates instance segmentation and self-calibration. Background Technology

[0002] Binocular vision ranging technology has been widely used in various fields such as robot navigation, 3D reconstruction, intelligent monitoring, and augmented reality due to its advantages such as not requiring active signal transmission, high accuracy, and low cost. Traditional binocular ranging methods typically rely on binocular cameras with a fixed focal length, and obtain their internal parameters (such as focal length, principal point position, and distortion coefficient) and external parameters (such as roll angle and pitch angle) through manual offline calibration, and then complete depth recovery through parallax calculation. These methods can achieve relatively stable results in static scenes or experimental environments, but they face many challenges in real-world complex environments.

[0003] First, traditional methods heavily rely on manual calibration, and the accuracy of this calibration directly determines the reliability of the depth calculation results. However, in dynamic operating scenarios, cameras may experience changes in internal parameters due to external impacts, zoom adjustments, or temperature variations, rendering the original calibration results invalid and requiring recalibration. This significantly reduces the system's flexibility and practicality. Especially in binocular systems with zoom capabilities, the focal length, as a key intrinsic parameter, changes continuously with zoom, making it difficult for traditional methods to update calibration parameters in real time, leading to a substantial decrease in ranging accuracy.

[0004] Secondly, most existing ranging systems only focus on acquiring dense depth maps at the pixel level, lacking the ability to identify and segment specific objects in the image, making it difficult to meet the semantic perception needs of "which object is how far away" in real-world scenarios. Although some systems attempt to combine object detection models to obtain object location information, they are usually based on bounding box recognition, which can only identify the approximate location of objects and is difficult to achieve pixel-level region extraction. Furthermore, their ability to distinguish between multiple similar targets is insufficient, limiting their application in manipulation and interaction tasks.

[0005] In addition, existing systems generally treat image acquisition, parameter estimation, target recognition and depth calculation as independent modules, lacking a unified software architecture, which is not conducive to the real-time operation and embedded deployment of the system.

[0006] In summary, existing binocular ranging technologies still have significant shortcomings in terms of dynamic focal length adaptation, autonomous calibration capability, instance-level semantic perception, and system integration. Therefore, there is an urgent need to propose a novel ranging method and system that can achieve automatic calibration without human intervention, and combine instance segmentation to complete independent identification and accurate ranging of multiple targets, while possessing good real-time performance and system integration to meet the practical needs of intelligent perception systems in complex environments. Summary of the Invention

[0007] In view of this, the present invention provides a variable-focus binocular ranging method and system that integrates instance segmentation and self-calibration. The binocular camera can zoom to adapt to different distances for ranging, uses the acquired images for real-time self-calibration, and performs instance segmentation on the images to achieve the purpose of displaying image information of each object instance.

[0008] The technical solution for implementing the present invention is as follows: In a first aspect, the present invention provides a variable-focus binocular ranging method that integrates instance segmentation and self-calibration, the specific process of which is as follows: Camera self-calibration: Using images acquired by a stereo camera, a self-calibration method based on deep learning is used to perform self-calibration and obtain the camera's intrinsic and extrinsic parameters; Instance segmentation: Obtain the mask region, class label, and confidence score for each detected instance in the image; Depth estimation: The left and right images acquired by the binocular camera are preprocessed and input into the stereo matching network to obtain a dense disparity map. The dense disparity map is then adjusted to the original image size. Using the focal length obtained from self-calibration, a dense depth map is calculated and converted into a pseudo-color depth map. The depth values ​​corresponding to the pixels of the mask region are extracted from the dense depth map. The median depth of this region is used as the distance estimate of the instance and superimposed on the original image acquired by the camera to obtain an instance depth overlay map. Image visualization: Convert pseudo-color depth maps and instance depth overlay maps into image messages and publish them.

[0009] Optionally, the self-calibration method based on deep learning described in this invention performs self-calibration to obtain camera intrinsic and extrinsic parameters; specifically: Step 2.1: Extract multi-scale features using a neural network based on an encoder-decoder architecture, and predict the perspective field pixel by pixel. , and For each pixel, the up vector and latitude angle are used to generate corresponding confidence weights pixel by pixel. and ; Step 2.2: Construct a nonlinear weighted least squares objective function This allows the theoretical perspective field generated by the current camera parameters to be displayed. Compared with network prediction results Keep it as consistent as possible; update camera parameters incrementally to minimize residuals; Step 2.3: When the iteration termination condition is met, output the optimized camera intrinsic and extrinsic parameters.

[0010] Optionally, the objective function of this invention is:

[0011] in, This indicates the camera's internal and external parameters that need to be optimized.

[0012] Optionally, the present invention employs The algorithm updates the camera parameters incrementally to minimize the residuals. The algorithm update formula is:

[0013] in, For Jacobian matrices, Here, λ is the Hessian matrix, and λ is the damping factor. This is a diagonal matrix of confidence weights. The residual vector is used to iteratively update the camera parameters. .

[0014] Optionally, the specific process of instance segmentation described in this invention is as follows: Step 3.1: After the input image is processed by the backbone convolutional neural network to extract multi-level feature maps, the results are output to the target detection branch and the pixel-level segmentation branch respectively. Step 3.2: The object detection branch generates a set of candidate detection boxes, each containing an instance category label and a confidence score; the pixel-level segmentation branch generates a set of pixel-level masks, each mask marking the pixel affiliation of the corresponding target region in the image space. Step 3.3: Information fusion. The fused instance segmentation result consists of a mask, category label, and confidence score.

[0015] Optionally, the process of obtaining the dense disparity map of the present invention is as follows: Step 4.1: The preprocessed left image Compared with the right figure Simultaneously, the input is fed into a shared-weights 2D convolutional feature extraction network to extract multi-scale feature maps. and ; Step 4.2: Within the set parallax search range Inside, the feature map on the right. By parallax Perform a horizontal translation and compare it with the feature map in the left image. The inner product is calculated using the feature vectors at corresponding positions to construct the 3D relevant cost volume. ; Step 4.3: Place the cost body The input is a 3D convolutional network for cost aggregation, and information from neighboring pixels and different disparity planes is fused in the spatial and disparity dimensions to obtain an initial disparity map. ; Step 4.4: Initial disparity map Convolutional gated recurrent unit with contextual feature input in the left figure The refinement module is iteratively updated to obtain the final high-precision disparity map. This refers to a dense parallax map.

[0016] Optionally, the preprocessing described in this invention is as follows: adjusting the left and right images to the same multiple, converting the images to RGB and normalizing them.

[0017] Optionally, the calculation process of the dense depth map described in this invention is as follows: Step 4.5: Utilize the geometric relationships of binocular imaging to generate a high-precision disparity map. Convert to depth map for:

[0018] Where f is the focal length, b is the baseline length, d is the parallax, and Z is the depth.

[0019] Optionally, the present invention sets up a binocular image topic to be subscribed to and a pseudo-color depth map and instance depth overlay map topic to be published; the pseudo-color depth map and instance depth overlay map are converted into image messages respectively and published to the corresponding topics. The image messages include instance region, type, confidence and depth information. Then the left and right image caches are cleared and the next frame is processed.

[0020] In a second aspect, the present invention provides a variable-focus binocular ranging system that integrates instance segmentation and self-calibration, comprising: a variable-focus binocular camera, a camera self-calibration module, an instance segmentation module, a depth estimation module, and a visualization module; A zoomable dual-target camera is used to acquire left and right synchronized frame images; The camera self-calibration module uses images acquired by a stereo camera to perform self-calibration based on a deep learning-based self-calibration method to obtain the camera's intrinsic and extrinsic parameters; The instance segmentation module is used to obtain the mask region, class label, and confidence score of each detected instance in the image; The depth estimation module is used to preprocess the left and right images and input them into the stereo matching network to obtain a dense disparity map. The dense disparity map is then adjusted to the size of the original image. The dense depth map is calculated using the focal length obtained by self-calibration and converted into a pseudo-color depth map. The depth values ​​corresponding to the pixels of the mask region are extracted from the dense depth map, and the median depth of the region is used as the distance estimate of the instance and superimposed on the original image acquired by the camera to obtain an instance depth overlay map. Visualization module: Converts pseudo-color depth maps and instance depth overlay maps into image messages and publishes them.

[0021] Beneficial effects: (1) By introducing a self-calibration method based on deep learning, this invention realizes real-time estimation of camera internal and external parameters without human intervention. It can automatically update focal length and attitude information under zoom conditions, thereby ensuring the accuracy and stability of depth calculation and overcoming the drawback of traditional offline calibration methods where calibration results fail when focal length changes or camera attitude shifts.

[0022] (2) This invention integrates instance segmentation technology into the ranging process, enabling independent identification and segmentation of different targets of the same category. It extracts the precise region, category, and confidence information of each target and combines it with the depth map calculated by dense stereo matching. The median depth of the mask region is used to achieve precise ranging at the instance level. This design not only improves the semantic level of scene understanding but also significantly enhances the applicability of the system in tasks such as multi-target interaction and robot manipulation.

[0023] (3) This invention constructs an integrated data processing framework based on the Robot Operating System (ROS), and achieves the organic integration of functions such as self-calibration, instance segmentation, stereo matching, depth estimation and visualization through node-based design. This architecture has efficient data flow and scalability between modules, and can achieve stable real-time operation on embedded platforms and edge computing devices, meeting the deployment requirements in complex dynamic environments. Attached Figure Description

[0024] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0025] Figure 1 This is a schematic diagram of the overall structure of the present invention; Figure 2 This is a system processing flowchart of the present invention; Figure 3 This is a flowchart illustrating the principle of the self-calibration method in this invention; Figure 4 This is a flowchart illustrating the principle of instance segmentation in this invention. Figure 5 This is a flowchart illustrating the principle of stereo matching and depth estimation in this invention. Figure 6 This is a schematic diagram illustrating the principle of using a binocular camera to estimate the depth of an object in this invention. Detailed Implementation

[0026] The embodiments of the present invention will now be described in detail with reference to the accompanying drawings.

[0027] It should be noted that, in the absence of conflict, the following embodiments and features can be combined with each other; and, based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0028] It should be noted that various aspects of embodiments within the scope of the appended claims are described below. It will be apparent that the aspects described herein can be embodied in a wide variety of forms, and any particular structure and / or function described herein is merely illustrative. Based on this disclosure, those skilled in the art will understand that one aspect described herein can be implemented independently of any other aspect, and two or more of these aspects can be combined in various ways. For example, any number of aspects set forth herein can be used to implement the device and / or practice the method. Additionally, this device and / or method can be implemented using structures and / or functionalities other than one or more of the aspects set forth herein.

[0029] Technical feature definition: Self-calibration: refers to estimating the camera's intrinsic and extrinsic parameters directly from a single image without prior manual calibration; Instance segmentation refers to the process of identifying and separating different objects belonging to the same category in an image. Confidence level: refers to the degree of certainty that a system is certain about the predicted category of an object; Depth: refers to the physical distance between an object in an image and the camera lens.

[0030] like Figure 1-2 As shown in the figure, an embodiment of this application provides a variable-focus binocular ranging method that integrates instance segmentation and self-calibration. The specific process is as follows: Camera self-calibration: Using images acquired by a stereo camera, a self-calibration method based on deep learning is used to perform self-calibration and obtain the camera's intrinsic and extrinsic parameters; Instance segmentation: Obtain the mask region, class label, and confidence score for each detected instance in the image; Depth estimation: The left and right images acquired by the binocular camera are preprocessed and input into the stereo matching network to obtain a dense disparity map. The dense disparity map is then adjusted to the original image size. Using the focal length obtained from self-calibration, a dense depth map is calculated and converted into a pseudo-color depth map. The depth values ​​corresponding to the pixels of the mask region are extracted from the dense depth map. The median depth of this region is used as the distance estimate of the instance and superimposed on the original image acquired by the camera to obtain an instance depth overlay map. Image visualization: Convert pseudo-color depth maps and instance depth overlay maps into image messages and publish them.

[0031] In this embodiment, the process is as follows: Step 1, System Initialization: Initialize nodes, load the camera self-calibration model (used to perform the deep learning self-calibration process), the instance segmentation model (used to perform the instance segmentation process), and the stereo matching network, set the stereo image topics to subscribe to and the pseudo-color depth map and instance depth overlay map topics to publish, and set the camera baseline length; Step 2, Image Acquisition and Caching: Input the left and right images from the stereo camera respectively, cache the left and right images of the current frame for subsequent processing. Each time a left image is received, the image processing flow is triggered. Step 3: Camera self-calibration and image preprocessing: Input the left image into the self-calibration model to estimate the camera's intrinsic and extrinsic parameters. The estimated intrinsic parameters include the camera's focal length and vertical field of view, while the extrinsic parameters include the camera's roll and pitch angles. The camera's focal length can be used to estimate the depth of objects. Image preprocessing involves adjusting the left and right images to the same magnification, converting the images to RGB and normalizing them to meet the input requirements of the stereo matching network. Step 4: Instance segmentation processing: Input the image on the left into the instance segmentation model to obtain the mask, class label and confidence score of each detected instance (i.e., target); Step 5: Stereo Matching and Pseudo-Color Depth Map Generation: Input the preprocessed left and right images into the stereo matching network to obtain a dense disparity map, and then adjust the dense disparity map to the original image size; using the self-calibrated focal length and known baseline, calculate the dense depth map and convert it into a pseudo-color depth map. The depth calculation method is as follows: (1) Where f is the focal length (in pixels), b is the baseline length (in meters), d is the parallax (in pixels), and Z is the depth (in meters). Step Six: Instance Region Depth Estimation and Instance Depth Overlay Map Generation: Based on the mask region of each detected target (i.e., instance), extract the depth values ​​corresponding to the pixels of the mask region from the depth map, and use the median depth of that region as the distance estimate of the object. Assign a unique color to each instance, draw its outline and a semi-transparent mask to visualize the instance. Display the category, confidence score, and depth near the edge of the instance in the original left image, generate an instance depth overlay map, and simultaneously display the self-calibration results and video frame rate in the upper left corner of the image.

[0032] Step 7, Message Publishing: Convert the pseudo-color depth map and the instance depth overlay map into image messages respectively, publish them to the corresponding topics, then clear the left and right image caches and wait for the next frame to be processed.

[0033] This method constructs an integrated data processing framework based on a robot operating system. By establishing nodes in the workspace, it can receive left and right images acquired by binocular cameras in real time, and sequentially complete camera self-calibration, instance segmentation, and depth estimation tasks. Furthermore, it displays the contour, category, confidence level, and distance information of each object on the object.

[0034] like Figure 1-2 As shown in the figure, an embodiment of this application provides a variable-focus binocular ranging system that integrates instance segmentation and self-calibration, including: a variable-focus binocular camera, a camera self-calibration module, an instance segmentation module, a depth estimation module, and a visualization module; A zoomable dual-target camera is used to acquire left and right synchronized frame images; The camera self-calibration module uses images acquired by a stereo camera to perform self-calibration based on a deep learning-based self-calibration method to obtain the camera's intrinsic and extrinsic parameters; The instance segmentation module is used to obtain the mask region, class label, and confidence score of each detected instance in the image; The depth estimation module is used to preprocess the left and right images and input them into the stereo matching network to obtain a dense disparity map. The dense disparity map is then adjusted to the size of the original image. The dense depth map is calculated using the focal length obtained by self-calibration and converted into a pseudo-color depth map. The depth values ​​corresponding to the pixels of the mask region are extracted from the dense depth map, and the median depth of the region is used as the distance estimate of the instance and superimposed on the original image acquired by the camera to obtain an instance depth overlay map. Visualization module: Converts pseudo-color depth maps and instance depth overlay maps into image messages and publishes them.

[0035] The device used in this embodiment is a RealSense D435i binocular camera with a binocular baseline. It is 50 mm.

[0036] like Figure 3 As shown, this embodiment proposes a self-calibration module that employs a deep learning-based self-calibration method to acquire camera intrinsic and extrinsic parameters end-to-end based on a single image. The specific process is as follows: Step 2.1: Input a single image, extract multi-scale features using a neural network based on an encoder-decoder architecture, and predict the perspective field pixel by pixel. That is, the up vector of each pixel. with latitude angle Simultaneously, generate corresponding confidence weights for each pixel. and This is used to measure the reliability of each pixel's predicted value for subsequent optimizations. Indicates the direction of gravity corresponding to the world coordinates of a pixel in the camera's field of view; latitude angle. This indicates the angle between the line of sight at that pixel and the horizontal plane.

[0037] Step 2.2: Using these predicted geometric cues as observation inputs, construct a nonlinear weighted least squares objective function. This allows the theoretical perspective field generated by the current camera parameters to be displayed. Compared with network prediction results As consistent as possible. The objective function is... (2) in, This indicates the camera's intrinsic and extrinsic parameters to be optimized. and These are the theoretical vector and latitude angle calculated from the current camera parameters, respectively. The objective function is controlled by camera parameters such as gravity direction, focal length, and distortion coefficients, and exhibits nonlinear characteristics. Therefore, it utilizes... The iterative optimization module updates the camera parameters incrementally to minimize the residuals. The algorithm update formula is: (3) in, For Jacobian matrices, Here, λ is the Hessian matrix, and λ is the damping factor. This is a diagonal matrix of confidence weights. The residual vector is used to iteratively update the camera parameters. .

[0038] Step 2.3, when When the value is sufficiently small or the maximum number of iterations is reached, stop the optimization and output the optimized camera intrinsic and extrinsic parameters.

[0039] like Figure 4 As shown, the instance segmentation module proposed in this invention achieves accurate pixel-level segmentation of individuals of various categories in a scene through multi-scale feature extraction, detection, and fusion of segmentation results, providing target regions for depth estimation. The specific process is as follows: Step 3.1: Input Image First, the backbone convolutional neural network extracts multi-level feature maps to obtain multi-scale information from low-level details to high-level semantics. After feature fusion, the features are then fed into the detection branch and the segmentation branch, respectively.

[0040] Step 3.2: The object detection branch generates a set of candidate detection boxes, each containing an instance category label and confidence score; the pixel-level segmentation branch generates a set of pixel-level masks, each mask marking the pixel affiliation of the corresponding target region in the image space.

[0041] Step 3.3: The fused instance segmentation result consists of a mask, category label, and confidence score, which can accurately distinguish different categories and individual targets in the scene. This result will serve as the input to the subsequent depth estimation module, providing an accurate pixel-level range for each target region.

[0042] like Figure 5 As shown, the stereo matching and depth estimation module in this embodiment achieves high-precision measurement of target spatial distance through feature extraction, cost volume construction and cost aggregation, disparity refinement, and depth calculation. The specific process is as follows: Step 4.1, Binocular Image Input and Feature Extraction: The preprocessed left view image Image with right view Simultaneously, the input is fed into a shared-weights 2D convolutional feature extraction network to extract multi-scale feature maps. and The preprocessing described above involves adjusting the left and right images to the same ratio, converting the images to RGB, and normalizing them.

[0043] Step 4.2, Cost Body Construction: Within the set parallax search range Inside, the feature map on the right. By parallax Perform a horizontal translation and compare it with the feature map in the left image. The inner product of the feature vectors at corresponding positions is calculated to construct the 3D relevant cost volume. The calculation formula is as follows: (4) in, These are the feature maps for the left and right images, respectively. Number of channels; This represents the final generated 3D cost volume; the inner product operation characterizes the matching similarity between the left and right feature maps at corresponding positions.

[0044] Step 4.3, Cost Aggregation: Cost body The input is a 3D convolutional network for cost aggregation, fusing information from neighboring pixels and different disparity planes in both spatial and disparity dimensions. This makes the matching cost more consistent across both dimensions, resulting in more stable and accurate disparity estimation, and ultimately outputting an initial disparity map. .

[0045] Step 4.4, Parallax Refinement: Initial disparity map Convolutional gated recurrent unit with contextual feature input in the left figure The refinement module reduces matching errors in occluded and weakly textured areas through an iterative update mechanism, resulting in a final high-precision disparity map. This is a dense disparity map, which is then adjusted to the original image size.

[0046] Step 4.5, Depth Calculation Utilizing the geometric relationships of binocular imaging, a high-precision disparity map is generated. Convert to depth map .like Figure 6 As shown, the depth at a certain location can be calculated based on camera parameters and a disparity map.

[0047] The distance from the baseline to the line connecting the projection centers of the two cameras any point in three-dimensional space The imaging point of the left camera is The imaging point of the right camera is Line segment and Representing points in three-dimensional space The horizontal coordinates projected onto the imaging planes of the left and right cameras are then the points The parallax between the left and right cameras can be defined as follows: (5) According to the theory of similar triangles, we can conclude that: (6) in, Indicates focal length.

[0048] You can get points Distance to the center plane of projection : (7) This invention constructs a binocular depth perception system based on a robot operating system, integrating camera self-calibration, instance segmentation, and depth estimation technologies. It estimates the camera focal length and calculates the depth map in real time using left and right eye images, and finally presents the left image with instance depth annotations by combining the instance segmentation results.

[0049] In summary, this invention enables a complete process from binocular image input to target depth visualization output. The above are merely preferred embodiments of this invention and are not intended to limit the scope of protection of this invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this invention should be included within the scope of protection of this invention.

Claims

1. A variable-focus binocular ranging method integrating instance segmentation and self-calibration, characterized in that, The specific process is as follows: Camera self-calibration: Using images acquired by a stereo camera, a self-calibration method based on deep learning is used to perform self-calibration and obtain the camera's intrinsic and extrinsic parameters; Instance segmentation: Obtain the mask region, class label, and confidence score for each detected instance in the image; Depth estimation: The left and right images acquired by the binocular camera are preprocessed and input into the stereo matching network to obtain a dense disparity map. The dense disparity map is then adjusted to the original image size. Using the focal length obtained from self-calibration, a dense depth map is calculated and converted into a pseudo-color depth map. The depth values ​​corresponding to the pixels of the mask region are extracted from the dense depth map. The median depth of this region is used as the distance estimate of the instance and superimposed on the original image acquired by the camera to obtain an instance depth overlay map. Image visualization: Convert pseudo-color depth maps and instance depth overlay maps into image messages and publish them.

2. The variable-focus binocular ranging method based on fusion instance segmentation and self-calibration according to claim 1, characterized in that, The deep learning-based self-calibration method performs self-calibration to obtain camera intrinsic and extrinsic parameters; specifically: Step 2.1: Extract multi-scale features using a neural network based on an encoder-decoder architecture, and predict the perspective field pixel by pixel. , and For each pixel, the up vector and latitude angle are used to generate corresponding confidence weights pixel by pixel. and ; Step 2.2: Construct a nonlinear weighted least squares objective function This allows the theoretical perspective field generated by the current camera parameters to be displayed. Compared with network prediction results Keep it as consistent as possible; update camera parameters incrementally to minimize residuals; Step 2.3: When the iteration termination condition is met, output the optimized camera intrinsic and extrinsic parameters.

3. The variable-focus binocular ranging method based on fusion instance segmentation and self-calibration according to claim 2, characterized in that, The objective function is: in, This indicates the camera's internal and external parameters that need to be optimized.

4. The variable-focus binocular ranging method based on fusion instance segmentation and self-calibration according to claim 3, characterized in that, use The algorithm updates the camera parameters incrementally to minimize the residuals. The algorithm update formula is: in, For Jacobian matrices, Here, λ is the Hessian matrix, and λ is the damping factor. This is a diagonal matrix of confidence weights. The residual vector is used to iteratively update the camera parameters. .

5. The variable-focus binocular ranging method based on fusion instance segmentation and self-calibration according to claim 1, characterized in that, The specific process of instance segmentation is as follows: Step 3.1: After the input image is processed by the backbone convolutional neural network to extract multi-level feature maps, the results are output to the target detection branch and the pixel-level segmentation branch respectively. Step 3.2: The object detection branch generates a set of candidate detection boxes, each containing an instance category label and a confidence score; the pixel-level segmentation branch generates a set of pixel-level masks, each mask marking the pixel affiliation of the corresponding target region in the image space. Step 3.3: Information fusion. The fused instance segmentation result consists of a mask, category label, and confidence score.

6. The variable-focus binocular ranging method based on fusion instance segmentation and self-calibration according to claim 1, characterized in that, The process of obtaining the dense disparity map is as follows: Step 4.1: The preprocessed left image Compared with the right figure Simultaneously, the input is fed into a shared-weights 2D convolutional feature extraction network to extract multi-scale feature maps. and ; Step 4.2: Within the set parallax search range Inside, the feature map on the right. By parallax Perform a horizontal translation and compare it with the feature map in the left image. The inner product is calculated using the feature vectors at corresponding positions to construct the 3D relevant cost volume. ; Step 4.3: Place the cost body The input is a 3D convolutional network for cost aggregation, and information from neighboring pixels and different disparity planes is fused in the spatial and disparity dimensions to obtain an initial disparity map. ; Step 4.4: Initial disparity map Convolutional gated recurrent unit with contextual feature input in the left figure The refinement module is iteratively updated to obtain the final high-precision disparity map. This refers to a dense parallax map.

7. The variable-focus binocular ranging method based on fusion instance segmentation and self-calibration according to claim 6, characterized in that, The preprocessing involves adjusting the left and right images to the same ratio, converting the images to RGB and normalizing them.

8. The variable-focus binocular ranging method based on fusion instance segmentation and self-calibration according to claim 6, characterized in that, The calculation process for the dense depth map is as follows: Step 4.5: Utilize the geometric relationships of binocular imaging to generate a high-precision disparity map. Convert to depth map for: Where f is the focal length, b is the baseline length, d is the parallax, and Z is the depth.

9. The variable-focus binocular ranging method based on fusion instance segmentation and self-calibration according to claim 1, characterized in that, Set the topics for subscribing to in stereo images and the topics for publishing pseudo-color depth maps and instance depth overlay maps; convert the pseudo-color depth maps and instance depth overlay maps into image messages and publish them to the corresponding topics. The image messages include instance regions, types, confidence levels, and depth information. Then clear the left and right image caches and wait for the next frame to be processed.

10. A variable-focus binocular ranging system integrating instance segmentation and self-calibration, characterized in that, include: The system includes a variable-focus binocular camera, a camera self-calibration module, an instance segmentation module, a depth estimation module, and a visualization module. A zoomable dual-target camera is used to acquire left and right synchronized frame images; The camera self-calibration module uses images acquired by a stereo camera to perform self-calibration based on a deep learning-based self-calibration method to obtain the camera's intrinsic and extrinsic parameters; The instance segmentation module is used to obtain the mask region, class label, and confidence score of each detected instance in the image; The depth estimation module is used to preprocess the left and right images and input them into the stereo matching network to obtain a dense disparity map. The dense disparity map is then adjusted to the size of the original image. The dense depth map is calculated using the focal length obtained by self-calibration and converted into a pseudo-color depth map. The depth values ​​corresponding to the pixels of the mask region are extracted from the dense depth map, and the median depth of the region is used as the distance estimate of the instance and superimposed on the original image acquired by the camera to obtain an instance depth overlay map. Visualization module: Converts pseudo-color depth maps and instance depth overlay maps into image messages and publishes them.

Images