A virtual laser point cloud generation and optimization method and device based on a binocular camera

Abstract

The embodiment of the present disclosure provides a virtual laser point cloud generation and optimization method and device based on a binocular camera, relates to the technical field of embodied robots, and the method comprises the following steps: collecting images of an environment through a binocular camera arranged on an embodied robot to obtain left and right perspective environment images; preprocessing the left and right perspective environment images; performing stereo matching on the preprocessed left and right perspective environment images to generate a parallax map; generating a virtual laser point cloud according to the parallax map and calibration parameters of the binocular camera; optimizing the quality of the virtual laser point cloud; performing semantic labeling on the virtual laser point cloud after quality optimization to assign semantic category labels to each point; and converting the virtual laser point cloud with semantic category labels into a virtual laser point cloud that is adapted to a target task to be executed by the embodied robot, so as to be called when the embodied robot executes the target task. In this way, a virtual laser point cloud with high quality and adapted to the target task to be executed can be generated.

Description

Technical Field

[0001] This disclosure relates to the field of embodied robot technology, and in particular to a method and apparatus for generating and optimizing virtual laser point clouds based on binocular cameras. Background Technology

[0002] Currently, embodied robots rely heavily on three-dimensional perception of their environment when performing tasks such as navigation, grasping, and obstacle avoidance. LiDAR has become the mainstream three-dimensional perception sensor due to its high ranging accuracy and minimal susceptibility to lighting conditions. However, its high cost, high power consumption, and space requirements severely limit the economical deployment of embodied robots in home, factory, warehouse, and lightweight service scenarios. Summary of the Invention

[0003] In a first aspect, embodiments of this disclosure provide a method for generating and optimizing virtual laser point clouds based on a binocular camera, the method comprising: Images of the environment around the embodied robot are captured by binocular cameras deployed on the embodied robot, obtaining environmental images from the left and right perspectives; Preprocess the environmental images from the left and right perspectives; Stereo matching is performed on the preprocessed left and right view environmental images to generate a disparity map; Based on the parallax map and the focal length, baseline distance, and principal point coordinates in the calibration parameters of the binocular camera, a virtual laser point cloud of the environment in which the embodied robot is located is generated. Optimize the quality of the virtual laser point cloud in the environment of the embodied robot; Semantic annotation is performed on the optimized virtual laser point cloud, and each point is assigned a semantic category label. The virtual laser point cloud with semantic category labels is converted into a virtual laser point cloud adapted to the target task to be performed by the embodied robot, so that the embodied robot can call it when performing the target task.

[0004] In some possible implementations of the first aspect, images of the environment in which the embodied robot is located are acquired by using a binocular camera deployed on the embodied robot to obtain left and right view environmental images, including: The hardware trigger signal generated by the synchronization signal generator is transmitted to the left and right cameras of the binocular camera via the synchronization signal line. In response to the same hardware trigger signal, the left and right cameras are controlled to simultaneously capture images of the environment in which the embodied robot is located, thus obtaining environmental images from both left and right perspectives.

[0005] In some possible implementations of the first aspect, preprocessing of the left and right view environment images includes: The distortion correction is performed on the environmental images of the left and right viewpoints based on the distortion coefficients in the calibration parameters of the binocular camera to eliminate the radial and tangential distortion of the lens. Based on the rotation matrix and translation vector in the calibration parameters of the binocular camera, stereo correction is performed on the distortion-corrected left and right view environmental images so that the projection positions of the same spatial point in the left and right view environmental images are located on the same horizontal epipolar line. Image enhancement is performed on the left and right view environmental images after stereo correction; image enhancement includes noise reduction, contrast enhancement and illumination compensation.

[0006] In some possible implementations of the first aspect, stereo matching is performed on the preprocessed left and right view environment images to generate a disparity map, including: For each pixel in the preprocessed left and right view environmental images, multiple features are fused within the set disparity search range to calculate the mixed cost, and the matching cost of each pixel under each disparity value is obtained. Construct an image pyramid of the preprocessed left and right view environment images. The image pyramid contains multiple scale levels. At the coarsest level of the image pyramid, the matching cost of each pixel at each disparity value is dynamically planned and aggregated to obtain the aggregation cost at the coarsest level. The disparity value with the smallest aggregation cost is selected for each pixel at the coarsest level through disparity selection to generate the disparity map at the coarsest level. The disparity map of the previous scale level is propagated to the adjacent finer scale level as the initial disparity estimate for each pixel in the finer scale level. In the finer scale level, the disparity search range is narrowed based on the initial disparity estimate of each pixel, and the matching cost of each pixel at each disparity value is recalculated within the narrowed disparity search range. The recalculated matching cost is dynamically aggregated to obtain the aggregate cost of the finer scale level. The disparity value with the minimum aggregate cost is selected for each pixel in the finer scale level through disparity selection to generate the disparity map of the finer scale level. The disparity map propagation, disparity search range shrinking, matching cost recalculation, dynamic programming aggregation, and disparity selection are performed step by step until the most detailed disparity map is generated. During the dynamic programming aggregation process at each scale level, the disparity smoothing penalty parameter is adaptively adjusted according to the local texture richness to control disparity continuity. Perform left-right consistency detection on the disparity map at the finest scale level, mark inconsistent pixels as unreliable regions, and use the disparity values ​​in the neighborhood to fill and correct the unreliable regions; Subpixel interpolation is performed on the disparity map after fill correction. A deep learning network is used to enhance the interpolated disparity map to obtain the final disparity map.

[0007] In some possible implementations of the first aspect, a virtual laser point cloud of the embodied robot's environment is generated based on the disparity map and the focal length, baseline distance, and principal point coordinates from the calibration parameters of the binocular camera, including: For each pixel in the disparity map, the depth value of the pixel is calculated based on the pixel's disparity value, the horizontal focal length in the focal length, and the baseline distance; Based on the pixel's depth value, focal length, and principal point coordinates, the pixel's image coordinates are transformed from the image coordinate system to the camera coordinate system, and combined with the depth value, the three-dimensional coordinates of the spatial point are obtained. Based on the three-dimensional coordinates of each spatial point, a virtual laser point cloud is generated to represent the environment in which the embodied robot is located.

[0008] Among the possible implementations of the first aspect, the quality of the virtual laser point cloud of the embodied robot's environment is optimized, including: The virtual laser point cloud is denoised, and the denoised virtual laser point cloud is then downsampled using a voxel grid to obtain a reduced-density virtual laser point cloud. Detect sparse regions in de-density virtual laser point clouds and obtain semantic information corresponding to the sparse regions from preprocessed left and right view environmental images; Determine the target category to which the sparse region belongs based on semantic information, and extract the geometric distribution features of existing points at the edge of the sparse region from the reduced-density virtual laser point cloud. Based on the geometric distribution characteristics of existing points at the edge of the sparse region and the geometric completion prior corresponding to the target category, points are added to the sparse region. The color information of each pixel in the reduced-density virtual laser point cloud after point filling is extracted from the preprocessed left and right view environmental images, and each point is assigned a color attribute to obtain a complete virtual laser point cloud. If a historical optimized virtual laser point cloud exists, the complete virtual laser point cloud is registered and fused with the historical optimized virtual laser point cloud to obtain the current optimized virtual laser point cloud; if no historical optimized virtual laser point cloud exists, the complete virtual laser point cloud is used as the current optimized virtual laser point cloud.

[0009] In some possible implementations of the first aspect, points are added to the sparse region based on the geometric distribution characteristics of existing points at the edge of the sparse region and the geometric completion prior corresponding to the target category, including: Based on the geometric distribution characteristics of existing points at the edge of the sparse region, the normal, curvature and local density of existing points at the edge of the sparse region are extracted, and an envelope surface is constructed by contracting along the normal to limit the space of points that can be added in the sparse region. The geometric completion prior corresponding to the target category is applied to the space of points that can be completed. The implicit shape field in the geometric completion prior is used to calculate the shape conformity of each position in the space of points that can be completed. The curvature of the existing points at the edge of the sparse region is used as the curvature constraint boundary condition of the implicit shape field. The shape conformity calculation result and the curvature constraint boundary condition are solved simultaneously to obtain the expected point density distribution and expected normal field in the space of points that can be completed. Based on the desired point density distribution and desired normal field, spatial advancement is carried out from the existing points at the edge of the sparse region into the interior of the sparse region within the space where points can be supplemented, generating multiple candidate points, and adjusting the spacing of the candidate points according to the local density of the existing points at the edge of the sparse region. Conflict detection is performed between multiple candidate points and existing points at the edge of the sparse region. Candidate points with a spacing smaller than the local density threshold in the geometric distribution feature are removed. The positions of the retained points are fine-tuned based on the normals of existing points at the edge of the sparse region to optimize geometric continuity.

[0010] In some possible implementations of the first aspect, semantic annotation is performed on the quality-optimized virtual laser point cloud, assigning semantic category labels to each point, including: A two-dimensional semantic segmentation network is used to perform pixel-level semantic segmentation on the preprocessed left and right view environment images to obtain two-dimensional semantic category labels for each pixel; A 3D semantic segmentation network is used to perform point-level semantic segmentation on the quality-optimized virtual laser point cloud to obtain the 3D semantic category label of each point. The two-dimensional semantic category labels of the corresponding pixels of each point are weighted and fused with the three-dimensional semantic category labels of each point to obtain the semantic category labels of each point; The semantic category labels of each point are optimized for cross-frame consistency to eliminate semantic category label jumps between adjacent frames.

[0011] In some possible implementations of the first aspect, the virtual laser point cloud with semantic category labels is converted into a virtual laser point cloud adapted to the target task to be performed by the embodied robot, including: Based on the coordinate and data format requirements of the target task to be performed by the embodied robot, the virtual laser point cloud with semantic category labels is transformed in coordinate and data format to generate a virtual laser point cloud adapted to the target task.

[0012] Secondly, embodiments of this disclosure provide a virtual laser point cloud generation and optimization device based on a binocular camera, the device comprising: The acquisition module is used to acquire images of the environment in which the embodied robot is located through binocular cameras deployed on the embodied robot, and obtain environmental images from the left and right perspectives; The preprocessing module is used to preprocess the environmental images from the left and right perspectives; The matching module is used to perform stereo matching on the preprocessed left and right view environmental images to generate a disparity map; The generation module is used to generate a virtual laser point cloud of the environment in which the embodied robot is located, based on the parallax map and the focal length, baseline distance, and principal point coordinates in the calibration parameters of the binocular camera. The optimization module is used to optimize the quality of the virtual laser point cloud in the environment where the embodied robot is located; The annotation module is used to perform semantic annotation on the optimized virtual laser point cloud, assigning semantic category labels to each point. The conversion module is used to convert virtual laser point clouds with semantic category labels into virtual laser point clouds adapted to the target task to be performed by the embodied robot, so that the embodied robot can call them when performing the target task.

[0013] Thirdly, embodiments of this disclosure provide an electronic device comprising: at least one processor; and a memory communicatively connected to the at least one processor; the memory storing instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform the method described above.

[0014] Fourthly, embodiments of this disclosure provide a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the methods described above.

[0015] In summary, the embodiments of this disclosure acquire environmental images from both left and right perspectives using dual-axis cameras deployed on the embodied robot. These images are then preprocessed, stereo matching is used to generate a disparity map, and virtual laser point clouds are generated in real-time based on calibration parameters such as the focal length, baseline distance, and principal point coordinates of the dual-axis cameras. This replaces costly, power-intensive, and space-consuming lidar and depth cameras with limited range and susceptibility to ambient light interference. While maintaining 3D perception accuracy, this significantly reduces sensor hardware costs, system power consumption, and size constraints, enabling the embodied robot to be economically deployed in homes, factories, warehouses, and lightweight service scenarios. Furthermore, by optimizing the quality and semantically labeling the virtual laser point clouds, a high-quality semantic point cloud with less noise, stronger consistency, and semantic category labels for each point is obtained, enhancing the robot's semantic understanding of the environment. Finally, the virtual laser point cloud with semantic category labels is converted into a form compatible with the target task to be performed by the embodied robot, enabling direct integration of perception data with tasks such as navigation, grasping, and obstacle avoidance, effectively improving the real-time performance, accuracy, and success rate of task execution.

[0016] It should be understood that the description in the Summary of the Invention is not intended to limit the key or essential features of the embodiments of this disclosure, nor is it intended to restrict the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description. Attached Figure Description

[0017] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. The drawings are provided for a better understanding of the invention and are not intended to limit the scope of this disclosure. In the drawings, the same or similar reference numerals denote the same or similar elements, wherein: Figure 1 A flowchart illustrating a method for generating and optimizing virtual laser point clouds based on a binocular camera, provided in this embodiment of the disclosure; Figure 2 A structural diagram of a virtual laser point cloud generation and optimization device based on a binocular camera provided in an embodiment of this disclosure; Figure 3 This is a structural diagram of an electronic device provided in an embodiment of the present disclosure. Detailed Implementation

[0018] To make the objectives, technical solutions, and advantages of the embodiments of this disclosure clearer, the technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this disclosure.

[0019] Furthermore, the term "and / or" in this article is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. Additionally, the character " / " in this article generally indicates that the preceding and following related objects have an "or" relationship.

[0020] To address the technical problems mentioned in the background art, this disclosure provides a method, apparatus, device, and storage medium for generating and optimizing virtual laser point clouds based on binocular cameras. It acquires environmental images from left and right perspectives using dual-axis cameras deployed on an embodied robot, and sequentially performs preprocessing, stereo matching to generate a disparity map, and real-time generation of virtual laser point clouds based on calibration parameters such as the focal length, baseline distance, and principal point coordinates of the dual-axis cameras. This replaces costly, power-intensive, and space-consuming lidar, as well as depth cameras with limited operating range and susceptibility to ambient light interference. While maintaining 3D perception accuracy, it significantly reduces sensor requirements. The limitations of hardware cost, system power consumption, and size constraints enable embodied robots to be economically deployed in homes, factories, warehouses, and lightweight service scenarios. Simultaneously, by optimizing and semantically annotating virtual laser point clouds, high-quality semantic point clouds with less noise, stronger consistency, and semantic category labels for each point are obtained, enhancing the robot's semantic understanding of the environment. Furthermore, the virtual laser point clouds with semantic category labels are converted into a form adapted to the target tasks to be performed by the embodied robot, enabling direct integration of perception data with tasks such as navigation, grasping, and obstacle avoidance, effectively improving the real-time performance, accuracy, and success rate of task execution.

[0021] The following detailed description, with reference to the accompanying drawings and specific embodiments, illustrates a method, apparatus, device, and storage medium for generating and optimizing virtual laser point clouds based on a binocular camera, as provided in this disclosure.

[0022] Figure 1 A flowchart illustrating a method for generating and optimizing virtual laser point clouds based on a binocular camera, as provided in this disclosure, is shown below. Figure 1 As shown, method 100 may include the following steps: S110 uses a binocular camera deployed on the embodied robot to collect images of the environment in which the embodied robot is located, obtaining environmental images from the left and right perspectives.

[0023] In some embodiments, the binocular cameras deployed on the embodied robot include a left camera and a right camera, which are rigidly connected to maintain a stable relative position, with a baseline distance set to 15-20 cm. Optionally, each camera is an industrial-grade global shutter camera with a resolution of not less than 1280×720, a frame rate of not less than 60fps, a sensor size of not less than 1 / 2.3 inches, a pixel size of not less than 3.0μm, a dynamic range of not less than 60dB, and an interface type of GigE Vision or USB 3.0. Simultaneously, the left and right cameras have hardware synchronization capabilities, supporting hardware-level frame synchronization. During image acquisition, a hardware trigger signal generated by a synchronization signal generator can be transmitted to the left and right cameras respectively via a synchronization signal line. The left and right cameras respond to the same hardware trigger signal, synchronously capturing images of the environment in which the embodied robot is located, obtaining left and right perspective environmental images.

[0024] It is worth noting that before image acquisition, the binocular camera needs to be accurately calibrated to obtain its calibration parameters. The calibration process consists of two steps: intrinsic parameter calibration and extrinsic parameter calibration. Intrinsic parameter calibration is used to calibrate the binocular camera's focal length (fx, fy), principal point coordinates (cx, cy), and distortion coefficients (k1, k2, k3, p1, p2). Extrinsic parameter calibration is used to calibrate the binocular camera's rotation matrix R, translation vector T, and baseline distance B.

[0025] S120 performs preprocessing on the environmental images from the left and right perspectives.

[0026] In some embodiments, distortion correction is performed on the left and right view environmental images based on distortion coefficients (k1, k2, k3, p1, p2) in the calibration parameters of the binocular camera to eliminate lens radial and tangential distortion. Specifically, k1, k2, and k3 in the distortion coefficients can be used to perform radial distortion correction on the left and right view environmental images to eliminate lens radial distortion, and p1 and p2 in the distortion coefficients can be used to perform tangential distortion correction on the left and right view environmental images to eliminate lens tangential distortion.

[0027] Based on the rotation matrix R and translation vector T in the calibration parameters of the binocular camera, stereo correction is performed on the distortion-corrected left and right view environmental images to ensure that the projected positions of the same spatial point in the left and right view environmental images are located on the same horizontal epipolar line. Stereo correction can be implemented using the Bouguet algorithm.

[0028] Image enhancement is performed on the left and right view environmental images after stereo correction, including denoising, contrast enhancement, and illumination compensation. Denoising can be achieved using Gaussian filtering, bilateral filtering, or nonlocal mean filtering algorithms; contrast enhancement can be achieved using histogram equalization, Gamma correction, or the CLAHE algorithm; and illumination compensation can be achieved using the Retinex algorithm or homomorphic filtering algorithms.

[0029] S130: Perform stereo matching on the preprocessed left and right view environmental images to generate a disparity map.

[0030] In some embodiments, for each pixel in the preprocessed left and right view environmental images, multiple features are fused within a set disparity search range to calculate the hybrid cost, thereby obtaining the matching cost of each pixel at each disparity value. These multiple features may include grayscale features, gradient features and Census transform features, color space features, texture features, etc. The specific method for calculating the hybrid cost can be as follows: calculate the matching cost under each feature separately, and then perform weighted fusion according to preset weight coefficients to obtain the final matching cost. The disparity search range can be set according to the baseline distance and the expected minimum or maximum ranging distance, for example, it can be set to 0 to 128 pixels.

[0031] An image pyramid is constructed from the preprocessed left and right view environmental images, containing multiple scale levels. This image pyramid construction can be achieved using the Gaussian pyramid algorithm, which involves layer-by-layer Gaussian filtering and downsampling of the preprocessed left and right view environmental images to obtain multiple scale levels from coarse to fine.

[0032] At the coarsest level of the image pyramid, the matching costs of each pixel at each disparity value are dynamically programmed and aggregated to obtain the aggregated cost at the coarsest level. Disparity selection is then used to choose the disparity value with the minimum aggregated cost for each pixel at the coarsest level, generating the disparity map at the coarsest level. The dynamic programming aggregation here employs a semi-global matching (SGM) algorithm, selecting eight directions for cost aggregation.

[0033] The disparity map from the previous scale level is propagated to adjacent finer scale levels as the initial disparity estimate for each pixel at the finer scale level. At the finer scale level, the disparity search range is narrowed based on the initial disparity estimates for each pixel (i.e., the search is performed within a small range near the initial disparity estimates, rather than a full range search). Within the narrowed disparity search range, the matching cost for each pixel at each disparity value is recalculated. The recalculated matching costs are then dynamically aggregated to obtain the aggregated cost at the finer scale level. Disparity selection is used to select the disparity value with the minimum aggregated cost for each pixel at the finer scale level, generating the disparity map for the finer scale level.

[0034] The disparity map propagation, disparity search range shrinking, matching cost recalculation, dynamic programming aggregation, and disparity selection are performed step by step until the most detailed disparity map is generated.

[0035] During the dynamic programming aggregation process at each scale level, the disparity smoothing penalty parameter (i.e., the smoothing penalty term in the SGM algorithm) is adaptively adjusted based on the local texture richness to control disparity continuity. Specifically, the disparity smoothing penalty parameter is reduced in texture-rich regions to allow disparity variation and preserve details; the disparity smoothing penalty parameter is increased in texture-deficient regions to maintain disparity smoothness; and the disparity smoothing penalty parameter is reduced in edge regions to maintain edge sharpness.

[0036] Left-right consistency detection is performed on the disparity map at the finest scale level. Inconsistent pixels are marked as unreliable regions, and reliable disparity values ​​from the neighborhood are used to fill and correct these unreliable regions. The principle of left-right consistency detection is to compare the consistency between the left and right disparity maps. For inconsistent pixels, they are deleted and marked as unreliable regions, and then filled and corrected using reliable disparity values ​​from the neighborhood.

[0037] Subpixel interpolation is performed on the disparity map after fill correction. Subpixel interpolation can be performed using parabolic fitting, fitting a 3-pixel window parabola near the disparity peak; or it can use quadratic interpolation, based on a quadratic function.

[0038] A deep learning network is used to enhance the interpolated disparity map, resulting in the final disparity map. The deep learning network is a disparity optimization sub-network, employing an encoder-decoder architecture. Its input is the interpolated disparity map, and its output is the enhanced final disparity map. This enhancement process effectively corrects residual mismatched pixels in the interpolated disparity map, smooths noisy regions, and maintains the sharpness of object edges.

[0039] S140 generates a virtual laser point cloud of the environment in which the embodied robot is located, based on the parallax map and the focal length, baseline distance, and principal point coordinates in the calibration parameters of the binocular camera.

[0040] In some embodiments, for each pixel in the disparity map, the depth value Z of the pixel is calculated based on the pixel's disparity value d, the horizontal focal length fx in the focal length (fx, fy), and the baseline distance B. The depth value calculation formula can be expressed as: Based on the pixel's depth value Z, focal length (fx, fy), and principal point coordinates (cx, cy), the pixel's image coordinates (u, v) are transformed from the image coordinate system to the camera coordinate system, yielding (X, Y). Combined with the depth value Z, the three-dimensional coordinates (X, Y, Z) of the spatial point are obtained. This transformation process can be represented as: Based on the three-dimensional coordinates (X, Y, Z) of each spatial point, a virtual laser point cloud of the environment in which the embodied robot is located is generated. The data structure of the virtual laser point cloud is a point cloud set containing N points, each point containing three-dimensional coordinates (X, Y, Z).

[0041] S150 optimizes the quality of the virtual laser point cloud in the environment of the embodied robot.

[0042] In some embodiments, the virtual laser point cloud is denoised. Specifically, for each point in the virtual laser point cloud, its neighboring points within its neighborhood radius are searched, the distances between the point and each neighboring point are calculated and averaged as the average neighborhood distance of the point. The average neighborhood distances of all points are statistically analyzed, and their mean μ and standard deviation σ are calculated. A threshold for the average neighborhood distance is set based on the mean μ and standard deviation σ. α is a preset standard deviation multiple (e.g., 1, 2, or 3). Points whose average neighborhood distance is greater than a threshold are removed as first outliers. For each point in the virtual laser point cloud after removing the first outliers, its neighboring points within a specified radius are searched, the number of neighboring points is counted, and points whose number of neighboring points is less than a preset threshold are removed as second outliers.

[0043] The denoised virtual laser point cloud is downsampled using a voxel grid to obtain a reduced-density virtual laser point cloud. Specifically, the three-dimensional space containing the denoised virtual laser point cloud is divided into voxel grids of preset side lengths. For each non-empty voxel grid, the centroid position of all points within it is calculated, and the points at the centroid positions are used to replace all points within the non-empty voxel grids, thus obtaining the reduced-density virtual laser point cloud.

[0044] Sparse regions in a de-density virtual laser point cloud are detected, and semantic information corresponding to these sparse regions is obtained from preprocessed left and right view environmental images. The detection of sparse regions can be based on point cloud density analysis, identifying regions with point cloud density below a preset threshold as sparse regions.

[0045] The target category (e.g., wall, floor, equipment, obstacle, etc.) of the sparse region is determined based on semantic information, and the geometric distribution features of existing points at the edge of the sparse region are extracted from the reduced-density virtual laser point cloud.

[0046] Based on the geometric distribution characteristics of existing points at the edge of the sparse region and the geometric completion prior corresponding to the target category, points are added to the sparse region. Specifically, based on the geometric distribution characteristics of existing points at the edge of the sparse region, the normal, curvature, and local density of existing points at the edge of the sparse region are extracted, and an envelope surface is constructed by contracting along the normal to limit the space of points that can be added to the sparse region. Here, the envelope surface refers to the surface formed after contracting inward a certain distance in the normal direction of the edge points, which is used to define the spatial range of the point addition operation and ensure that the point addition result maintains geometric continuity with the surrounding point cloud. The geometric completion prior corresponding to the target category is applied to the point-addressable space. The implicit shape field in the geometric completion prior is used to calculate the shape conformity of each position in the point-addressable space, and the curvature of existing points at the edge of the sparse region is used as the curvature constraint boundary condition of the implicit shape field. The shape conformity calculation result and the curvature constraint boundary condition are solved simultaneously to obtain the desired point density distribution and desired normal field in the point-addressable space. The geometric completion prior is a pre-established shape model for different types of objects (e.g., planes, spheres, cylinders, etc.), describing the typical geometric shape of these objects in space. Based on the desired point density distribution and desired normal field, spatial advancement is performed from the edge of the sparse region to the interior of the sparse region within the point completion space, generating multiple candidate points. The spacing of the candidate points is adjusted according to the local density of the existing points at the edge of the sparse region. Conflict detection is performed between the multiple candidate points and the existing points at the edge of the sparse region. Candidate points with a spacing smaller than the local density threshold in the geometric distribution features are eliminated. The positions of the retained points are fine-tuned according to the normal of the existing points at the edge of the sparse region to optimize geometric continuity.

[0047] The color information of each pixel in the reduced-density virtual laser point cloud after point filling is extracted from the preprocessed left and right view environmental images, and each point is assigned a color attribute to obtain a complete virtual laser point cloud. The color information is extracted by projecting each point in the point cloud onto the preprocessed left and right view environmental images to obtain the RGB color value of the corresponding pixel.

[0048] If a historical optimized virtual laser point cloud exists, the complete virtual laser point cloud is registered and fused with the historical optimized virtual laser point cloud to obtain the current optimized virtual laser point cloud. If no historical optimized virtual laser point cloud exists, the complete virtual laser point cloud is used as the current optimized virtual laser point cloud. The registration and fusion can be implemented using an iterative nearest-point algorithm or a normal distribution transformation algorithm. Specifically, a rigid body transformation registration is performed on the complete virtual laser point cloud of the current frame and the historical optimized virtual laser point cloud. Then, point cloud fusion is performed on overlapping areas, and non-overlapping areas are directly stitched together.

[0049] S160 performs semantic annotation on the optimized virtual laser point cloud, assigning semantic category labels to each point.

[0050] In some embodiments, a two-dimensional semantic segmentation network is used to perform pixel-level semantic segmentation on the preprocessed left and right view environmental images to obtain two-dimensional semantic category labels for each pixel. The two-dimensional semantic segmentation network can adopt mature semantic segmentation architectures such as DeepLab series, UNet or SegFormer, and be pre-trained on large semantic segmentation datasets (e.g., Cityscapes, ADE20K, etc.) to have the ability to recognize common environmental elements at the pixel level.

[0051] A 3D semantic segmentation network is used to perform point-level semantic segmentation on the optimized virtual laser point cloud, obtaining 3D semantic category labels for each point. The 3D semantic segmentation network can employ point cloud semantic segmentation architectures such as PointNet++, RandLA-Net, or KPConv, and is pre-trained on large point cloud semantic segmentation datasets (e.g., SemanticKITTI, S3DIS), possessing the ability to perform point-level semantic recognition on point cloud data.

[0052] The two-dimensional semantic category labels of the corresponding pixels at each point are weighted and fused with the three-dimensional semantic category labels of each point to obtain the semantic category label of each point. The weights of the two-dimensional and three-dimensional semantic category labels can be flexibly adjusted according to the actual situation, and there is no restriction here.

[0053] Inter-frame consistency optimization is performed on the semantic category labels of each point to eliminate semantic category label jumps between adjacent frames. The method of inter-frame consistency optimization is as follows: for each point in the current frame, the semantic category label of the corresponding position in the previous frame is found. If the semantic category label of the point in the current frame is inconsistent with the semantic category label of the previous frame and the confidence level is low, the semantic category label of the previous frame is used for correction; otherwise, the semantic category label of the current frame is used directly.

[0054] S170, convert the virtual laser point cloud with semantic category labels into a virtual laser point cloud adapted to the target task to be performed by the embodied robot, so that the embodied robot can call it when performing the target task.

[0055] In some embodiments, different target tasks for the embodied robot require different coordinates and data formats for the virtual laser point cloud. For example, for navigation tasks, it may be necessary to convert the virtual laser point cloud to a world coordinate system and use a raster map format; for grasping tasks, it may be necessary to convert the virtual laser point cloud to a robotic arm base coordinate system and use a point cloud file in a specific format. Therefore, it is necessary to perform corresponding conversions on the virtual laser point cloud with semantic category labels according to the specific requirements of the target task. Specifically, based on the coordinate and data format requirements of the target task to be performed by the embodied robot, the virtual laser point cloud with semantic category labels undergoes coordinate and data format conversion to generate a virtual laser point cloud adapted to the target task, which is then stored locally or in the cloud for the embodied robot to access when performing the target task.

[0056] In summary, the embodiments disclosed herein achieve at least the following technical effects compared to the prior art: (1) This embodiment of the present disclosure utilizes a low-cost, small-size, and low-power dual-axis camera to replace traditional LiDAR and depth cameras as the 3D perception front end of the embodied robot, which significantly reduces the system hardware cost, power consumption, and installation space requirements, and effectively promotes the economical deployment of embodied robots in home, factory, warehouse, and lightweight service scenarios. Compared with existing virtual laser point cloud generation schemes based on ordinary cameras, this embodiment of the present disclosure greatly reduces the computational complexity through a coarse-to-fine stereo matching strategy and dynamic programming aggregation, and can generate dense and accurate virtual laser point clouds in real time, meeting the stringent requirements of navigation, grasping, obstacle avoidance, and other tasks for real-time 3D perception.

[0057] (2) In the image acquisition stage, a hardware trigger signal is generated by a synchronization signal generator to control the left and right cameras of the dual-axis camera to shoot synchronously, ensuring the temporal consistency of the environmental images from the left and right perspectives, eliminating the parallax calculation error caused by the camera exposure time difference, and improving the accuracy of virtual laser point cloud generation in dynamic environments. In the preprocessing stage, through distortion correction, stereo correction and image enhancement, the radial distortion and tangential distortion of the lens are effectively eliminated, and the epipolar lines of the environmental images from the left and right perspectives are strictly aligned. At the same time, noise is suppressed, contrast is improved and illumination changes are compensated, providing high-quality, illumination-robust epipolar constraint input for subsequent stereo matching. This allows the scheme to work stably in adverse environments such as low light and non-uniform illumination, overcoming the problem that traditional depth cameras are easily affected by ambient light interference.

[0058] (3) In the stereo matching process, multiple features are fused for mixed cost calculation, which improves the matching discrimination of weak texture and repetitive texture regions; an image pyramid is constructed and a coarse-to-fine disparity propagation and search range shrinkage mechanism is adopted, which not only ensures the coverage of a large disparity range, but also achieves high-precision matching at a fine scale; in the dynamic programming aggregation process, the disparity smoothing penalty parameter is adaptively adjusted according to the local texture richness, which preserves the disparity continuity in weak texture regions while retaining the disparity discontinuity at the edges and details; combined with left-right consistency detection and unreliable region correction, sub-pixel interpolation and deep learning network enhancement processing, the final generated disparity map has high precision, high integrity and clear object boundaries, laying the foundation for the realism and accuracy of virtual laser point clouds.

[0059] (4) Virtual laser point clouds are generated based on disparity maps and dual-axis camera calibration parameters, eliminating the need for additional depth sensors and enabling low-cost acquisition of dense 3D information. In the quality optimization of virtual laser point clouds, data redundancy is reduced while preserving the geometric structure through denoising and voxel mesh downsampling. For detected sparse regions, the target category is determined by combining 2D semantic information. Based on the geometric completion prior corresponding to the target category and the normal, curvature, and local density of existing points on the edge, candidate points that conform to the continuity of the object shape are generated in the limited space of points that can be filled. After conflict detection and normal fine-tuning, geometric continuity is optimized. After filling the points, color attributes are assigned and registered and fused with historically optimized virtual laser point clouds, effectively solving the problem of sparsity and missing virtual laser point clouds caused by factors such as occlusion and reflection, and significantly improving the completeness, uniformity, and overall consistency of virtual laser point clouds.

[0060] (5) In the semantic annotation stage, the semantic segmentation results of two-dimensional image and three-dimensional virtual laser point cloud are integrated and weighted. Then, label jumps are eliminated through cross-frame consistency optimization. The resulting virtual laser point cloud has both the fine-grained category discrimination capability of image texture and the spatial positioning advantage of three-dimensional geometric structure. The semantic category label is accurate and stable. Finally, according to the coordinate requirements and data format requirements of the target task, the virtual laser point cloud with semantic category label is subjected to coordinate transformation and format transformation to generate an adapted virtual laser point cloud that can be directly called by downstream tasks such as navigation, grasping, and obstacle avoidance. This eliminates the need for repeated parsing and transformation, greatly improving the flexibility and efficiency of the embodied robot's task execution.

[0061] To facilitate further understanding, the method 100 provided in this disclosure will be described in detail below with reference to a specific embodiment, as shown in the following figure: Scenario Setting: A robot equipped with binocular cameras is deployed in a factory environment to replace traditional LiDAR for low-cost 3D environmental perception. The robot needs to perform daily inspections within the factory workshop. The following method 100 is used to generate and optimize virtual LiDAR point clouds.

[0062] (1) Binocular camera calibration Before deploying the embodied robot, its binocular cameras were precisely calibrated offline using a checkerboard calibration board. The left and right cameras were industrial-grade global shutter cameras with a resolution of 1280×720, a frame rate of 60fps, and a baseline distance of 18cm. During the calibration process, 20 sets of checkerboard images were captured in different poses to obtain the intrinsic parameters: focal length (fx, fy), where fx=fy=850 pixels, principal point coordinates (cx, cy) = (640, 360), distortion coefficients (k1, k2, k3, p1, p2) = (-0.28, 0.08, -0.001, 0.0005, -0.0003); extrinsic parameters: rotation matrix R is approximately an identity matrix, translation vector T = [-0.18, 0.001, 0.002], and baseline distance B = 0.18m. The calibration parameters were stored in the embodied robot's local storage for subsequent steps.

[0063] (2) Synchronously acquire environmental images from left and right perspectives The embodied robot moves to the densely piped section of Area A in the workshop. The synchronization signal generator produces a hardware trigger signal at a frequency of 60Hz, which simultaneously reaches the left and right cameras via the synchronization signal line. The left and right cameras capture images from the same trigger edge, synchronously acquiring left and right view environmental images that include steam pipes, pressure gauges, valves, and metal supports, ensuring no time deviation in the motion scene.

[0064] (3) Image preprocessing Distortion correction: Radial and tangential distortion correction is performed on the environmental images from the left and right perspectives using intrinsic distortion coefficients (-0.28, 0.08, -0.001, 0.0005, -0.0003) to eliminate barrel-shaped bends at the straight edges of pipes.

[0065] Stereo correction: Based on the external parameter rotation matrix R and translation vector T, the Bouguet algorithm is used to perform stereo correction on the distortion-corrected left and right view environmental images, so that the projection points of the same pipe flange in the left and right view environmental images strictly fall on the same horizontal polar line.

[0066] Image enhancement: There are local highlights and shadows in the overhead lighting of the workshop. First, bilateral filtering is used to remove noise and preserve the edges. Then, the CLAHE algorithm is used to enhance the local contrast, making the scale of the instrument in the dark area clearly visible. Finally, homomorphic filtering is used to compensate for uneven lighting and highlight the surface texture of the pipe.

[0067] (4) Stereo matching generates disparity map Hybrid cost calculation: The disparity search range is set to 0-128 pixels. For the preprocessed left and right images, three features are fused: absolute grayscale difference, gradient magnitude difference, and Census transform Hamming distance. They are weighted and fused with weights of 0.3, 0.3, and 0.4 respectively, and the matching cost of each pixel under the disparity range of 0-128 is calculated.

[0068] Image pyramid construction: A three-layer image pyramid is constructed using Gaussian filtering and downsampling, with the scale ranging from coarse to fine.

[0069] Parallax optimization from coarse to fine: At the coarsest scale (1 / 4 resolution), the SGM algorithm is used to aggregate costs along 8 directions to generate a coarse disparity map.

[0070] The coarse disparity map is propagated to the intermediate scale, and the search range is narrowed to ±8 pixels centered on the coarse disparity estimate. The matching cost is recalculated and aggregated to generate an intermediate-scale disparity map.

[0071] Similarly, the intermediate disparity map is propagated to the finest scale (original resolution), further narrowing the search range to ±4 pixels to generate a fine disparity map.

[0072] Adaptive smoothing penalty: In areas with weak texture, such as pipe walls, the smoothing penalty parameters P1 and P2 are automatically increased to maintain parallax smoothness and continuity; at pipe edges and valve outlines, the penalty parameters are reduced to preserve clear boundaries.

[0073] Post-processing of the disparity map: Left-right consistency detection is performed on the refined disparity map to remove occluded and mismatched pixels, and the median disparity value from the neighborhood is used for filling. Then, a parabolic fitting method is used to perform sub-pixel interpolation near the disparity peak to obtain sub-pixel precision disparity. Finally, a pre-trained lightweight disparity optimization sub-network (encoder-decoder structure) is used for further enhancement to eliminate residual noise and sharpen edges, outputting the final disparity map.

[0074] (5) Generate virtual laser point cloud For each valid pixel in the final disparity map, the depth Z is calculated using the depth calculation formula based on the pixel's disparity value d, horizontal focal length fx, and baseline distance B. For example, if the disparity d = 30 pixels on the surface of a pipe, then the depth value... Based on the depth value Z, focal length (fx, fy), and principal point coordinates (cx, cy), a transformation formula is used to convert the pixel's image coordinates (u, v) from the image coordinate system to the camera coordinate system, obtaining (X, Y). Combined with the depth value Z, the three-dimensional coordinates (X, Y, Z) of the spatial point are obtained. All effective pixels in the image are converted into spatial points (X, Y, Z), generating a dense virtual laser point cloud for the current frame, containing approximately 300,000 points, completely covering the pipes, valves, instruments, and ground area.

[0075] (6) Optimization of virtual laser point cloud quality Noise reduction: First, calculate the average neighborhood distance of each point (neighborhood radius r=5cm), calculate the global mean μ and standard deviation σ, set the threshold μ+2σ, and remove the first type of outliers that are obviously outliers; then search for neighboring points within a radius r=3cm for the remaining points, and remove isolated points with fewer than 5 neighboring points as the second type of outliers.

[0076] Voxel mesh downsampling: A 2cm voxel mesh is used to downsample the denoised point cloud. Each non-empty voxel is represented by the centroid point, which homogenizes the point cloud density and reduces the number of points to about 80,000, thus preserving the geometric structure while reducing the amount of data.

[0077] Sparse Region Point Completion: During the inspection, a narrow, sparse region behind the steam pipe, formed by obstruction, was detected. Semantic information of this region was extracted from the preprocessed left-view environmental image, classifying it as a "pipeline." The normal, curvature, and local density of existing points at the edge of the sparse region were extracted, and an envelope surface was constructed by contracting inward along the normal to define the point completion space. A prior geometric completion method for cylinders was invoked, using edge curvature as a boundary constraint, to simultaneously solve for the desired point density distribution and normal field. Candidate points were generated by advancing from the edge inward, with adaptive matching of spacing and local density. Conflict detection eliminated redundant candidate points, and the positions of retained points were fine-tuned based on the edge normal to ensure the smooth continuity of the point cloud behind the completed pipe with its surroundings.

[0078] Color assignment: Project each point of the completed point cloud onto the stereo-corrected left-view environment image, obtain the RGB values ​​of the corresponding pixels, assign color attributes to each point, and obtain a complete color virtual laser point cloud.

[0079] Inter-frame registration and fusion: This frame is the Kth frame of continuous inspection, and there is already a historical optimized point cloud in the (K-1)th frame. The ICP algorithm is used to register the current complete point cloud to the coordinate system of the historical point cloud, merge overlapping areas, and directly stitch non-overlapping areas together to update the current optimized virtual laser point cloud, gradually building a complete 3D map of the workshop.

[0080] (7) Semantic annotation Two-dimensional semantic segmentation: The SegFormer network is used to perform pixel-level semantic segmentation on the left-view environmental image to obtain a two-dimensional label for each pixel. Categories include pipes, valves, pressure gauges, ground, walls, obstacles, etc.

[0081] 3D semantic segmentation: RandLA-Net is used to perform point-level semantic segmentation on the current optimized virtual laser point cloud, and the 3D category label of each point is output.

[0082] Fusion and Optimization: The 2D labels are back-projected onto the point cloud and fused with the 3D labels using a weighted ratio of 0.4:0.6 to obtain the semantic labels for each point. Then, corresponding points are found through projection from previous and next frames. Labels with low confidence in the current frame and inconsistent with the previous frame are corrected to stable labels from the previous frame, eliminating semantic jumps. Finally, the steam pipe area is stably labeled as "pipeline," the pressure gauge area as "instrument," and the ground as "passable area."

[0083] (8) Format conversion for downstream tasks to use Currently, the embodied robot is performing a "leak detection" task during inspections. The task requires converting virtual laser point clouds to the robot's body coordinate system and publishing them in real-time using the ROS point cloud message format for use by downstream leak detection modules. Specifically, the virtual laser point cloud with semantic tags is converted to the robot's base coordinate system according to a calibrated extrinsic parameter matrix, encapsulated in the sensor_msgs / PointCloud2 format, and published to the / leak_detection_pointcloud topic. The message carries the 3D coordinates, RGB color values, and semantic tag fields for each point. After subscribing to this topic, the downstream leak detection module first quickly extracts point clouds of key components such as "pipes," "valves," and "flange joints" based on the semantic tags. Then, combining the point cloud's color attributes (such as localized color lightening caused by steam escaping and bright areas of droplet reflection) and local geometric abrupt changes (such as small protrusions and discontinuous clusters), it uses rules or lightweight networks to determine whether there are any leaks and locates the 3D spatial position of the leak point. Meanwhile, the navigation module can still subscribe to the same topic or global map point cloud, and use "ground" semantic tags and obstacle point clouds to achieve accurate positioning and dynamic obstacle avoidance, ensuring the safety of the inspection process.

[0084] It should be noted that, for the sake of simplicity, the foregoing method embodiments are all described as a series of actions. However, those skilled in the art should understand that this disclosure is not limited to the described order of actions, because according to this disclosure, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are all optional embodiments, and the actions and modules involved are not necessarily essential to this disclosure.

[0085] The above is an introduction to the method embodiments. The following describes the solution described in this disclosure further through device embodiments.

[0086] Figure 2 A structural diagram of a virtual laser point cloud generation and optimization device based on a binocular camera provided in this disclosure is shown below. Figure 2 As shown, the device 200 may include: The acquisition module 210 is used to acquire images of the environment in which the embodied robot is located through a binocular camera deployed on the embodied robot, and obtain environmental images from the left and right perspectives.

[0087] The preprocessing module 220 is used to preprocess the environmental images from the left and right perspectives.

[0088] The matching module 230 is used to perform stereo matching on the preprocessed left and right view environmental images to generate a disparity map.

[0089] The generation module 240 is used to generate a virtual laser point cloud of the environment in which the embodied robot is located, based on the parallax map and the focal length, baseline distance, and principal point coordinates in the calibration parameters of the binocular camera.

[0090] The optimization module 250 is used to optimize the quality of the virtual laser point cloud in the environment of the embodied robot.

[0091] The annotation module 260 is used to perform semantic annotation on the optimized virtual laser point cloud, assigning semantic category labels to each point.

[0092] The conversion module 270 is used to convert virtual laser point clouds with semantic category labels into virtual laser point clouds adapted to the target task to be performed by the embodied robot, so that the embodied robot can call them when performing the target task.

[0093] Understandable, Figure 2 Each module / unit in the illustrated device 200 has the ability to implement Figure 1 The functions of each step in method 100 shown, and their corresponding technical effects, will not be elaborated here for the sake of brevity.

[0094] Figure 3 This is a structural diagram of an electronic device provided for embodiments of the present disclosure. The electronic device 300 is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device 300 may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0095] like Figure 3As shown, the electronic device 300 may include a computing unit 301, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 302 or a computer program loaded from a storage unit 308 into a random access memory (RAM) 303. The RAM 303 may also store various programs and data required for the operation of the electronic device 300. The computing unit 301, ROM 302, and RAM 303 are interconnected via a bus 304. An input / output (I / O) interface 305 is also connected to the bus 304.

[0096] Multiple components in electronic device 300 are connected to I / O interface 305, including: input unit 306, such as keyboard, mouse, etc.; output unit 307, such as various types of displays, speakers, etc.; storage unit 308, such as disk, optical disk, etc.; and communication unit 309, such as network card, modem, wireless transceiver, etc. Communication unit 309 allows electronic device 300 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0097] The computing unit 301 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 301 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 301 performs the various methods and processes described above, such as method 100. For example, in some embodiments, method 100 may be implemented as a computer program product, including a computer program tangibly contained in a computer-readable medium, such as storage unit 308. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 300 via ROM 302 and / or communication unit 309. When the computer program is loaded into RAM 303 and executed by the computing unit 301, one or more steps of method 100 described above may be performed. Alternatively, in other embodiments, the computing unit 301 may be configured to perform method 100 by any other suitable means (e.g., by means of firmware).

[0098] The various embodiments described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), system-on-a-chip (SoCs), payload programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0099] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0100] In the context of this disclosure, a computer-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. A computer-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of computer-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0101] It should be noted that this disclosure also provides a non-transitory computer-readable storage medium storing computer instructions. These computer instructions are used to cause a computer to execute method 100 and achieve the corresponding technical effects achieved by executing the method in the embodiments of this disclosure; for the sake of brevity, they will not be elaborated further here.

[0102] In addition, this disclosure also provides a computer program product including a computer program that implements method 100 when executed by a processor.

[0103] To provide interaction with a user, the embodiments described above can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0104] The embodiments described above can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with the implementations of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0105] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0106] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired results of this disclosure are achieved, and this is not limited herein.

[0107] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A method for generating and optimizing virtual laser point clouds based on a binocular camera, characterized in that, The method includes: Images are captured using a binocular camera to obtain environmental images from both left and right perspectives; The left and right view environmental images are preprocessed; Stereo matching is performed on the preprocessed left and right view environmental images to generate a disparity map; Based on the disparity map and the focal length, baseline distance, and principal point coordinates in the calibration parameters of the binocular camera, a virtual laser point cloud is generated; Optimize the quality of virtual laser point clouds; Semantic annotation is performed on the optimized virtual laser point cloud, and each point is assigned a semantic category label. The virtual laser point cloud with semantic category labels is converted into a virtual laser point cloud adapted for the embodied robot to perform the target task.

2. The method according to claim 1, characterized in that, The process of acquiring images of the environment in which the android is located through a binocular camera deployed on the android to obtain left and right view environmental images includes: The hardware trigger signal generated by the synchronization signal generator is transmitted to the left and right cameras of the binocular camera via the synchronization signal line. In response to the same hardware trigger signal, the left and right cameras are controlled to simultaneously capture images of the environment in which the robot is located, thereby obtaining environmental images from both left and right perspectives.

3. The method according to claim 1, characterized in that, The preprocessing of the left and right view environmental images includes: The distortion correction is performed on the left and right view environmental images according to the distortion coefficient in the calibration parameters of the binocular camera to eliminate lens radial distortion and lens tangential distortion. Based on the rotation matrix and translation vector in the calibration parameters of the binocular camera, stereo correction is performed on the distortion-corrected left and right view environmental images so that the projection positions of the same spatial point in the left and right view environmental images are located on the same horizontal epipolar line. Image enhancement is performed on the left and right view environmental images after stereo correction; the image enhancement includes noise reduction, contrast enhancement and illumination compensation.

4. The method according to claim 1, characterized in that, The step of performing stereo matching on the preprocessed left and right view environment images to generate a disparity map includes: For each pixel in the preprocessed left and right view environmental images, multiple features are fused within the set disparity search range to calculate the mixed cost, and the matching cost of each pixel under each disparity value is obtained. Construct an image pyramid of the preprocessed left and right view environmental images, the image pyramid containing multiple scale levels; At the coarsest level of the image pyramid, the matching cost of each pixel at each disparity value is dynamically planned and aggregated to obtain the aggregation cost of the coarsest level. The disparity value with the smallest aggregation cost is selected for each pixel at the coarsest level through disparity selection to generate the disparity map of the coarsest level. The disparity map of the previous scale level is propagated to the adjacent finer scale level as the initial disparity estimate for each pixel in the finer scale level. In the finer scale level, the disparity search range is narrowed based on the initial disparity estimate of each pixel, and the matching cost of each pixel at each disparity value is recalculated within the narrowed disparity search range. The recalculated matching cost is dynamically aggregated to obtain the aggregate cost of the finer scale level. The disparity value with the minimum aggregate cost is selected for each pixel in the finer scale level through disparity selection to generate the disparity map of the finer scale level. The disparity map propagation, disparity search range shrinking, matching cost recalculation, dynamic programming aggregation, and disparity selection are performed step by step until the most detailed disparity map is generated. During the dynamic programming aggregation process at each scale level, the disparity smoothing penalty parameter is adaptively adjusted according to the local texture richness to control disparity continuity. Perform left-right consistency detection on the disparity map at the finest scale level, mark inconsistent pixels as unreliable regions, and fill and correct the unreliable regions using the disparity values ​​in the neighborhood; Subpixel interpolation is performed on the disparity map after fill correction. A deep learning network is used to enhance the interpolated disparity map to obtain the final disparity map.

5. The method according to claim 1, characterized in that, The step of generating a virtual laser point cloud of the environment in which the embodied robot is located, based on the disparity map and the focal length, baseline distance, and principal point coordinates in the calibration parameters of the binocular camera, includes: For each pixel in the disparity map, the depth value of the pixel is calculated based on the disparity value of the pixel, the horizontal focal length in the focal length, and the baseline distance; Based on the depth value of the pixel, the focal length, and the principal point coordinates, the image coordinates of the pixel are transformed from the image coordinate system to the camera coordinate system, and combined with the depth value, the three-dimensional coordinates of the spatial point are obtained. Based on the three-dimensional coordinates of each spatial point, a virtual laser point cloud of the environment in which the embodied robot is located is generated.

6. The method according to claim 1, characterized in that, The quality optimization of the virtual laser point cloud of the environment in which the android is located includes: The virtual laser point cloud is denoised, and the denoised virtual laser point cloud is downsampled using a voxel grid to obtain a reduced-density virtual laser point cloud. The sparse regions in the reduced-density virtual laser point cloud are detected, and the semantic information corresponding to the sparse regions is obtained from the preprocessed left and right view environmental images. Based on the semantic information, the target category to which the sparse region belongs is determined, and the geometric distribution features of existing points at the edge of the sparse region are extracted from the reduced-density virtual laser point cloud. Based on the geometric distribution characteristics of existing points at the edge of the sparse region and the geometric completion prior corresponding to the target category, points are added to the sparse region. The color information of each pixel in the reduced-density virtual laser point cloud after point filling is extracted from the preprocessed left and right view environmental images, and each point is assigned a color attribute to obtain a complete virtual laser point cloud. If a historical optimized virtual laser point cloud exists, the complete virtual laser point cloud is registered and fused with the historical optimized virtual laser point cloud to obtain the current optimized virtual laser point cloud; if no historical optimized virtual laser point cloud exists, the complete virtual laser point cloud is used as the current optimized virtual laser point cloud.

7. The method according to claim 6, characterized in that, The step of filling in points in the sparse region based on the geometric distribution characteristics of existing points at the edge of the sparse region and the geometric completion prior corresponding to the target category includes: Based on the geometric distribution characteristics of the existing points at the edge of the sparse region, the normal, curvature and local density of the existing points at the edge of the sparse region are extracted, and an envelope surface is constructed by contracting along the normal to limit the space of points that can be filled in the sparse region. The geometric completion prior corresponding to the target category is applied to the space of points that can be completed. The implicit shape field in the geometric completion prior is used to calculate the shape conformity of each position in the space of points that can be completed. The curvature of the existing points at the edge of the sparse region is used as the curvature constraint boundary condition of the implicit shape field. The shape conformity calculation result and the curvature constraint boundary condition are solved simultaneously to obtain the expected point density distribution and expected normal field in the space of points that can be completed. Based on the desired point density distribution and the desired normal field, spatial advancement is performed from the edge of the sparse region to the interior of the sparse region within the fillable point space to generate multiple candidate points, and the spacing between the candidate points is adjusted according to the local density of the existing points at the edge of the sparse region. Conflict detection is performed between multiple candidate points and existing points at the edge of the sparse region. Candidate points with a spacing smaller than the local density threshold in the geometric distribution features are removed. The positions of the retained points are fine-tuned according to the normals of existing points at the edge of the sparse region to optimize geometric continuity.

8. The method according to claim 1, characterized in that, The process of semantically annotating the optimized virtual laser point cloud, assigning semantic category labels to each point, includes: A two-dimensional semantic segmentation network is used to perform pixel-level semantic segmentation on the preprocessed left and right view environment images to obtain two-dimensional semantic category labels for each pixel; A 3D semantic segmentation network is used to perform point-level semantic segmentation on the quality-optimized virtual laser point cloud to obtain the 3D semantic category label of each point. The two-dimensional semantic category labels of the corresponding pixels of each point are weighted and fused with the three-dimensional semantic category labels of each point to obtain the semantic category labels of each point; The semantic category labels of each point are optimized for cross-frame consistency to eliminate semantic category label jumps between adjacent frames.

9. The method according to claim 1, characterized in that, The process of converting virtual laser point clouds with semantic category labels into virtual laser point clouds adapted to the target task to be performed by the embodied robot includes: Based on the coordinate and data format requirements of the target task to be performed by the embodied robot, the virtual laser point cloud with semantic category labels is transformed in coordinate and data format to generate a virtual laser point cloud adapted to the target task.

10. A virtual laser point cloud generation and optimization device based on a binocular camera, characterized in that, The device includes: The acquisition module is used to acquire images of the environment in which the android is located through a binocular camera deployed on the android, and obtain environmental images from the left and right perspectives. The preprocessing module is used to preprocess the left and right view environmental images; The matching module is used to perform stereo matching on the preprocessed left and right view environmental images to generate a disparity map; The generation module is used to generate a virtual laser point cloud of the environment in which the embodied robot is located, based on the disparity map and the focal length, baseline distance, and principal point coordinates in the calibration parameters of the binocular camera. The optimization module is used to optimize the quality of the virtual laser point cloud in the environment where the embodied robot is located; The annotation module is used to perform semantic annotation on the optimized virtual laser point cloud, assigning semantic category labels to each point. The conversion module is used to convert virtual laser point clouds with semantic category labels into virtual laser point clouds adapted to the target task to be performed by the embodied robot, so that the embodied robot can call them when performing the target task.

Images