Robot control method, computing device and readable storage medium

Abstract

The invention discloses a robot control method, computing equipment and a readable storage medium. The method comprises the following steps: acquiring an original polarization image and an original depth map of a target object; extracting polarization physical parameters including physical attributes of the surface of the object from the original polarization image, and deriving geometric prior information based on the polarization physical parameters; fusing the polarization physical parameters, the geometric prior information and the original depth map to generate a multi-modal condition tensor for guiding depth information recovery; performing depth completion processing on the original depth map based on the multi-modal condition tensor to generate a completed depth map; and according to the complemented depth map, carrying out grabbing pose planning on the target object, and outputting a robot grabbing instruction. In this way, the transparent and reflective objects can be accurately and stably grabbed, and the success rate of the grabbing task is increased.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a robot control method, computing device, and readable storage medium. Background Technology

[0002] In the fields of industrial automation and service robots, reliably grasping transparent objects (such as plastic pipes) and mirror-reflective objects (such as polished metal parts and automotive lights) has been a long-standing technical challenge. Due to the unique optical properties of these objects, vision systems based on traditional RGB (red, green, blue) or RGB-D (red, green, blue-depth) cameras may fail. For example, light emitted by the depth sensor may be transmitted through the surface of a transparent object or mirror-reflected on the surface of a reflective object, failing to return a valid depth signal. This results in large areas of voids or severe noise in the depth map, making it difficult to grasp transparent and reflective objects due to missing or distorted visual information. Summary of the Invention

[0003] The purpose of this application is to provide a robot control method, computing device, and readable storage medium that can accurately and stably grasp transparent and reflective objects, thereby improving the success rate of grasping tasks.

[0004] To achieve the above objectives: In a first aspect, embodiments of this application provide a robot control method, the method comprising: Obtain the original polarization image and original depth map of the target object; Polarization physical parameters containing the physical properties of the object surface are extracted from the original polarization image, and geometric prior information is derived based on the polarization physical parameters; The polarization physical parameters, the geometric prior information, and the original depth map are fused to generate a multimodal conditional tensor for guiding depth information recovery; The original depth map is depth-completed based on the multimodal conditional tensor to generate a completed depth map; Based on the completed depth map, the robot performs grasping pose planning for the target object and outputs grasping instructions.

[0005] In some embodiments, the polarization physical parameters include an intensity image, a degree of polarization map, and a polarization angle map; extracting the polarization physical parameters from the original polarization image includes: Calculate the Stokes vector corresponding to the original polarization image; Based on the Stokes vector, the intensity image, the degree of polarization map, and the angle of polarization map of the original polarization image are extracted.

[0006] In some embodiments, the geometric prior information includes at least one of the following: a surface normal map and an edge map; the derivation of the geometric prior information based on the polarization physics parameters includes at least one of the following: Based on the principle of polarization vision, the surface normal map is determined using the degree of polarization map and the angle of polarization map; Edge detection is performed on the intensity image and the polarization map respectively, and the edge detection results are fused to generate the edge map.

[0007] In some embodiments, the step of performing depth completion processing on the original depth map based on the multimodal conditional tensor to generate a completed depth map includes: Using the multimodal conditional tensor as a condition, the trained depth completion network is used to perform depth completion processing on the original depth map to generate the completed depth map.

[0008] In some embodiments, the depth completion network includes a conditional flow matching model; the step of using the multimodal conditional tensor as a condition to perform depth completion processing on the original depth map using the trained depth completion network to generate the completed depth map includes: Using the multimodal conditional tensor as a condition, the trained conditional flow matching model is used to iteratively update the random noise tensor N times through an ordinary differential equation numerical solver, so as to gradually remove noise and approximate the target depth distribution; the random noise tensor is sampled from a standard Gaussian distribution, and N is a preset positive integer; The output tensor obtained after N iterations is determined as the completed depth map.

[0009] In some embodiments, the step of planning the grasping pose of the target object based on the completed depth map and outputting robot grasping instructions includes: The completed depth map is back-projected into a 3D point cloud through a differentiable operation; Based on the three-dimensional point cloud, at least one candidate grasping pose for the target object is generated; A differentiable grasping evaluation model is used to score the quality of each candidate grasping pose, and the robot grasping command is generated and output based on the candidate grasping pose with the highest score.

[0010] In some embodiments, the step of using a differentiable grasping evaluation model to score the quality of each candidate grasping pose includes: Based on the candidate grasping pose, determine the differentiable spatial symbolic distance field of the robot gripper in three-dimensional space; Calculate the interaction information between the three-dimensional point cloud and the spatial symbol distance field; Based on the interaction information, the grasping quality score of the candidate grasping pose is calculated.

[0011] In some embodiments, the conditional flow matching model and the differentiable grasping evaluation model are obtained through end-to-end optimization training using a joint loss function that includes depth reconstruction loss and grasping task loss.

[0012] In a second aspect, embodiments of this application provide a computing device, including: a storage medium and a controller, wherein a computer program is stored on the storage medium, and the computer program, when executed by the controller, implements the robot control method as described in the first aspect.

[0013] Thirdly, embodiments of this application provide a readable storage medium storing a computer program, which, when executed by a processor, implements the robot control method as described in the first aspect.

[0014] The robot control method, computing device, and readable storage medium provided in this application include: acquiring an original polarization image and an original depth map of a target object; extracting polarization physical parameters containing physical properties of the object surface from the original polarization image, and deriving geometric prior information based on the polarization physical parameters; fusing the polarization physical parameters, geometric prior information, and the original depth map to generate a multimodal conditional tensor for guiding depth information recovery; performing depth completion processing on the original depth map based on the multimodal conditional tensor to generate a completed depth map; and performing grasping pose planning on the target object based on the completed depth map, and outputting robot grasping instructions. Thus, by acquiring the original polarization image of the target object, extracting polarization physical parameters from the original polarization image and deriving geometric prior information, a multimodal conditional tensor rich in physical meaning is generated and fused to provide physical constraints for depth completion. Then, the depth map of the target object is completed based on this multimodal conditional tensor, and the grasping pose of the target object is accurately planned according to the completed depth map. This effectively solves the problem that transparent and reflective objects are difficult to grasp due to the lack or distortion of visual information, and can achieve accurate and stable grasping of transparent and reflective objects, thus improving the success rate of grasping tasks. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the robot control method provided in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram illustrating the implementation architecture of the robot control method provided in an embodiment of the present invention.

[0017] Figure 3 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0018] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements.

[0019] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0020] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, can be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, or mean any one or any combination thereof. Therefore, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C". Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0021] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0022] It should be noted that step designations such as S101 and S102 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the protection scope of this application.

[0023] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0024] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0025] See Figure 1 This application provides a robot control method, which can be executed by a robot control device provided in this application. The device can be implemented in software and / or hardware, such as a processor or computer. In this embodiment, the executing entity is a computing device. The method provided in this embodiment includes: S101, acquire the original polarization image and original depth map of the target object.

[0026] The target object is the object the robot wants to grasp, such as a plastic pipe or a car headlight. The raw polarization image is a collection of images captured by a polarization camera, containing light intensity information in specific polarization directions (typically 0°, 45°, 90°, and 135°). Each pixel in the image contains intensity information in multiple directions, characterizing the change in polarization state after light interacts with the object's surface. The raw depth map is an image or matrix composed of depth data directly acquired by a depth sensor (such as a structured light camera, a time-of-flight camera, or a stereo vision system). The value of each pixel in this raw depth map typically represents the physical distance or parallax of the corresponding point in the scene from the sensor plane.

[0027] For example, a polarization camera (such as a camera equipped with a four-way polarization filter) mounted on the robot's wrist and a depth sensor (such as an RGB-D camera) mounted on the robot's head are aligned with a target object (such as a transparent plastic tube) in the working area, and its original polarization image (four intensity maps at different angles) and original depth map are acquired simultaneously.

[0028] S102, extract polarization physical parameters containing the physical properties of the object surface from the original polarization image, and derive geometric prior information based on the polarization physical parameters.

[0029] Among them, polarization physics parameters refer to parameters calculated from the original polarization image that directly reflect the physical properties (such as material and micro-geometry) of an object's surface. Polarization physics parameters can include intensity images, degree of polarization maps, and angle of polarization maps. The intensity image (RGB-I) is calculated using Stokes vectors and characterizes the total light intensity of the object's surface; it can be approximated as a regular grayscale or color image after removing polarization effects. The degree of polarization map (DoP) reflects the object's ability to polarize incident light. Highly reflective materials (such as metals and glass) have high DoP values ​​at specific observation angles, providing important clues for distinguishing materials and inferring surface orientation. The angle of polarization map (AoP) encodes the projection direction (azimuth angle) of the surface normal onto the camera's imaging plane, providing crucial information for reconstructing the surface's three-dimensional orientation.

[0030] In some embodiments, extracting polarization physical parameters from the original polarization image includes: Calculate the Stokes vector corresponding to the original polarization image; Based on the Stokes vector, extract the intensity image, degree of polarization map, and angle of polarization map of the original polarization image.

[0031] Specifically, firstly, the Stokes vector [S0, S1, S2] corresponding to the original polarization image is calculated according to the formulas S0 = I0 + I90, S1 = I0 - I90, and S2 = I45 - I135, where I0, I45, I90, and I135 are the intensity values ​​for the four polarization directions of 0°, 45°, 90°, and 135°, respectively. Next, S0 is treated as unpolarized intensity information and converted into a grayscale image or fused with the image after interpolation by a color filter array to generate a three-channel RGB intensity image. The degree of polarization is calculated according to the formula DoP = sqrt(S1^2 + S2^2) / S0, and the polarization angle is calculated according to the formula AoP = 0.5 * arctan2(S2, S1). Thus, calculating the polarization physical parameters based on the Stokes vector ensures that the maximum amount of physically meaningful surface information is extracted from the original sensor data, providing multi-dimensional and interpretable constraints for subsequent geometric prior derivation and depth completion.

[0032] Geometric prior information refers to estimates of the macroscopic or microscopic geometric structure of an object's surface, derived from polarization physics parameters through physical models or image processing techniques. Geometric prior information may include at least one of the following: a surface normal map and an edge map. The surface normal map is calculated based on the principle of polarization vision, utilizing the relationship between DoP and zenith angle, and the azimuth information provided by AoP, to determine the three-dimensional normal direction of each point on the object's surface. The edge map is an image obtained by performing edge detection (such as using Sobel or Canny operators) on the intensity image and the polarization degree map, and then fusing the results. Because the polarization degree map is sensitive to abrupt changes in intensity caused by refraction and reflection, obtaining the edge map based on the polarization degree map can effectively enhance the edge information of the outline and internal structure of transparent objects.

[0033] In some embodiments, the derivation of geometric prior information based on polarization physical parameters includes at least one of the following: Based on the principle of polarization vision, the surface normal map is determined using the degree of polarization map and the angle of polarization map. Edge detection is performed on the intensity image and polarization map respectively, and the edge detection results are fused to generate an edge map.

[0034] Specifically, using polarization degree maps and polarization angle maps, combined with the known camera-light source relative positions, a preliminary surface normal map is estimated through a physical inversion algorithm. Simultaneously, Sobel or Canny edge detection operators are applied to the intensity image and polarization degree map respectively to extract edges. The two edge maps are then logically ORed to fuse them into a more sensitive edge map that captures the object's contour and internal refractive edges. This is understandable, as the polarization degree map is extremely sensitive to intensity changes caused by refraction and reflection, effectively capturing characteristic edges inside transparent objects and on reflective objects. Fusion with the intensity image edges yields a more complete and accurate representation of the object's contour than a single modality, significantly improving the accuracy of object boundary localization in subsequent pose estimation. Furthermore, since the surface normal map directly provides 3D orientation information of the object's surface, and the edge map reinforces the object's contour and internal structural boundaries, these two types of geometric prior information provide strong geometric shape guidance for the depth completion network, especially in areas with missing data, effectively preventing the generation of unreasonable or blurry geometric shapes. The derivation of the surface normal map based on polarization physics parameters can specifically include: based on the principle of physical polarization vision, there is a functional relationship between the degree of polarization map and the zenith angle of the surface normal. Using this functional relationship, combined with the degree of polarization map and known refractive index parameters, the zenith angle of the normal at each point on the surface can be estimated. Then, combining the normal azimuth angle directly obtained from the polarization angle map, the surface normal vector in three-dimensional space is calculated through coordinate transformation, forming the surface normal map.

[0035] Thus, by deriving geometric prior information through polarization physics parameters, the geometric constraints of depth completion can be significantly enhanced, while also improving the robustness of edge perception for transparent and reflective objects.

[0036] S103 fuses polarization physical parameters, geometric prior information, and the original depth map to generate a multimodal conditional tensor for guiding depth information recovery.

[0037] The multimodal conditional tensor refers to a unified data structure (e.g., a multi-channel image tensor) formed by stitching or fusing information from different sources and with different physical meanings (polarization physical parameters, geometric prior information, and the original depth map) along the channel dimension. This provides rich and complementary physical constraints and guiding information for the subsequent depth completion process. Specifically, the intensity image, polarization degree map, polarization angle map, surface normal map, edge map, and original depth map are stitched together along the channel dimension to form a multi-channel multimodal conditional tensor. Assuming that each image except the surface normal map (three channels) is a single channel (grayscale), the multimodal conditional tensor is an 8-channel tensor. It should be noted that during the stitching of different information along the channel dimension, the original depth map acquired by the camera mounted on the robot's head, along with all feature maps extracted from the original polarization image (including intensity image, polarization degree map, polarization angle map, surface normal map, and edge map), are first uniformly transformed to the robot's base coordinate system (i.e., the global coordinate system) using camera calibration parameters and the robot's kinematic model, and then resampled to the same spatial resolution. Subsequently, the original depth map and each feature map are stitched along the channel dimension to obtain a multimodal conditional tensor with K channels, where K is the sum of the number of channels in each feature map. This multimodal conditional tensor integrates spatially aligned complementary information from different physical perception dimensions, providing strong constraints for subsequent depth completion.

[0038] S104, perform depth completion processing on the original depth map based on the multimodal conditional tensor to generate a completed depth map.

[0039] It is understandable that depth sensor failure for transparent or reflective objects results in large areas of voids or noise in the original depth map. Polarization physics parameters can reveal the surface micro-geometric properties and material optical properties that still exist in these areas. By fusing polarization physics parameters, their derived geometric priors (such as surface normals and edges), and the effective portion of the original depth map, a multimodal conditional tensor is formed, which together constitutes a strongly constrained blueprint (including orientation, contour, material, and known spatial points) regarding "how the surface of the target object should be." In other words, the multimodal conditional tensor provides the physical basis and geometric constraints for the completion operation; therefore, depth completion processing can be performed on the original depth map based on the multimodal conditional tensor.

[0040] In some embodiments, S104, depth completion processing is performed on the original depth map based on the multimodal conditional tensor to generate a completed depth map, including: Using the multimodal conditional tensor as a condition, the trained depth completion network is used to perform depth completion processing on the original depth map to generate a completed depth map.

[0041] In this embodiment, the depth completion network specifically refers to a trained generative neural network model guided by a multimodal conditional tensor. Its core function is to receive the original depth map containing missing or distorted regions, and, under the rich physical and geometric constraints provided by the multimodal conditional tensor, predict and generate a pixel-accurate and complete completed depth map. The completed depth map is a depth image with the same resolution and coordinate alignment as the original depth map, in which all missing, hollow, or noisy regions caused by transparency or reflection have been reasonably and accurately filled.

[0042] Understandably, since the trained depth completion network has learned through a large amount of data how to restore a depth region containing missing parts (which can be modeled as specific noise or a mask) into a complete shape that conforms to physical laws and geometric constraints, given a multimodal conditional tensor, the completed depth map output by the depth completion network can be obtained by using a multimodal conditional tensor that integrates physical parameters, geometric priors, and original depth information from polarization images, along with the original depth map, as input to the depth completion network.

[0043] In some embodiments, the deep completion network includes a conditional flow matching model; using a multimodal conditional tensor as a condition, the trained deep completion network performs depth completion processing on the original depth map to generate a completed depth map, including: Using the multimodal conditional tensor as a condition, the trained conditional flow matching model is used to iteratively update the random noise tensor N times through an ordinary differential equation numerical solver, so as to gradually remove noise and approximate the target depth distribution; the random noise tensor is sampled from a standard Gaussian distribution, and N is a preset positive integer; The output tensor obtained after N iterations is used to complete the depth map.

[0044] Among them, the conditional flow matching model is an advanced generative artificial intelligence model. Unlike the diffusion model, which denoises through hundreds of random iterations, conditional flow matching smoothly and deterministically transforms simple distributions (such as Gaussian noise) to complex target data distributions (such as true depth maps) by directly learning a deterministic vector field. It uses a multimodal conditional tensor as the "condition" to guide the entire generation process. This model has the advantages of stable training, fast convergence, and extremely high inference efficiency (typically requiring only about 10 deterministic iterations).

[0045] Specifically, first, the multimodal conditional tensor C is used as input, and a random noise tensor z_1 with the same size as the completed depth map is sampled from a standard Gaussian distribution. The total number of iterations N is set to 10, and the steps from... arrive The time step sequence. Next, for the... The next iteration ( From 1 to ), perform the following sub-steps: (1) change the current noise tensor Current time step The multimodal conditional tensor C is input together with the pre-trained conditional flow matching model. This yields a deterministic vector field f_θ(z_t, t, C) as the output. The direction of this vector field is uniquely determined by the multimodal conditional tensor C, which precisely indicates how to transform the current noisy depth estimate. (1) Move towards a complete depth map that conforms to physical reality (i.e., is consistent with cues such as polarization and normals); (2) Use a numerical solver (such as the Euler method) to update the state based on the predicted vector field, with the update formula being z_s = z_t + (t - s) * f_θ(z_t, t, C), It is the next time step that is closer to 0 (e.g., if...) ,but The calculation, guided by the multimodal conditional tensor C, moves z_t a small step along the vector field direction to obtain an intermediate state z_s with less noise and a clearer structure; (3) z_s is assigned to z_t, and the value is transferred to z_t. Assign to This prepares for the next iteration. Finally, repeat the above steps a total of After that, time Reaching state 0, we obtain the final state z_0. At this point, the initial random noise z_1, guided entirely by the multimodal conditional tensor C, has been completely and deterministically transformed into a high-fidelity depth map. z_0 is then directly used as the final output, thus completing the depth map. ,Right now: = z_0 This completed depth map retains the original values ​​in the valid areas of the original depth map, while generating accurate and reasonable depth values ​​in areas that were originally missing or distorted based on physical information such as polarization.

[0046] Thus, the entire iterative optimization process is driven by a multimodal conditional tensor. The physical information in the multimodal conditional tensor, such as the degree of polarization, polarization angle, and normal, ensures that the completed geometry is physically reliable, fundamentally solving the problem of transparent and reflective objects being unable to be completed due to a lack of texture. At the same time, the iteration process requires fewer iterations, enabling efficient inference and improving processing efficiency.

[0047] S105 performs grasping pose planning for the target object based on the completed depth map and outputs robot grasping instructions. Specifically, the completed depth map can be input into the trained grasping planning network, which then plans the grasping pose for the target object. Based on this planned grasping pose, a robot grasping command is generated and output, which the robot then executes to grasp the target object. It should be noted that the specific implementation process of planning the grasping pose based on the completed depth map and outputting the robot grasping command can refer to existing technologies and will not be elaborated here.

[0048] In some embodiments, S105, the robot performs grasping pose planning based on the completed depth map and outputs grasping instructions, including: The completed depth map is back-projected into a 3D point cloud through a differentiable operation; Based on the 3D point cloud, generate at least one candidate grasping pose; A differentiable grasping evaluation model is used to score the quality of each candidate grasping pose, and a robot grasping command is generated and output based on the candidate grasping pose with the highest score.

[0049] The grasping planning network can include a differentiable grasping evaluation model. This differentiable grasping evaluation model refers to a computational model that can score the quality of candidate grasping poses, and the entire scoring process (from input pose and point cloud to output score) is mathematically differentiable. It typically includes a differentiable spatial symbolic distance field, differentiable interactive computation, and a differentiable evaluation function. By substituting the grasping pose parameters into the differentiable spatial symbolic distance field, the specific geometric configuration of the robot gripper in space under that pose can be calculated differentiably.

[0050] Specifically, firstly, the depth map can be completed. y hat A 3D point cloud P is generated by backprojection using the camera intrinsic parameter matrix and differentiable operations. Then, a grasping planning network based on PointNet++ is used to predict multiple candidate grasping poses, taking the point cloud P as input. G i Next, a differentiable grasping evaluation model is used to score the quality of each candidate grasping pose. The candidate grasping pose with the highest score is selected, and a robot grasping command is generated and output based on the candidate grasping pose with the highest score.

[0051] In some embodiments, a differentiable grasping evaluation model is used to score the quality of each candidate grasping pose, including: Based on the candidate grasping pose, determine the spatial symbolic distance field corresponding to the robot gripper in three-dimensional space; Calculate the interaction information between the 3D point cloud and the spatial symbolic distance field; Based on the interaction information, the grasping quality score of the candidate grasping pose is calculated.

[0052] For each candidate pose, the following operations can be performed using a differentiable grasping evaluation model: First, based on the rotation matrix and translation parameters in the candidate pose, the predefined gripper spatial symbolic distance field is transformed to the world coordinate system to obtain the gripper spatial symbolic distance field under that candidate pose. Then, the set of distance values ​​for all points in the 3D point cloud within that gripper spatial symbolic distance field is calculated, serving as the interaction information between the 3D point cloud and the spatial symbolic distance field. Finally, a lightweight neural network is used as a differentiable evaluation function, taking the statistical characteristics of the distance value set (such as mean, variance, and quantiles) as input, and outputting a grasping quality score between 0 and 1 as the quality score result for that candidate grasping pose.

[0053] In some embodiments, the conditional flow matching model and the differentiable grasping evaluation model are obtained through end-to-end optimization training using a joint loss function that includes depth reconstruction loss and grasping task loss.

[0054] Specifically, a multi-task loss function can be constructed that includes depth reconstruction loss and grasping task loss; through gradient backpropagation, the gradient generated by the grasping task loss is passed to the conditional flow matching model via the differentiable grasping evaluation model, and the conditional flow matching model and the differentiable grasping evaluation model are jointly optimized in an end-to-end manner.

[0055] The end-to-end joint training method means that the deep completion network (including the conditional flow matching model) and the grasping planning network (including the conditional flow matching model) are no longer trained in isolation, but are optimized simultaneously through a unified multi-task loss function and a gradient backpropagation path throughout the entire system. Failure signals in the grasping task can directly affect the parameters of the deep completion network, causing it to learn to generate geometry more favorable for grasping. This involves first collecting a large number of data pairs containing transparent and / or reflective objects and their true depth maps. Then, only the depth reconstruction loss is used. L depth (e.g., Smooth L1 Loss) is used to train the conditional flow matching model v_θ. The goal of this stage is to enable the conditional flow matching model to generate geometrically accurate depth maps under the guidance of multimodal conditions such as polarization parameters. Next, a small dataset containing successful and failed capture annotations is used to construct the total loss function. L total During this phase, the parameters of both the conditional flow matching model v_θ and the differentiable grasping evaluation model are updated simultaneously. The grasping loss... L grasp The gradient will propagate back along the differentiable path, all the way back to the conditional flow matching model v_θ, so that v_θ not only learns to reconstruct accurate depth, but also learns to generate depth map features that enable the differentiable grasping evaluation model to give high scores, thereby achieving task-driven perception optimization.

[0056] During the joint training phase, the total losses were... L total The gradients are simultaneously backpropagated to both the conditional flow matching model and the differentiable grasping evaluation model. First, the gradients within the grasping planning network are updated. Specifically, based on the grasping score... s i Compared to real tag crawling l i The binary cross-entropy loss between them is used to calculate the loss function. L grasp Scraping score s i partial derivatives L grasp / s i This gradient is further propagated to its network parameters. , obtain gradient Finally, the gradient is used. The process involves updating all trainable parameters of the grasping planning network, which teaches it to provide more accurate grasping scores based on the depth map. Then, gradients are transferred from the grasping planning network to the depth map, specifically through a differentiable projection transformation, converting point cloud-level gradients to pixel-level gradients in the depth map. This step calculates the impact of changes in each pixel value in the depth map on the success rate of the capture. For example, capturing pixels near the contact point will yield a larger gradient value. Finally, the gradient is propagated from the depth map to the depth completion network, and through a conditional flow matching model (such as the U-Net network), it eventually reaches all the parameters of the depth completion network. According to the chain rule, the gradient formula for the grasping loss with respect to the parameters of the depth completion network can be expressed as: This allows it to generate a depth map that is more advantageous for crawling tasks. Furthermore, L depth Gradient backpropagation will also be performed, which means directly backpropagating along the depth map calculated by the depth completion network to update the same network parameters. Therefore, the total update gradient of a deep completion network can be viewed as... L total / θ d = λ1 ×( L depth / θ d ) + λ2 × ( L grasp / θ d ).

[0057] Thus, by employing a two-stage joint training strategy, deep perception and grasping tasks are closely integrated, enabling the deep completion network to directly receive optimization signals from the grasping task and learn to generate the deep representation most favorable for grasping planning. This achieves task-level collaborative optimization of perception and grasping, improving the model's success rate and efficiency in grasping tasks.

[0058] Based on the same inventive concept as the foregoing embodiments, the above method will be described in detail below through a specific example.

[0059] This embodiment provides a robot control system to implement the above-described robot control method. The system may include a polarization multimodal sensing module, a deep completion network based on conditional flow matching, a differentiable grasping prediction and evaluation module, and a joint optimization strategy module, such as... Figure 2 As shown below, the functions of each part are briefly explained: (I) Polarization Multimodal Sensing Module This module converts raw polarization camera data and depth sensor data into multimodal feature representations rich in physical information. The inputs are the raw polarization image (typically angles in four directions, such as 0°, 45°, 90°, and 135°) and the raw depth map. The output is a multi-channel tensor containing an intensity image, polarization degree map, polarization angle map, and derived geometric priors (surface normal map, edge map). The specific implementation process is as follows: (1) Stokes vector calculation Calculate the Stokes vector [S0, S1, S2] from the original polarization image. This is a physical quantity describing the polarization state of light: S0 = I0 + I90; S1 = I0 - I90; S2 = I45 - I135. Where I0, I45, I90, and I135 correspond to the intensity values ​​in the four polarization directions, respectively.

[0060] (2) Polarization parameter extraction: Intensity Image (RGB-I): Treating S0 as unpolarized intensity information, it can be converted into a grayscale image or fused with an image interpolated by a color filter array to generate a three-channel RGB intensity image.

[0061] Degree of polarization (DoP): DoP = sqrt(S1^2 + S2^2) / S0. The DoP diagram reflects the polarization capability of an object's surface. Highly reflective materials such as metals and glass have high DoP values ​​at specific angles, which can be used to distinguish materials.

[0062] Polarization angle (AoP): AoP = 0.5 * arctan2(S2, S1). The AoP map encodes the azimuth information of the surface normal in the camera coordinate system and is a key clue for recovering the surface normal.

[0063] (3) Geometric prior extraction Surface normal map: Based on the principle of physical polarization vision, there is a functional relationship between the incident zenith angle and the DoP, and the corresponding zenith angle θ can be estimated. By combining the AoP (the value of each pixel is equal to the azimuth angle of the surface normal) and the estimated zenith angle θ, a preliminary surface normal map N_polar∈ R^(HxWx3) can be calculated.

[0064] Edge map: Apply the Sobel or Canny operator to the intensity image (S0) and DoP image respectively to extract edges, and then fuse the results to obtain an edge map E∈R^(HxWx1) that is more sensitive to the outline of transparent objects and internal refraction edges.

[0065] (4) Feature fusion and splicing The depth map D_global obtained from the head RGB-D camera and other features mapped to the global coordinate system are concatenated along the channel dimension to form the final multimodal conditional tensor C, i.e.: C=Concat(RGB-I, DoP, AoP, N_polar, E, D_global).

[0066] (II) Deep Completion Network Based on Conditional Flow Matching This module aims to learn a conditional probability path that transforms a simple noise distribution into a target depth map distribution, and performs inference with far greater efficiency than diffusion models. The inputs are a noisy depth map z_t, a time step t, and a multimodal conditional tensor C. The outputs are either the denoised depth map z_{t-1} (during inference) or the predicted vector field v_θ (during training).

[0067] The specific implementation process is as follows: 1) Network Architecture: An improved U-Net is adopted as the backbone network. Its encoder part downsamples the multimodal conditional tensor C and extracts multi-scale features. The extracted conditional features are injected into the decoder layer of U-Net through a cross-attention mechanism and fused with the current noise latent variable z_t (also known as the noise point) and the time step embedding t.

[0068] 2) Training objective: The true depth map distribution.

[0069] The core idea of ​​flow matching is to directly regress a target vector field v. In conditional flow matching, the goal of the network v_θ is to predict the flow from noise point z_t to the real data y.

[0070] For any time point t ~ U(0,1], the noise latent variable is z_t = (1 - t) * y + t *ε, where ε ~ N(0, I).

[0071] The target vector field is v_t = y -ε.

[0072] The training loss function L_CFM uses a simple mean squared error loss: L_CFM = E_{t, y, ε} [ || v_θ(z_t, t, C) - (y - ε) ||^2 ].

[0073] This training objective is simpler than the variational lower bound of the diffusion model, resulting in more stable training and faster convergence.

[0074] 3) Reasoning (Sampling): After training, the Euler method ODE solver is used to solve the ordinary differential equations in reverse from random noise z_1 ~ N(0, I) to generate a depth map.

[0075] In each iteration, the noise latent variable is updated according to the formula z_{s} = z_t + (t - s) * v_θ(z_t, t, C). It only takes about 10 steps (s = t - 1 / 10) to integrate from z_1 to z_0, obtaining a high-fidelity depth map prediction. y hat = z_0.

[0076] (III) Differentiable Grasping Attitude Prediction and Evaluation Module This module will complete the depth map. y hat It is converted into a grasping pose that can be used for robot execution and designed to be differentiable to allow gradients to backpropagate from the grasping task loss to the depth map.

[0077] The specific implementation process is as follows: 1) Differentiable point cloud generation: Using the camera intrinsic parameter matrix, the depth map y_hat is back-projected into a 3D point cloud P∈R^(Nx3) using differentiable operations, so that the gradient can be back-propagated from the point cloud level to every pixel of the depth map.

[0078] 2) Candidate grasping pose generation: A learning-based grasping planning network is adopted. This network takes the point cloud P as input and extracts global and local features through architectures such as PointNet++ to predict a set of 6-DoF candidate grasping poses for each point or point group.G i = (R_{3x3}, T_{3x1}), and the grab width w i .

[0079] 3) Differentiable capture scoring Design a differentiable grasping evaluation model that simulates the grasping action, with the grasping pose as its input. G i Given a point cloud P, the output is a capture quality score. s i ∈ [0, 1].

[0080] The key to achieving differentiability lies in using approximate differentiable methods such as soft rendering or continuous collision detection. For example, a signed distance field (SDF) in space can be defined for each of the two grippers in the grasp evaluation model, and the interaction between the point cloud and the gripper SDF can be calculated to approximate a differentiable version of the force closure index.

[0081] Finally, the crawling evaluation model is used to evaluate each candidate crawling location. G i Output a score s i .

[0082] 4) Grab Selection Select the candidate capture pose with the highest score G* = argmax( s i This will be used as the final output.

[0083] (iv) Joint Optimization Strategy Module By designing a multi-task loss function, the two modules mentioned above are connected into an end-to-end trainable system, with the goal of optimizing the parameters of the deep completion network. and capture planning network parameters This maximizes the final success rate of data capture.

[0084] The specific implementation process is as follows: 1) Multi-task loss function Total loss function L total It is the loss of deep reconstruction. L depth and capture task losses L grasp Weighted sum: L total =λ1* L depth + λ2 * .

[0085] L depth Calculated depth map prediction and true depth map y gt The Smooth L1 loss ensures the basic geometric accuracy of the depth map.

[0086] L grasp A binary cross-entropy loss is used. For each candidate grasp pose... G i Compare it with the actual successfully captured pose. G gt Calculate IoU or distance, and generate a binary label. l i (1 indicates a positive capture, 0 indicates a negative capture). L grasp = BCE( s i , l i ).

[0087] 2) Gradient backpropagation: Scraping loss function L grasp The gradient can propagate backward along a differentiable path, that is: Update the network parameters for capture planning.

[0088] The gradient continues to propagate forward.

[0089] Gradient generation through differentiable point cloud The parameters are calculated and then backpropagated to the deep completion network v_θ. : .

[0090] 3) Training strategy Phase 1 (Pre-training): Using only... L depth The loss is obtained by pre-training a deep completion network on a large deep completion dataset, resulting in a good initialization model.

[0091] Phase Two (Joint Fine-tuning): Introducing Fetch Loss L grasp On a smaller dataset containing ground truth data for crawling, the entire workflow is jointly fine-tuned end-to-end with a smaller learning rate to achieve the ultimate goal of perception serving crawling.

[0092] In summary, the robot control system and method provided in the above embodiments have the following technical effects: (1) By introducing the calculation of Stokes vectors, the polarization degree map and polarization angle map are parsed from the original polarization image, and the preliminary surface normal map is further derived. Finally, the features are spliced ​​and fused to form a multimodal conditional tensor rich in physical meaning, which provides a strong physical constraint for depth completion and solves the fundamental problem of missing or distorted visual information of transparent and reflective objects. (2) Depth completion is modeled as learning a deterministic vector field from a simple noise distribution to a complex real depth map distribution. During training, it directly regresses the target vector through a simple mean square error loss; during inference, it generates a high-fidelity depth map from noise in about 10 steps through an ordinary differential equation solver, realizing real-time high-precision completion and maintaining a generation quality comparable to or even better than the diffusion model. (3) A multi-task loss function is designed to combine the pixel-level reconstruction loss of the depth completion network with the task-level grasping loss of the grasping planning network. By using a differentiable grasping model, the gradient of the grasping task can be backpropagated to the deep completion network, thereby guiding the deep completion network to generate a more grasping-friendly geometric representation, realizing task-driven perception and directly improving the task success rate.

[0093] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a computing device, such as... Figure 3 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 3 The processor 310 shown in the diagram does not indicate that there is only one processor 310, but only indicates the positional relationship of the processor 310 relative to other devices. In practical applications, there can be one or more processors 310; similarly, Figure 3 The memory 311 shown in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of memory 311 relative to other devices. In practical applications, there can be one or more memories 311. When the processor 310 runs the computer program, the robot control method applied to the above-mentioned computing device is implemented.

[0094] The device may also include at least one network interface 312. The various components of the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to implement communication between these components. In addition to a data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in... Figure 3 The general designated all buses as Bus System 313.

[0095] The memory 311 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 311 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0096] The memory 311 in this embodiment of the invention is used to store various types of data to support the operation of the device. Examples of this data include: any computer programs used to operate on the device, such as operating systems and applications; contact data; phonebook data; messages; pictures; videos, etc. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications, such as media players, browsers, etc., used to implement various application services. Here, the program implementing the method of this embodiment of the invention can be included in the application.

[0097] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the robot control method as described above.

[0098] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0099] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0100] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A robot control method characterized by, The method comprises: obtaining an original polarization image and an original depth map of a target object; extracting a polarization physical parameter containing a physical property of an object surface from the original polarization image, and deriving geometric prior information based on the polarization physical parameter; fusing the polarization physical parameter, the geometric prior information and the original depth map to generate a multi-modal conditional tensor for guiding depth information recovery; performing depth completion processing on the original depth map based on the multi-modal conditional tensor to generate a completed depth map; planning a grasping pose of the target object according to the completed depth map and outputting a robot grasping instruction.

2. The method of claim 1, wherein, The polarization physical parameter comprises an intensity image, a degree of polarization image and a polarization angle image; the extracting of the polarization physical parameter from the original polarization image comprises: calculating a Stokes vector corresponding to the original polarization image; extracting the intensity image, the degree of polarization image and the polarization angle image of the original polarization image according to the Stokes vector.

3. The method of claim 2, wherein, The geometric prior information comprises at least one of a surface normal map and an edge map; the deriving of the geometric prior information based on the polarization physical parameter comprises at least one of: determining the surface normal map by using the degree of polarization image and the polarization angle image based on a polarization vision principle; respectively performing edge detection on the intensity image and the degree of polarization image, and fusing the edge detection results to generate the edge map.

4. The method of claim 1, wherein, The depth completion processing on the original depth map based on the multi-modal conditional tensor to generate a completed depth map comprises: performing depth completion processing on the original depth map by using a trained depth completion network with the multi-modal conditional tensor as a condition to generate the completed depth map.

5. The method of claim 4, wherein, The depth completion network comprises a conditional flow matching model; the depth completion processing on the original depth map by using the trained depth completion network with the multi-modal conditional tensor as a condition to generate the completed depth map comprises: using the trained conditional flow matching model to perform N times of iterative updates on a random noise tensor by using a numerical solver of ordinary differential equations with the multi-modal conditional tensor as a condition, so as to gradually remove noise and approximate a target depth distribution; the random noise tensor is sampled from a standard Gaussian distribution, and N is a preset positive integer; determining an output tensor obtained after the N times of iterative updates as the completed depth map.

6. The method according to any one of claims 1 to 5, characterized in that, The grasping pose planning of the target object according to the completed depth map and the outputting of the robot grasping instruction comprise: back-projecting the completed depth map into a three-dimensional point cloud by using a differentiable operation; generating at least one candidate grasping pose of the target object based on the three-dimensional point cloud; scoring the quality of each candidate grasping pose by using a differentiable grasping evaluation model, and generating and outputting the robot grasping instruction based on the candidate grasping pose with the highest score.

7. The method of claim 6, wherein, The scoring of the quality of each candidate grasping pose by using the differentiable grasping evaluation model comprises: determining a spatial signed distance field corresponding to a robot gripper in a three-dimensional space based on the candidate grasping pose; calculating interaction information between the three-dimensional point cloud and the spatial signed distance field; Based on the interaction information, a grasping quality score of the candidate grasping pose is calculated.

8. The method of claim 6, wherein, The conditional flow matching model and the differentiable grasping evaluation model are obtained by end-to-end optimization training of a joint loss function comprising a depth reconstruction loss and a grasping task loss.

9. A computing device, comprising: The method comprises: A storage medium and a controller, the storage medium storing a computer program, the computer program being executed by the controller to implement the robot control method according to any one of claims 1 to 8.

10. A readable storage medium, characterized by, A storage medium storing a computer program, the computer program being executed by a processor to implement the robot control method according to any one of claims 1 to 8.

Images