Robot control model training method, computing device and readable storage medium

Abstract

The invention discloses a robot control model training method, computing equipment and a readable storage medium, and the method comprises the steps: training a deep completion network through a deep reconstruction loss function based on a first training data set; the depth completion network is configured to fuse information carried by the original polarization image and the original depth map to generate a predicted depth map; based on the second training data set, performing end-to-end training on the deep completion network and the grabbing planning network by using a joint loss function; the joint loss function is the weighted sum of the depth reconstruction loss function and the capture task loss function; and the grabbing planning network is configured to perform grabbing pose planning according to the predicted depth map and determine a corresponding grabbing pose. In this way, the success rate and efficiency of the model grabbing task can be improved.

Description

Technical Field

[0001] This invention relates to the field of robotics, and in particular to a method for training a robot control model, a computing device, and a computer-readable storage medium. Background Technology

[0002] In the fields of industrial automation and service robots, reliably grasping transparent objects (such as plastic pipes) and reflective objects (such as polished metal parts and automotive lights) using trained robot control models has long been a technical challenge. These objects, due to their unique optical properties, cause vision systems based on traditional RGB (red, green, blue) or RGB-D (red, green, blue-depth) cameras to fail. For example, light emitted by depth sensors may be transmitted through the surface of transparent objects or specularly reflected on reflective surfaces, failing to return a valid depth signal. This results in large areas of voids or severe noise in the depth map, making it difficult to grasp transparent and reflective objects due to missing or distorted visual information. In other words, robot control models trained using existing methods suffer from low success rates in grasping tasks. Summary of the Invention

[0003] The purpose of this application is to provide a training method, computing device, and computer-readable storage medium for a robot control model, which can improve the success rate and efficiency of the model's grasping task.

[0004] To achieve the above objectives: In a first aspect, embodiments of this application provide a method for training a robot control model, the robot control model including a deep completion network and a grasping planning network, wherein the output of the deep completion network is connected to the input of the grasping planning network; the method includes: The depth completion network is trained using a depth reconstruction loss function based on a first training dataset. The first training dataset includes multiple first samples, each of which includes an original polarization image, an original depth map containing missing or distorted regions, and a true complete depth map corresponding to the original depth map. The depth completion network is configured to fuse the information carried by the original polarization image and the original depth map to generate a predicted depth map. Based on a second training dataset, the depth completion network and the grasping planning network are trained end-to-end using a joint loss function. The second training dataset includes multiple second samples, each of which includes an original polarization image, an original depth map containing missing or distorted regions, and a true complete depth map and a true grasping pose label corresponding to the original depth map. The joint loss function is a weighted sum of the depth reconstruction loss function and the grasping task loss function. The grasping planning network is configured to plan the grasping pose based on the predicted depth map to determine the corresponding grasping pose. In the end-to-end training step, the gradient generated by the grasping task loss function is backpropagated through the grasping planning network to the predicted depth map output by the depth completion network, and further propagated to the parameters of the depth completion network.

[0005] In some embodiments, the depth completion network includes a conditional flow matching model; the input to the depth completion network is a multimodal conditional tensor generated based on the original polarization image and the original depth map; The depth completion network is configured to perform depth completion using the conditional flow matching model, based on the multimodal conditional tensor, to generate the predicted depth map.

[0006] In some embodiments, the multimodal conditional tensor is generated through the following steps: Polarization physical parameters are extracted from the original polarization image, and geometric prior information is derived based on the polarization physical parameters; The polarization physical parameters, the geometric prior information, and the original depth map are fused to generate a multimodal conditional tensor for guiding depth information recovery.

[0007] In some embodiments, the polarization physical parameters include an intensity image, a degree of polarization map, and a polarization angle map; extracting the polarization physical parameters from the original polarization image includes: Calculate the Stokes vector corresponding to the original polarization image; Based on the Stokes vector, the intensity image, the degree of polarization map, and the angle of polarization map of the original polarization image are extracted.

[0008] In some embodiments, the geometric prior information includes at least one of the following: a surface normal map and an edge map; the derivation of the geometric prior information based on the polarization physics parameters includes at least one of the following: Based on the principle of polarization vision, the surface normal map is determined using the degree of polarization map and the angle of polarization map; Edge detection is performed on the intensity image and the polarization map respectively, and the edge detection results are fused to generate the edge map.

[0009] In some embodiments, the training and inference process of the conditional flow matching model includes: During training, the conditional flow matching model is trained using the multimodal conditional tensor as a condition, such that for a given time step t and a noise latent variable z_t constructed according to z_t = (1 - t) * y + t * ε, the conditional flow matching model outputs a prediction vector to approximate the target vector field v_t; where y is the true complete depth map, ε is random noise, and v_t = y - ε; the corresponding loss function is the mean square error between the prediction vector and the target vector field. During inference, the multimodal conditional tensor is used as a condition, starting from random noise z_1~N(0, I), and through an ordinary differential equation numerical solver, the vector field predicted by the trained conditional flow matching model is iteratively updated. In each iteration, the noise latent variable is updated according to the formula z_s = z_t + (t - s) * f_θ(z_t, t, C), where t and s are adjacent time steps, C is the multimodal conditional tensor, and f_θ is the conditional flow matching model. After a preset number of iterations N, the predicted depth map is obtained.

[0010] In some embodiments, the training and inference process of the conditional flow matching model includes: During training, the conditional flow matching model is trained using the multimodal conditional tensor as a condition, such that for a given time step t and a noise latent variable z_t constructed according to z_t = (1 - t) * y + t * ε, the conditional flow matching model outputs a prediction vector to approximate the target vector field v_t; where y is the true complete depth map, ε is random noise, and v_t = y - ε; the corresponding loss function is the mean square error between the prediction vector and the target vector field. During inference, the multimodal conditional tensor is used as a condition, starting from random noise z_1~N(0, I), and through an ordinary differential equation numerical solver, the vector field predicted by the trained conditional flow matching model is iteratively updated. In each iteration, the noise latent variable is updated according to the formula z_s = z_t + (t - s) * f_θ(z_t, t, C), where t and s are adjacent time steps, C is the multimodal conditional tensor, and f_θ is the conditional flow matching model. After a preset number of iterations N, the predicted depth map is obtained.

[0011] In some embodiments, the grasping task loss function is calculated based on the grasping quality score output by the differentiable grasping evaluation model; the gradient backpropagation process includes: Calculate the loss function of the crawling task The parameters of the grasping planning network gradient And according to the gradient Update the parameters of the crawling planning network. ; Calculate the loss function of the crawling task For the predicted depth map gradient The gradient It is used to characterize the impact of changes in each pixel value in the depth map on the success rate of capture; Calculate the crawling task loss function according to the chain rule. The parameters of the deep completion network Total gradient The parameters of the depth completion network are updated based on the total gradient. .

[0012] Secondly, embodiments of this application provide a computing device, including: a storage medium and a controller, wherein a computer program is stored on the storage medium, and the computer program, when executed by the controller, implements the training method for the robot control model as described in the first aspect.

[0013] Thirdly, embodiments of this application provide a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the training method for the robot control model as described in the first aspect.

[0014] This application provides a robot control model training method, computing device, and computer-readable storage medium. The robot control model includes a depth completion network and a grasping planning network, with the output of the depth completion network connected to the input of the grasping planning network. The method includes: training the depth completion network based on a first training dataset using a depth reconstruction loss function; wherein the first training dataset includes multiple first samples, each first sample including an original polarization image, an original depth map containing missing or distorted regions, and a true complete depth map corresponding to the original depth map; the depth completion network is configured to fuse the information carried by the original polarization image and the original depth map to generate a predicted depth map; and training the robot control model based on a second training dataset using a joint loss function. The depth completion network and the grasping planning network are trained end-to-end. The second training dataset includes multiple second samples, each including an original polarization image, an original depth map containing missing or distorted regions, and a true complete depth map and a true grasping pose label corresponding to the original depth map. The joint loss function is a weighted sum of the depth reconstruction loss function and the grasping task loss function. The grasping planning network is configured to plan the grasping pose based on the predicted depth map, determining the corresponding grasping pose. In the end-to-end training step, the gradient generated by the grasping task loss function is backpropagated through the grasping planning network to the predicted depth map output by the depth completion network, and further propagated to the parameters of the depth completion network. Thus, through an innovative two-stage joint training strategy, depth perception and the grasping task are closely integrated, enabling the depth completion network to directly receive optimization signals from the grasping task and learn to generate the most favorable depth representation for grasping planning. This achieves task-level collaborative optimization of perception and grasping, improving the model's grasping task success rate and efficiency. Meanwhile, the introduction of polarization vision modality allows us to obtain surface physical properties that remain effective even when visual information is missing or distorted, providing irreplaceable physical constraints for depth recovery and enabling the trained model to accurately capture transparent and reflective objects. Attached Figure Description

[0015] Figure 1 This is a flowchart illustrating the training method for a robot control model provided in an embodiment of the present invention.

[0016] Figure 2 This is a schematic diagram of the structure of a computing device provided in an embodiment of the present invention. Detailed Implementation

[0017] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. In the following description relating to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements.

[0018] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes that element. Furthermore, components, features, and elements with the same names in different embodiments of this application may have the same meaning or different meanings, the specific meaning of which must be determined by its interpretation in that specific embodiment or further in conjunction with the context of that specific embodiment.

[0019] It should be understood that although the terms first, second, third, etc., may be used herein to describe various information, such information should not be limited to these terms. These terms are used only to distinguish information of the same type from one another. For example, without departing from the scope of this document, first information may also be referred to as second information, and similarly, second information may also be referred to as first information. Depending on the context, the word "if," as used herein, can be interpreted as "when," "when," or "in response to determination." Furthermore, as used herein, the singular forms "a," "an," and "the" are intended to also include the plural forms unless the context indicates otherwise. It should be further understood that the terms "comprising," "including," indicate the presence of the stated feature, step, operation, element, component, item, kind, and / or group, but do not exclude the presence, occurrence, or addition of one or more other features, steps, operations, elements, components, items, kinds, and / or groups. The terms "or" and "and / or" as used herein are to be interpreted as inclusive, or mean any one or any combination thereof. Therefore, "A, B, or C" or "A, B, and / or C" means "any one of the following: A; B; C; A and B; A and C; B and C; A, B, and C". Exceptions to this definition will only occur if the combination of elements, functions, steps, or operations is inherently mutually exclusive in some way.

[0020] It should be understood that although the steps in the flowcharts of this application's embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least a portion of the sub-steps or stages of other steps.

[0021] It should be noted that step designations such as S101 and S102 are used in this document for the purpose of more clearly and concisely describing the corresponding content, and do not constitute a substantial limitation on the order. In specific implementation, those skilled in the art may execute S102 first and then S101, etc., but these should all be within the protection scope of this application.

[0022] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0023] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustrative purposes and has no specific meaning in itself. Therefore, "module," "part," or "unit" may be used interchangeably.

[0024] See Figure 1 This application provides a method for training a robot control model. The robot control model includes a depth completion network and a grasping planning network. The output of the depth completion network is connected to the input of the grasping planning network. This method can be executed by a robot control model training device provided in this application. This device can be implemented in software and / or hardware, such as a computer, processor, or server. In this embodiment, the execution subject of the method is a computing device. The method provided in this embodiment includes: S101, a depth completion network is trained based on a first training dataset using a depth reconstruction loss function; wherein, the first training dataset includes multiple first samples, each first sample including an original polarization image, an original depth map containing missing or distorted regions, and a real complete depth map corresponding to the original depth map; the depth completion network is configured to fuse the information carried by the original polarization image and the original depth map to generate a predicted depth map.

[0025] The first training dataset contains a large number of first samples, each of which is a triplet (original polarization image, original depth map, and true complete depth map). The original depth map includes holes and noise caused by object transparency or reflection; the true complete depth map is a depth map obtained from the same scene using high-precision methods (such as scanning after applying a matte finish). A depth reconstruction loss function (such as Smooth L1 Loss) is used as a supervision signal to train the depth completion network. In each iteration, the network learns to fuse the information carried by the original polarization image and the original depth map, and outputs a predicted depth map. The goal of training the depth completion network is to minimize the difference between the predicted depth map and the true complete depth map.

[0026] In some embodiments, the depth completion network includes a conditional flow matching model; the input to the depth completion network is a multimodal conditional tensor generated based on the original polarization image and the original depth map; The depth completion network is configured to perform depth completion using a conditional flow matching model, based on a multimodal conditional tensor, to generate a predicted depth map.

[0027] Among them, the multimodal conditional tensor refers to a unified data structure (such as a multi-channel image tensor) formed by stitching or fusing information from different sources and with different physical meanings (such as polarization physical parameters, geometric prior information, and original depth maps) in the channel dimension, so as to provide rich and complementary physical constraints and guiding information for the subsequent depth completion process.

[0028] In this embodiment, the depth completion network specifically refers to a trained generative neural network model guided by a multimodal conditional tensor. Its core function is to receive the original depth map containing missing or distorted regions, and, under the rich physical and geometric constraints provided by the multimodal conditional tensor, predict and generate a pixel-accurate and complete completed depth map. The completed depth map is a depth image with the same resolution and coordinate alignment as the original depth map, in which all missing, hollow, or noisy regions caused by transparency or reflection have been reasonably and accurately filled.

[0029] Understandably, since the trained depth completion network has learned through a large amount of data how to restore a depth region containing missing parts (which can be modeled as specific noise or a mask) into a complete shape that conforms to physical laws and geometric constraints, given a multimodal conditional tensor, the completed depth map output by the depth completion network can be obtained by using a multimodal conditional tensor that integrates physical parameters, geometric priors, and original depth information from polarization images, along with the original depth map, as input to the depth completion network.

[0030] In some embodiments, the multimodal conditional tensor is generated through the following steps: Polarization physical parameters are extracted from the original polarization image, and geometric prior information is derived based on the polarization physical parameters; By fusing polarization physical parameters, geometric prior information, and the original depth map, a multimodal conditional tensor is generated to guide depth information recovery.

[0031] The raw polarization image refers to a collection of images captured by a polarization camera, containing light intensity information in specific polarization directions (typically 0°, 45°, 90°, and 135°). Each pixel in the image contains intensity information in multiple directions, characterizing the change in polarization state after light interacts with the object's surface. The raw depth map refers to an image or matrix composed of depth data directly acquired by a depth sensor (such as a structured light camera, a time-of-flight camera, or a stereo vision system). The value of each pixel in this raw depth map typically represents the physical distance or parallax of the corresponding point in the scene to the sensor plane. For example, a polarization camera (such as a camera equipped with a four-way polarization filter) mounted on the robot's wrist and a depth sensor (such as an RGB-D camera) mounted on the robot's head can be aimed at an object (such as a transparent plastic tube) within the working area, simultaneously acquiring its raw polarization image (four intensity maps at different angles) and raw depth map.

[0032] Among them, polarization physics parameters refer to parameters calculated from the original polarization image that directly reflect the physical properties (such as material and micro-geometry) of an object's surface. Polarization physics parameters can include intensity images, degree of polarization maps, and angle of polarization maps. The intensity image (RGB-I) is calculated using Stokes vectors and characterizes the total light intensity of the object's surface; it can be approximated as a regular grayscale or color image after removing polarization effects. The degree of polarization map (DoP) reflects the object's ability to polarize incident light. Highly reflective materials (such as metals and glass) have high DoP values ​​at specific observation angles, providing important clues for distinguishing materials and inferring surface orientation. The angle of polarization map (AoP) encodes the projection direction (azimuth angle) of the surface normal onto the camera's imaging plane, providing crucial information for reconstructing the surface's three-dimensional orientation.

[0033] Geometric prior information refers to estimates of the macroscopic or microscopic geometric structure of an object's surface, derived from polarization physics parameters through physical models or image processing techniques. Geometric prior information may include at least one of the following: a surface normal map and an edge map. The surface normal map is calculated based on the principle of polarization vision, utilizing the relationship between DoP and zenith angle, and the azimuth information provided by AoP, to determine the three-dimensional normal direction of each point on the object's surface. The edge map is an image obtained by performing edge detection (such as using Sobel or Canny operators) on the intensity image and the polarization degree map, and then fusing the results. Because the polarization degree map is sensitive to abrupt changes in intensity caused by refraction and reflection, obtaining the edge map based on the polarization degree map can effectively enhance the edge information of the outline and internal structure of transparent objects.

[0034] The process involves concatenating the intensity image, polarization degree map, polarization angle map, surface normal map, edge map, and original depth map along the channel dimension to form a multi-channel multimodal conditional tensor. Assuming each image except the surface normal map (three channels) is a single channel (grayscale), the multimodal conditional tensor is an 8-channel tensor. It's important to note that during the concatenation of different information along the channel dimension, the original depth map acquired by the camera mounted on the robot's head, along with all feature maps extracted from the original polarization image (including the intensity image, polarization degree map, polarization angle map, surface normal map, and edge map), are first uniformly transformed to the robot's base coordinate system (i.e., the global coordinate system) using camera calibration parameters and the robot's kinematic model, and then resampled to the same spatial resolution. Subsequently, the original depth map and each feature map are concatenated along the channel dimension to obtain a multimodal conditional tensor with K channels, where K is the sum of the number of channels in each feature map. This multimodal conditional tensor integrates spatially aligned complementary information from different physical perception dimensions, providing strong constraints for subsequent deep completion.

[0035] It is understandable that depth sensor failure for transparent or reflective objects results in large areas of voids or noise in the original depth map. Polarization physics parameters can reveal the surface micro-geometric properties and material optical properties that still exist in these areas. By fusing polarization physics parameters, their derived geometric priors (such as surface normals and edges), and the effective portion of the original depth map, a multimodal conditional tensor is formed, which together constitutes a strongly constrained blueprint (including orientation, contour, material, and known spatial points) regarding "how the surface of the target object should be." In other words, the multimodal conditional tensor provides the physical basis and geometric constraints for the completion operation; therefore, depth completion processing can be performed on the original depth map based on the multimodal conditional tensor.

[0036] In some embodiments, polarization physical parameters include an intensity image, a degree of polarization map, and a polarization angle map; extracting polarization physical parameters from the original polarization image includes: Calculate the Stokes vector corresponding to the original polarization image; Based on the Stokes vector, extract the intensity image, degree of polarization map, and angle of polarization map of the original polarization image.

[0037] First, the Stokes vectors [S0, S1, S2] corresponding to the original polarization image are calculated using the formulas S0 = I0 + I90, S1 = I0 - I90, and S2 = I45 - I135, where I0, I45, I90, and I135 represent the intensity values ​​at 0°, 45°, 90°, and 135°, respectively, for the four polarization directions. Next, S0 is treated as unpolarized intensity information and converted to a grayscale image or fused with an image interpolated by a color filter array to generate a three-channel RGB intensity image. The degree of polarization is calculated using the formula DoP = sqrt(S1^2 + S2^2) / S0, and the angle of polarization is calculated using the formula AoP = 0.5 * arctan2(S2, S1). Thus, calculating the polarization physical parameters based on the Stokes vectors ensures the extraction of the maximum amount of physically meaningful surface information from the original sensor data, providing multi-dimensional and interpretable constraints for subsequent geometric prior derivation and depth completion.

[0038] In some embodiments, the geometric prior information includes at least one of the following: surface normal map, edge map; the geometric prior information derived based on polarization physics parameters includes at least one of the following: Based on the principle of polarization vision, the surface normal map is determined using the degree of polarization map and the angle of polarization map. Edge detection is performed on the intensity image and polarization map respectively, and the edge detection results are fused to generate an edge map.

[0039] Specifically, using polarization degree maps and polarization angle maps, combined with the known camera-light source relative positions, a preliminary surface normal map is estimated through a physical inversion algorithm. Simultaneously, Sobel or Canny edge detection operators are applied to the intensity image and polarization degree map respectively to extract edges. The two edge maps are then logically ORed to fuse them into a more sensitive edge map that captures the object's contour and internal refractive edges. This is understandable, as the polarization degree map is extremely sensitive to intensity changes caused by refraction and reflection, effectively capturing characteristic edges inside transparent objects and on reflective objects. Fusion with the intensity image edges yields a more complete and accurate representation of the object's contour than a single modality, significantly improving the accuracy of object boundary localization in subsequent pose estimation. Furthermore, since the surface normal map directly provides 3D orientation information of the object's surface, and the edge map reinforces the object's contour and internal structural boundaries, these two types of geometric prior information provide strong geometric shape guidance for the depth completion network, especially in areas with missing data, effectively preventing the generation of unreasonable or blurry geometric shapes. The derivation of the surface normal map based on polarization physics parameters can specifically include: based on the principle of physical polarization vision, there is a functional relationship between the degree of polarization map and the zenith angle of the surface normal. Using this functional relationship, combined with the degree of polarization map and known refractive index parameters, the zenith angle of the normal at each point on the surface can be estimated. Then, combining the normal azimuth angle directly obtained from the polarization angle map, the surface normal vector in three-dimensional space is calculated through coordinate transformation, forming the surface normal map.

[0040] Thus, by deriving geometric prior information through polarization physics parameters, the geometric constraints of depth completion can be significantly enhanced, while also improving the robustness of edge perception for transparent and reflective objects.

[0041] In some embodiments, the training and inference process of the conditional flow matching model includes: During training, a conditional flow matching model is trained using a multimodal conditional tensor. This model, for a given time step t and a noisy latent variable z_t constructed according to z_t = (1 - t) * y + t * ε, outputs a prediction vector to approximate the target vector field v_t. Here, y is the true complete depth map, ε is random noise, and v_t = y - ε. The corresponding loss function is the mean square error between the predicted vector and the target vector field. During inference, using the multimodal conditional tensor as a condition, starting from random noise z_1~N(0, I), the vector field predicted by the trained conditional flow matching model is iteratively updated through an ordinary differential equation numerical solver. Each iteration updates the noise latent variable according to the formula z_s = z_t + (t - s) * f_θ(z_t, t, C), where t and s are adjacent time steps, C is the multimodal conditional tensor, and f_θ is the conditional flow matching model. After a preset number of iterations N, the predicted depth map is obtained.

[0042] Among them, the conditional flow matching model is an advanced generative artificial intelligence model. Unlike the diffusion model, which denoises through hundreds of random iterations, conditional flow matching smoothly and deterministically transforms simple distributions (such as Gaussian noise) to complex target data distributions (such as true depth maps) by directly learning a deterministic vector field. It uses a multimodal conditional tensor as the "condition" to guide the entire generation process. This model has the advantages of stable training, fast convergence, and extremely high inference efficiency (typically requiring only about 10 deterministic iterations).

[0043] First, the multimodal conditional tensor C is used as input, and a random noise tensor z_1 with the same size as the completed depth map is sampled from a standard Gaussian distribution. The total number of iterations N=10 is set, and the steps from... arrive The time step sequence. Next, for the... The next iteration ( From 1 to ), perform the following sub-steps: (1) change the current noise tensor Current time step The multimodal conditional tensor C is input together with the pre-trained conditional flow matching model. The deterministic vector field f_θ(z_t, t, C) of the output is obtained. The direction of this vector field is uniquely determined by the multimodal conditional tensor C. It precisely indicates how to move the current noisy depth estimate z_t toward the direction of the complete depth map that conforms to physical reality (i.e., consistent with cues such as polarization and normals); (2) The state is updated according to the predicted vector field using a numerical solver (such as the Euler method). The update formula is z_s = z_t + (t - s) * f_θ(z_t, t, C). It is the next time step that is closer to 0 (e.g., if...) ,but The calculation, guided by the multimodal conditional tensor C, moves z_t a small step along the vector field direction to obtain an intermediate state z_s with less noise and a clearer structure; (3) assigns z_s to z_t, and... Assign to This prepares for the next iteration. Finally, repeat the above steps a total of After that, time Reaching state 0, we obtain the final state z_0. At this point, the initial random noise z_1, guided entirely by the multimodal conditional tensor C, has been completely and deterministically transformed into a high-fidelity depth map. z_0 is then directly used as the final output, thus completing the depth map. ,Right now: =z_0. This completed depth map (also known as a predicted depth map) retains the original values ​​in the valid areas of the original depth map, and generates accurate and reasonable depth values ​​in the areas that were originally missing or distorted based on physical information such as polarization.

[0044] Thus, the entire iterative optimization process is driven by a multimodal conditional tensor. The physical information in the multimodal conditional tensor, such as the degree of polarization, polarization angle, and normal, ensures that the completed geometry is physically reliable, fundamentally solving the problem of transparent and reflective objects being unable to be completed due to a lack of texture. At the same time, the iteration process requires fewer iterations, enabling efficient inference and improving processing efficiency.

[0045] S102, based on the second training dataset, the depth completion network and the grasping planning network are trained end-to-end using a joint loss function; wherein, the second training dataset includes multiple second samples, each second sample including the original polarization image, the original depth map containing missing or distorted regions, and the real complete depth map and real grasping pose label corresponding to the original depth map; the joint loss function is a weighted sum of the depth reconstruction loss function and the grasping task loss function; the grasping planning network is configured to perform grasping pose planning based on the predicted depth map to determine the corresponding grasping pose.

[0046] In the end-to-end training step, the gradient generated by the capture task loss function is backpropagated through the capture planning network to the predicted depth map output by the depth completion network, and further propagated to the parameters of the depth completion network, so that the depth completion network learns to generate a depth map that is more conducive to the capture planning network to achieve successful capture.

[0047] The second training dataset, built upon the first training dataset, adds a true grasping pose label to each sample—that is, a robot grasping pose verified as successful in that scenario. Therefore, each second sample is a quadruple (original polarization image, original depth map, true complete depth map, true grasping label). It should be noted that the second training dataset and the first training dataset can be datasets constructed separately for different scenarios.

[0048] The end-to-end joint training method means that the deep completion network (including the conditional flow matching model) and the grasping planning network (including the conditional flow matching model) are no longer trained in isolation, but are optimized simultaneously through a unified multi-task loss function and a gradient backpropagation path throughout the entire system. Failure signals in the grasping task can directly affect the parameters of the deep completion network, causing it to learn and generate geometries more favorable for grasping. First, the pre-trained deep completion network is connected to the initialized grasping planning network, and a joint loss function is defined. L total :,in L depth For deep reconstruction losses, The loss function is used for the grasping task (e.g., binary cross-entropy loss). Next, for each sample in the second training dataset, the depth completion network generates a predicted depth map, and the grasping planning network plans and scores candidate grasping poses based on this predicted depth map. Furthermore, during prediction, the grasping pose planned by the grasping planning network can be used to generate and output robot grasping instructions, which the robot can then execute to grasp the object. It should be noted that the specific implementation process of planning the object pose and outputting robot grasping instructions can refer to existing technologies and will not be elaborated here.

[0049] In the actual training process, a large number of data pairs containing transparent and / or reflective objects and their true depth maps can be collected first. Then, only the depth reconstruction loss is used. L depth (e.g., Smooth L1 Loss) is used to train the conditional flow matching model v_θ. The goal of this stage is to enable the conditional flow matching model to generate geometrically accurate depth maps under the guidance of multimodal conditions such as polarization parameters. Next, a small dataset containing successful and failed capture annotations is used to construct the total loss function. L total During this phase, the parameters of both the conditional flow matching model v_θ and the differentiable grasping evaluation model are updated simultaneously. The grasping loss... L grasp The gradient will propagate back along the differentiable path, all the way back to the conditional flow matching model v_θ, so that v_θ not only learns to reconstruct accurate depth, but also learns to generate depth map features that enable the differentiable grasping evaluation model to give high scores, thereby achieving task-driven perception optimization.

[0050] In some embodiments, the grasping planning network includes a differentiable grasping evaluation model; the grasping planning network is configured to: The predicted depth map is back-projected into a 3D point cloud through a differentiable operation; Generate at least one candidate grasping pose based on 3D point cloud; A differentiable grasping evaluation model is used to score the quality of each candidate grasping pose, and the candidate grasping pose with the highest score is taken as the final grasping pose. The differentiable grasping evaluation model is configured as follows: for each candidate grasping pose, determine the spatial symbolic distance field corresponding to the robot gripper in three-dimensional space; calculate the interaction information between the three-dimensional point cloud and the spatial symbolic distance field; and calculate the grasping quality score of each candidate grasping pose based on the interaction information.

[0051] The grasping planning network can include a differentiable grasping evaluation model. This differentiable grasping evaluation model refers to a computational model that can score the quality of candidate grasping poses, and the entire scoring process (from input pose and point cloud to output score) is mathematically differentiable. It typically includes a differentiable spatial symbolic distance field, differentiable interactive computation, and a differentiable evaluation function. By substituting the grasping pose parameters into the differentiable spatial symbolic distance field, the specific geometric configuration of the robot gripper in space under that pose can be calculated differentiably.

[0052] Specifically, firstly, the depth map can be completed. y hat A 3D point cloud P is generated by backprojection using the camera intrinsic parameter matrix and differentiable operations. Then, a grasping planning network based on PointNet++ is used to predict multiple candidate grasping poses using the point cloud P as input. G i Next, a differentiable grasping evaluation model is used to score the quality of each candidate grasping pose, and the candidate grasping pose with the highest score is selected as the final grasping pose.

[0053] For each candidate pose, the following operations can be performed using a differentiable grasping evaluation model: First, based on the rotation matrix and translation parameters in the candidate pose, the predefined gripper spatial symbolic distance field is transformed to the world coordinate system to obtain the gripper spatial symbolic distance field under that candidate pose. Then, the set of distance values ​​for all points in the 3D point cloud within that gripper spatial symbolic distance field is calculated, serving as the interaction information between the 3D point cloud and the spatial symbolic distance field. Finally, a lightweight neural network is used as a differentiable evaluation function, taking the statistical characteristics of the distance value set (such as mean, variance, and quantiles) as input, and outputting a grasping quality score between 0 and 1 as the quality score result for that candidate grasping pose.

[0054] In some embodiments, the grasping task loss function is calculated based on the grasping quality score output by a differentiable grasping evaluation model; the gradient backpropagation process includes: Calculate the loss function for the crawling task Parameters of the planning network for extraction gradient And according to the gradient Update the parameters of the capture planning network ; Calculate the loss function for the crawling task For the predicted depth map gradient ,gradient It is used to characterize the impact of changes in each pixel value in the depth map on the success rate of capture; Calculate the loss function for the crawling task based on the chain rule. Parameters of the deep completion network Total gradient The parameters of the depth-complete network are updated based on the total gradient. .

[0055] During the joint training phase, the total losses were... L total The gradients are simultaneously backpropagated to both the conditional flow matching model and the differentiable grasping evaluation model. First, the gradients within the grasping planning network are updated. Specifically, based on the grasping score... s i Compared to real tag crawling l i The binary cross-entropy loss between them is used to calculate the loss function. L grasp Scraping score s i partial derivatives L grasp / s i This gradient is further propagated to its network parameters. , obtain gradient Finally, the gradient is used. The process involves updating all trainable parameters of the grasping planning network, which teaches it to provide more accurate grasping scores based on the depth map. Then, gradients are transferred from the grasping planning network to the depth map, specifically through a differentiable projection transformation, converting point cloud-level gradients to pixel-level gradients in the depth map. This step calculates the impact of changes in each pixel value in the depth map on the success rate of the capture. For example, capturing pixels near the contact point will yield a larger gradient value. Finally, the gradient is passed from the depth map to the depth completion network, and through a conditional flow matching model (such as the U-Net network), it eventually reaches all the parameters of the depth completion network. According to the chain rule, the gradient formula for the grasping loss with respect to the parameters of the depth completion network can be expressed as: This allows it to generate a depth map that is more advantageous for crawling tasks. Furthermore, Ldepth Gradient backpropagation will also be performed, which means directly backpropagating along the depth map calculated by the depth completion network to update the same network parameters. Therefore, the total update gradient of a deep completion network can be viewed as... L total / θ d = λ1 ×( L depth / θ d ) + λ2 × ( L grasp / θ d ).

[0056] In summary, the robot control model training method provided in the above embodiments, through an innovative two-stage joint training strategy, tightly integrates depth perception with grasping tasks. This allows the depth completion network to directly receive optimization signals from the grasping task and learn to generate the depth representation most favorable for grasping planning. This achieves task-level collaborative optimization of perception and grasping, improving the model's grasping success rate and efficiency. Simultaneously, the introduction of polarization vision modality acquires surface physical properties that remain valid even when visual information is missing or distorted, providing irreplaceable physical constraints for depth recovery. This enables the trained model to accurately grasp transparent and reflective objects.

[0057] Based on the same inventive concept as the foregoing embodiments, this embodiment of the invention provides a computing device, such as... Figure 2 As shown, the device includes: a processor 310 and a memory 311 storing a computer program; wherein, Figure 2 The processor 310 shown in the diagram does not indicate that there is only one processor 310, but only indicates the positional relationship of the processor 310 relative to other devices. In practical applications, there can be one or more processors 310; similarly, Figure 2 The memory 311 shown in the diagram has the same meaning, that is, it is only used to indicate the positional relationship of memory 311 relative to other devices. In practical applications, there can be one or more memories 311. When the processor 310 runs the computer program, it implements the training method for the robot control model applied to the above-mentioned computing device.

[0058] The device may also include at least one network interface 312. The various components of the device are coupled together via a bus system 313. It is understood that the bus system 313 is used to implement communication between these components. In addition to a data bus, the bus system 313 also includes a power bus, a control bus, and a status signal bus. However, for clarity, in… Figure 2 The general designated all buses as Bus System 313.

[0059] The memory 311 can be volatile memory or non-volatile memory, or both. Non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), ferromagnetic random access memory (FRAM), flash memory, magnetic surface memory, optical disc, or compact disc read-only memory (CD-ROM); magnetic surface memory can be disk storage or magnetic tape storage. Volatile memory can be random access memory (RAM), used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Synchronous Static Random Access Memory (SSRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDRSDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), SyncLink Dynamic Random Access Memory (SLDRAM), and Direct Rambus Random Access Memory (DRRAM).The memory 311 described in the embodiments of the present invention is intended to include, but is not limited to, these and any other suitable types of memory.

[0060] The memory 311 in this embodiment of the invention is used to store various types of data to support the operation of the device. Examples of this data include: any computer programs used to operate on the device, such as operating systems and applications; contact data; phonebook data; messages; pictures; videos, etc. The operating system includes various system programs, such as the framework layer, core library layer, driver layer, etc., used to implement various basic services and handle hardware-based tasks. Applications can include various applications, such as media players, browsers, etc., used to implement various application services. Here, the program implementing the method of this embodiment of the invention can be included in the application.

[0061] Based on the same inventive concept as the foregoing embodiments, this embodiment also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the training method for the robot control model as described above.

[0062] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0063] In this document, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, which includes not only the elements listed but also other elements not expressly listed.

[0064] The above description is merely a specific embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in the present invention should be included within the scope of protection of the present invention. Therefore, the scope of protection of the present invention should be determined by the scope of the claims.

Claims

1. A method for training a robot control model, characterized in that, The robot control model includes a depth completion network and a grasping planning network, with the output of the depth completion network connected to the input of the grasping planning network; the method includes: The depth completion network is trained using a depth reconstruction loss function based on a first training dataset. The first training dataset includes multiple first samples, each of which includes an original polarization image, an original depth map containing missing or distorted regions, and a true complete depth map corresponding to the original depth map. The depth completion network is configured to fuse the information carried by the original polarization image and the original depth map to generate a predicted depth map. Based on a second training dataset, the depth completion network and the grasping planning network are trained end-to-end using a joint loss function. The second training dataset includes multiple second samples, each of which includes an original polarization image, an original depth map containing missing or distorted regions, and a true complete depth map and a true grasping pose label corresponding to the original depth map. The joint loss function is a weighted sum of the depth reconstruction loss function and the grasping task loss function. The grasping planning network is configured to plan the grasping pose based on the predicted depth map to determine the corresponding grasping pose. In the end-to-end training step, the gradient generated by the grasping task loss function is backpropagated through the grasping planning network to the predicted depth map output by the depth completion network, and further propagated to the parameters of the depth completion network.

2. The method according to claim 1, characterized in that, The depth completion network includes a conditional flow matching model; the input of the depth completion network is a multimodal conditional tensor generated based on the original polarization image and the original depth map; The depth completion network is configured to perform depth completion using the conditional flow matching model, based on the multimodal conditional tensor, to generate the predicted depth map.

3. The method according to claim 2, characterized in that, The multimodal conditional tensor is generated through the following steps: Polarization physical parameters are extracted from the original polarization image, and geometric prior information is derived based on the polarization physical parameters; The polarization physical parameters, the geometric prior information, and the original depth map are fused to generate a multimodal conditional tensor for guiding depth information recovery.

4. The method according to claim 3, characterized in that, The polarization physical parameters include an intensity image, a degree of polarization map, and a polarization angle map; the extraction of polarization physical parameters from the original polarization image includes: Calculate the Stokes vector corresponding to the original polarization image; Based on the Stokes vector, the intensity image, the degree of polarization map, and the angle of polarization map of the original polarization image are extracted.

5. The method according to claim 4, characterized in that, The geometric prior information includes at least one of the following: surface normal map, edge map; the derivation of geometric prior information based on the polarization physical parameters includes at least one of the following: Based on the principle of polarization vision, the surface normal map is determined using the degree of polarization map and the angle of polarization map; Edge detection is performed on the intensity image and the polarization map respectively, and the edge detection results are fused to generate the edge map.

6. The method according to claim 2, characterized in that, The training and inference process of the conditional flow matching model includes: During training, the conditional flow matching model is trained using the multimodal conditional tensor as a condition, such that for a given time step t and a noise latent variable z_t constructed according to z_t = (1 - t) * y + t * ε, the conditional flow matching model outputs a prediction vector to approximate the target vector field v_t; where y is the true complete depth map, ε is random noise, and v_t = y - ε; the corresponding loss function is the mean square error between the prediction vector and the target vector field. During inference, the multimodal conditional tensor is used as a condition, starting from random noise z_1~N(0, I), and through an ordinary differential equation numerical solver, the vector field predicted by the trained conditional flow matching model is iteratively updated. In each iteration, the noise latent variable is updated according to the formula z_s = z_t + (t - s) * f_θ(z_t, t, C), where t and s are adjacent time steps, C is the multimodal conditional tensor, and f_θ is the conditional flow matching model. After a preset number of iterations N, the predicted depth map is obtained.

7. The method according to claim 1, characterized in that, The grasping planning network includes a differentiable grasping evaluation model; the grasping planning network is configured as follows: The predicted depth map is back-projected into a three-dimensional point cloud through a differentiable operation; At least one candidate grasping pose is generated based on the 3D point cloud; The differentiable grasping evaluation model is used to score the quality of each candidate grasping pose, and the candidate grasping pose with the highest score is taken as the final grasping pose. The differentiable grasping evaluation model is configured to: for each candidate grasping pose, determine the spatial symbolic distance field corresponding to the robot gripper in three-dimensional space; calculate the interaction information between the three-dimensional point cloud and the spatial symbolic distance field; and calculate the grasping quality score for each candidate grasping pose based on the interaction information.

8. The method according to claim 7, characterized in that, The crawling task loss function is calculated based on the crawling quality score output by the differentiable crawling evaluation model; the gradient backpropagation process includes: Calculate the loss function of the crawling task The parameters of the grasping planning network gradient And according to the gradient Update the parameters of the crawling planning network. ; Calculate the loss function of the crawling task For the predicted depth map gradient The gradient It is used to characterize the impact of changes in each pixel value in the depth map on the success rate of capture; Calculate the crawling task loss function according to the chain rule. The parameters of the deep completion network Total gradient And update the parameters of the depth completion network according to the total gradient. .

9. A computing device, characterized in that, include: A storage medium and a controller, wherein a computer program is stored on the storage medium, and the computer program, when executed by the controller, implements the training method for the robot control model as described in any one of claims 1 to 8.

10. A computer-readable storage medium, characterized in that, The system contains a computer program that, when executed by a processor, implements the training method for the robot control model as described in any one of claims 1 to 8.

Images