Method, device, equipment, medium and product for reconstructing high-resolution visual point cloud

Abstract

The application provides a high-resolution visual point cloud reconstruction method, device, equipment, medium and product, and relates to the technical field of three-dimensional point cloud data processing. The method comprises the following steps: obtaining initial acquisition data of each acquisition point according to low-resolution visual point cloud data, tactile point cloud data and tactile pressure values of an acquisition area; inputting each initial acquisition data into a super-resolution model to obtain high-resolution visual point cloud data of the acquisition area and grabbing reference information of each predicted point output by the super-resolution model. The application realizes feature enhancement of the low-resolution visual point cloud data, global coverage and data complementation of local details through the tactile point cloud data and the tactile pressure values, which is beneficial to improving the accuracy of the reconstructed high-resolution visual point cloud data. Meanwhile, the grabbing reference information of each predicted point is determined, which provides a decision basis for the robot to perform tasks such as grabbing and obstacle avoidance, and is beneficial to improving the practicability of the high-resolution visual point cloud data.

Description

Technical Field

[0001] This application relates to the field of three-dimensional point cloud data processing technology, and in particular to a method, apparatus, equipment, medium and product for reconstructing high-resolution visual point clouds. Background Technology

[0002] In the field of robotics, especially in applications such as automated grasping, assembly, and human-robot interaction, accurate perception of 3D objects is crucial. 3D point clouds, as the primary data form describing the geometric information of an object's surface, directly impact the success rate of subsequent robot task planning and execution. However, limited by cost, acquisition speed, and environmental factors, real-time point cloud data acquired by robot vision systems (such as depth cameras) is often sparse and low-resolution, resulting in the loss of significant amounts of fine geometric details on the object's surface (such as edges, holes, and curvature variations). This lack of detail hinders the robot's accurate object recognition, pose estimation, and stable grasping point planning, thereby reducing the reliability and efficiency of the entire automated system. Therefore, it is necessary to reconstruct high-precision 3D point clouds from low-resolution visual point clouds.

[0003] Existing high-resolution point cloud reconstruction methods include incorporating multimodal information for reconstruction assistance. However, existing methods for fusing multimodal information and low-resolution visual point cloud data are rather crude. Examples include simple feature stitching or homogeneous iterative fusion. This simplistic fusion fails to effectively handle the complementarity and differences between multimodal information and visual point cloud data, resulting in the dilution of key details or interference from redundant information.

[0004] In the existing high-resolution point cloud reconstruction process, the fusion of multimodal information and low-resolution visual point cloud data is not sufficient, resulting in low accuracy of the reconstructed high-resolution visual point cloud data. Summary of the Invention

[0005] This application provides a method, apparatus, device, medium, and product for reconstructing high-resolution visual point clouds, in order to solve the defect of low accuracy of reconstructed high-resolution visual point cloud data in the prior art, and to improve the accuracy of reconstructed high-resolution visual point cloud data.

[0006] In a first aspect, this application provides a method for reconstructing high-resolution visual point clouds, comprising: Based on the low-resolution visual point cloud data, tactile point cloud data and tactile pressure values ​​of the acquisition area, the initial acquisition data of each acquisition point is obtained; Each initial data acquisition is input into the super-resolution model to obtain high-resolution visual point cloud data of the acquisition area output by the super-resolution model, and capture reference information for each prediction point.

[0007] In one embodiment, the super-resolution model includes a feature extraction module, a fusion module, an upsampling module, and a capture prediction module. The super-resolution model is used for: Based on the feature extraction module, feature modulation and multi-layer sensing coding are performed on each initial data collection point to obtain the encoded features of each collection point. The fusion module performs weighted fusion of the encoded features to obtain the fused features. Based on the upsampling module, the fused features are compressed, extracted, expanded, and rearranged to obtain high-resolution visual point cloud data; The grasping prediction module extracts and activates features from high-resolution visual point cloud data to generate grasping reference information for each prediction point.

[0008] In one embodiment, the feature extraction module includes multiple stacked pressure-guided graph convolutional blocks, and the feature extraction module is used for: Based on the output of the previous pressure-guided graph convolutional block and the processing result of the current pressure-guided graph convolutional block, obtain the output of the current pressure-guided graph convolutional block; if the current pressure-guided graph convolutional block is the first pressure-guided graph convolutional block, then based on the processing result of the initial acquired data, obtain the output of the current pressure-guided graph convolutional block. Based on the output of the last pressure-guided graph convolutional block, the encoded features are obtained.

[0009] In one embodiment, the processing result of the current pressure-guided graph convolutional block is obtained based on the following method: Based on the output of the previous pressure-guided graph convolutional block, the aggregated features and tactile pressure values ​​of each sampling point are obtained; if the current pressure-guided graph convolutional block is the first pressure-guided graph convolutional block, then the aggregated features are obtained based on the aggregation results of low-resolution visual point cloud data and tactile point cloud data. Each aggregated feature is modulated based on a modulation factor to obtain each modulated feature; the modulation factor is generated based on tactile pressure values. When the tactile pressure value is greater than or equal to the pressure threshold, the modulated feature corresponding to the tactile pressure value is used as the sensitive feature. The sensitive feature is then subjected to enhanced multilayer perceptual coding to obtain the processing result of the graph convolution block guided by the current pressure. When the tactile pressure value is less than the pressure threshold, the modulated feature corresponding to the tactile pressure value is used as a general feature. The general feature is then subjected to general multilayer perceptual encoding to obtain the processing result of the graph convolution block guided by the current pressure.

[0010] In one embodiment, the fusion module is used for: Determine the contact frequency and uncertainty score for each collection point; The weights of each encoded feature are determined based on contact frequency, uncertainty score, and encoded features. Based on the weights of each encoded feature, multi-layer perceptual fusion is performed on each encoded feature to obtain fused features.

[0011] In one embodiment, the super-resolution model is trained as follows: Based on the actual capture reference information and actual high-resolution visual point cloud data of each sample collection point, the initial sample collection data of each sample collection point is labeled to obtain training samples. The super-resolution model is obtained by training the pre-defined model based on the training samples and the loss function. The loss function is determined based on the chamfer distance of the global point cloud, the chamfer distance of the stable sample collection points, the chamfer distance of the dangerous sample collection points, and the loss of the grasping reference information; the chamfer distance is determined based on the error between the actual high-resolution visual point cloud data and the predicted high-resolution visual point cloud data output by the preset model.

[0012] In one embodiment, stable sample collection points and hazardous sample collection points are determined based on the following method: Obtain at least one feature value of the neighborhood points of the target sample collection point, whereby the at least one feature value characterizes the degree of dispersion of the neighborhood points in at least one principal direction; the target sample collection point can be any sample collection point. Calculate the curvature and average curvature of the target sample collection points based on at least one feature value; When the curvature of the target sample collection point is greater than the curvature threshold, and the average curvature of the target sample collection point is greater than the average curvature threshold, the target sample collection point is determined to be a dangerous sample collection point. When the curvature of the target sample collection point is less than or equal to the curvature threshold, and the average curvature of the target sample collection point is less than or equal to the average curvature threshold, the target sample collection point is determined to be a stable sample collection point.

[0013] Secondly, this application provides a high-resolution visual point cloud reconstruction apparatus, comprising: The acquisition module is used to obtain the initial acquisition data of each acquisition point based on the low-resolution visual point cloud data, tactile point cloud data and tactile pressure value of the acquisition area; The reconstruction module is used to input the initial acquisition data into the super-resolution model, obtain the high-resolution visual point cloud data of the acquisition area output by the super-resolution model, and capture reference information of each prediction point.

[0014] Thirdly, this application also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement any of the above-described methods for reconstructing high-resolution visual point clouds.

[0015] Fourthly, this application also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the above-described methods for reconstructing high-resolution visual point clouds.

[0016] Fifthly, this application also provides a computer program product, including a computer program that, when executed by a processor, implements any of the above-described methods for reconstructing high-resolution visual point clouds.

[0017] This application provides a method, apparatus, device, medium, and product for reconstructing high-resolution visual point clouds. It acquires initial acquisition data for each acquisition point based on low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​of the acquisition area. This initial acquisition data is then input into a super-resolution model to obtain high-resolution visual point cloud data of the acquisition area output by the super-resolution model, along with grasping reference information for each predicted point. This application utilizes tactile point cloud data and tactile pressure values ​​to enhance the features of low-resolution visual point cloud data, as well as to achieve data complementarity between full coverage and local details, thereby improving the accuracy of the reconstructed high-resolution visual point cloud data. Simultaneously, determining the grasping reference information for each predicted point provides a decision-making basis for the robot to perform tasks such as grasping and obstacle avoidance, thus enhancing the practicality of the high-resolution visual point cloud data. Attached Figure Description

[0018] To more clearly illustrate the technical solutions in this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0019] Figure 1 This is one of the flowcharts illustrating the high-resolution visual point cloud reconstruction method provided in this application.

[0020] Figure 2 This is the second flowchart illustrating the high-resolution visual point cloud reconstruction method provided in this application.

[0021] Figure 3 This is a schematic diagram of the process for acquiring high-resolution visual point cloud data and capturing reference information provided in this application.

[0022] Figure 4This is a schematic diagram of the feature extraction module provided in this application.

[0023] Figure 5 This is a flowchart illustrating the process of obtaining preprocessed fusion features provided in this application.

[0024] Figure 6 This is a schematic diagram of the process for obtaining high-resolution visual point cloud data provided in this application.

[0025] Figure 7 This is a schematic diagram of the high-resolution visual point cloud reconstruction device provided in this application.

[0026] Figure 8 This is a schematic diagram of the structure of the electronic device provided in this application. Detailed Implementation

[0027] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0028] The demand for high-precision 3D point cloud data (high-resolution visual point cloud data) is crucial for robot training environments, as this data directly affects the robot's performance and effectiveness in tasks such as object manipulation simulation and dynamic scene modeling. However, in practical applications, due to limitations in scanning equipment technology, constraints in acquisition time, and interference from complex environments, the acquired 3D point cloud data often suffers from problems such as sparsity, unevenness, and noise. These issues severely restrict the efficiency and quality of robot training.

[0029] This application covers high-resolution point cloud reconstruction of equipment and environments in robot training fields. The core objective of this application is to address the problems of insufficient accuracy in point cloud acquisition and the disconnect between traditional super-resolution results and grasping tasks caused by factors such as equipment and environment in robot training fields. By fusing global low-resolution visual point cloud data, multi-regional tactile point cloud data, and tactile pressure values, it achieves the synergy between "reconstruction of high-resolution visual point cloud data" and "perception of the grasping area," providing high-precision data support for robot grasping tasks.

[0030] The overall process of high-resolution point cloud reconstruction in this application is as follows: First, global low-resolution visual point cloud data and multiple sets of local tactile point cloud data are collected. Second, high-resolution visual point cloud data is generated through a grasping-guided super-resolution network, focusing on enhancing the details of the touch-sensitive area. Finally, geometric features are calculated based on the high-resolution visual point cloud data, the "graspingability" of the region is quantified, and the annotation results (grasping reference information) are output.

[0031] The following is combined Figures 1-8 This application describes the method, apparatus, medium, and product for reconstructing high-resolution visual point clouds.

[0032] Figure 1 This is one of the flowcharts illustrating the high-resolution visual point cloud reconstruction method provided in this application. Figure 2 This is the second flowchart illustrating the high-resolution visual point cloud reconstruction method provided in this application, as shown below. Figure 1 and Figure 2 As shown, the high-resolution visual point cloud reconstruction method includes steps S100 to S200, and the specific steps are as follows.

[0033] S100: Based on the low-resolution visual point cloud data, tactile point cloud data and tactile pressure value of the acquisition area, obtain the initial acquisition data of each acquisition point.

[0034] Since tactile point cloud data can provide high-density and high-precision local surface information, this application uses tactile point cloud data and tactile pressure values ​​to enhance low-resolution visual point cloud data.

[0035] A 3D scanner is mounted on the robot's body. This scanner uses non-contact scanning technology, employing devices such as LiDAR, structured light, or depth cameras to scan the surface of the target object and record the spatial position and geometry of each scan point, generating 3D point cloud data. In the robot training area, the 3D scanner is used to acquire low-resolution visual point cloud data of the target object, quickly capturing its global geometric information and providing a foundation for subsequent detail additions.

[0036] When a 3D scanner is working, it uses laser beams, illumination, or structured light to illuminate the surface of a target object, capturing the reflected light signals to measure distance information on the object's surface. Through multiple scans, the 3D scanner can accurately record point cloud data at different locations. Finally, the 3D scanner converts this distance information into 3D coordinates, obtaining low-resolution visual point cloud data of the target object. During the 3D scanning process, because the overall point cloud scanning process requires the acquisition of a large area of ​​the target object, a certain scanning accuracy and sampling frequency need to be set to balance scanning time and point cloud quality. Therefore, the low-resolution visual point cloud data acquired by the 3D scanner has a relatively low density, large distances between points, and cannot capture fine local details; it is suitable for providing the general structure and outline of the target object.

[0037] A tactile sensor is installed at the end of the robot's robotic arm. The tactile sensor is used to repeatedly touch the surface of a target object to acquire high-resolution tactile point cloud data and tactile pressure values ​​of multiple local areas of the object. The tactile sensor is used to accurately capture the geometric details of the target object's surface, compensating for the limitations of low-resolution visual point cloud data in capturing local fine structures.

[0038] Specifically, tactile sensors include the Digit360 sensor. The Digit360 sensor is primarily used to acquire minute details of the surface of a target object. Its working principle relies on an elastic colloid within the sensor and an integrated camera system. When the tactile sensor comes into contact with the surface of the target object, the elastic colloid deforms. The camera system inside the sensor monitors this deformation, capturing local geometric changes on the target object's surface. The Digit360 sensor utilizes visual-tactile sensing technology, achieving micron-level resolution to accurately perceive the microscopic features of the target object's surface. It can not only perceive the shape of the target object and capture subtle surface textures, minute bumps, and other structural features, but also detect pressure information. This localized information is used to supplement details in low-resolution visual point cloud data.

[0039] In acquiring tactile point cloud data and tactile pressure values, the robotic arm is responsible for precisely positioning the tactile sensor to different parts of the target object's surface for touch. Equipped with a high-precision motion control system, the robotic arm can accurately move the tactile sensor along a predetermined path and angle, ensuring that each touch covers different areas of the target object's surface. Through multiple touches, the tactile sensor gradually collects high-resolution tactile point cloud data from various key areas of the target object's surface. The data from each touch can meticulously capture the microscopic details of the object's surface, especially those that cannot be precisely captured by a 3D scanner, such as fine textures and minute structures.

[0040] Based on the low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​of the acquisition area, the initial acquisition data of each acquisition point in the acquisition area is obtained.

[0041] S200: Input each initial acquisition data into the super-resolution model to obtain high-resolution visual point cloud data of the acquisition area output by the super-resolution model, and capture reference information of each prediction point.

[0042] The capture reference information includes a capture importance score (a scalar value). The capture importance score characterizes the likelihood that each predicted point in each high-resolution visual point cloud dataset will be a capture point. A higher capture importance score indicates that the predicted point is more likely to be a capture point. A lower capture importance score indicates that the predicted point is less likely to be a capture point.

[0043] A pre-defined model is trained to obtain a super-resolution model. Initial sample data for each sampling point is acquired based on low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​from the sample collection area. The initial sample data for each sampling point is labeled using actual grasping reference information and actual high-resolution visual point cloud data, resulting in labeled training samples. The pre-defined model is then trained using these labeled training samples to obtain the super-resolution model.

[0044] The initial data collected from all sampling points is input into the super-resolution model. The super-resolution model extracts and fuses features from the tactile point cloud data, low-resolution visual point cloud data, and tactile pressure values ​​to generate high-resolution visual point cloud data, as well as grasping reference information for each prediction point in the high-resolution visual point cloud data.

[0045] Optionally, the capture reference information can be the capture difficulty level (discrete value) of each prediction point, or the capture mask of each prediction point. The capture mask includes the mask of the captureable region and the mask of the non-captureable region.

[0046] The high-resolution visual point cloud reconstruction method provided in this application obtains initial acquisition data for each acquisition point based on low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​of the acquisition area. The initial acquisition data is then input into a super-resolution model to obtain high-resolution visual point cloud data of the acquisition area output by the super-resolution model, along with grasping reference information for each predicted point. This application achieves feature enhancement of low-resolution visual point cloud data and data complementarity between full coverage and local details through tactile point cloud data and tactile pressure values, which is beneficial for improving the accuracy of the reconstructed high-resolution visual point cloud data. Simultaneously, determining the grasping reference information for each predicted point provides a decision-making basis for the robot to perform tasks such as grasping and obstacle avoidance, thus improving the practicality of the high-resolution visual point cloud data.

[0047] This application acquires global low-resolution visual point cloud data using a 3D scanner, ensuring the overall structural integrity of the target object's surface data. Combined with active multi-regional touch detection by a tactile sensor at the end of a robotic arm, it collects local high-resolution tactile point cloud data and tactile pressure values. This achieves data complementarity of "full coverage + local detail," overcoming the problem of missing details in single-modal point clouds during training.

[0048] Based on the above embodiments, the super-resolution model includes a feature extraction module, a fusion module, an upsampling module, and a capture prediction module. The super-resolution model is used for: Based on the feature extraction module, feature modulation and multi-layer sensing coding are performed on each initial data collection point to obtain the encoded features of each collection point. The fusion module performs weighted fusion of the encoded features to obtain the fused features. Based on the upsampling module, the fused features are compressed, extracted, expanded, and rearranged to obtain high-resolution visual point cloud data; The grasping prediction module extracts and activates features from high-resolution visual point cloud data to generate grasping reference information for each prediction point.

[0049] like Figure 3 As shown, the super-resolution model includes a feature extraction module, a fusion module, an upsampling module, and a capture prediction module.

[0050] like Figure 4 As shown, the feature extraction module includes a multilayer perceptron (MLP) and at least one (e.g., L) stacked pressure-guided graph convolutional blocks. The feature extraction module first performs initial feature embedding and concatenation operations on the initial acquired data (including low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values) using the MLP to obtain initial feature vectors for each acquisition point. The initial feature vectors include the visual point cloud features, tactile point cloud features, and tactile pressure values ​​for each acquisition point. The L stacked pressure-guided graph convolutional blocks modulate the initial feature vectors for each acquisition point and perform MLP encoding to obtain the encoded features for each acquisition point. .

[0051] The feature extraction module sends the encoded features from each acquisition point to the fusion module. The fusion module determines the weight of each encoded feature. Based on these weights, the fusion module processes each encoded feature... Weighted fusion is performed to obtain fusion features. .

[0052] The fusion module sends the fused features to the upsampling module. The upsampling module compresses, extracts, expands, and rearranges the fused features to obtain high-resolution visual point cloud data.

[0053] The upsampling module includes a preprocessing unit and a feature rearrangement unit. The preprocessing unit is used to compress and extract features from the fused features. The preprocessing unit includes a bottleneck layer, a dense graph convolutional unit, and a global pooling layer. The bottleneck layer is used to compress the dimensionality of the fused features to reduce the computational cost of subsequent networks. The compressed fused features are then further processed by the dense graph convolutional unit and the global pooling layer to obtain the preprocessed fused features. The dense graph convolutional unit includes a multi-layer dense graph convolutional network (DenseGCN). Figure 5 As shown, the dense graph convolutional unit (DPU) comprises three layers of dense graph convolutional networks (DPU 1, DPU 2, and DPU 3). In the DPU, the output of each layer is passed not only to the next layer but also to all subsequent layers. This inter-layer connection mechanism allows the DPU to retain the original input information at each layer and accumulate features layer by layer as the network progresses, enhancing feature transfer and reuse. The preprocessing unit outputs the preprocessed fused features and sends them to the feature rearrangement unit.

[0054] like Figure 6 As shown, the feature rearrangement unit expands and rearranges the preprocessed fused features to obtain rearranged features. The feature rearrangement unit then remaps the rearranged features to three-dimensional space to obtain high-resolution visual point cloud data. For example, if the preprocessed fused features are N×C, their dimensions are expanded by a factor of r (r is the upsampling factor) through convolution, and then rearrangement is performed to obtain rN×C rearranged features. Through coordinate reconstruction, the rN×C rearranged features are remapped to three-dimensional space to obtain high-resolution visual point cloud data. , The formula for calculating the rearrangement feature is as follows.

[0055] ; in, The fusion features are the results of preprocessing. , The weight matrix is ​​a learnable matrix. These are the learnable initial parameter values. This is a rearrangement feature. .

[0056] Specifically, computing high-resolution visual point cloud data includes the following steps: (1) Channel expansion. Using a learnable weight matrix. and learnable initial parameter values The dimensions of the preprocessed fusion features are changed from Expand to (2) Rearrangement. The channel-expanded fusion features are rearranged into The rearrangement features are obtained. (3) Reconstruction. The rearranged features are remapped back to the three-dimensional coordinate space through a multilayer perceptron (MLP) to obtain high-resolution visual point cloud data. .

[0057] The upsampling module sends the obtained high-resolution visual point cloud data to the grasping prediction module. The grasping prediction module outputs grasping reference information for each predicted point in the high-resolution visual point cloud data. The grasping reference information includes a grasping importance score; this embodiment uses the grasping importance score as an example. The grasping importance score indicates the importance or probability of the predicted point being grasped. For example, the final output of the super-resolution model is... ,in, , and For high-resolution visual point cloud data, To capture the importance score.

[0058] Optionally, the grasping prediction module includes an input layer, a fully connected layer, an activation function, and an output layer. The input layer receives high-resolution visual point cloud data. The fully connected layer has 4 neurons, with the first three neurons (corresponding to...) , and No activation function is used. The fourth neuron (corresponding to) The output values ​​are normalized to the range [0, 1] using an activation function (e.g., sigmoid). Furthermore, after prediction is complete, the values ​​for each predicted point can be... Visualize the values. For example, Values ​​are mapped to colors (for example, if the capture importance score is greater than a set threshold, the predicted point is rendered in green; if the capture importance score is less than or equal to the set threshold, the predicted point is rendered in red), thus visually showing which areas in the predicted point cloud are easy to capture.

[0059] This application achieves deep feature extraction from low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​by performing feature modulation and multi-layer perceptual coding on the initial acquired data. Through weighted fusion of the encoded features, it achieves the fusion of multimodal visual point cloud data, tactile point cloud data, and tactile pressure values. By compressing, extracting, expanding, and rearranging the fused features, it achieves the reconstruction of high-resolution visual point cloud data. By generating grasping reference information for predicted points, the reconstructed high-resolution visual point cloud data not only pursues geometric accuracy but also achieves deep adaptation with downstream grasping planning, obstacle avoidance, and other tasks through graspability annotation (grasping reference information), enhancing the practicality of high-resolution visual point cloud data in robot training.

[0060] This application integrates global low-resolution visual point cloud data acquired by a robotic 3D scanner with local high-precision tactile point cloud data collected by a tactile sensor at the end of a robotic arm. By combining this with dynamic weighted feature fusion technology, it fully explores the subtle differences and contributions of various tactile information, achieving adaptive weighted fusion of multimodal features and effectively improving the completeness and accuracy of the point cloud data. Simultaneously, by deeply binding the super-resolution results (high-resolution visual point cloud data) with the grasping perception task, it outputs an enhanced point cloud with grasping reference information, providing high-quality 3D environmental data support for tasks such as motion planning, object grasping, and dynamic scene modeling in robot training environments.

[0061] This application covers key areas of an object by collecting complementary tactile point cloud data from different regions; it designs a cross-modal fusion module that adaptively adjusts the fusion weights (weights of encoded features) based on tactile pressure values ​​and spatial distribution (such as high curvature regions) to enhance the contribution of key details; and it adds a grasping prediction module that outputs a "graspability" scalar value (grasping reference information) for each predicted point while generating high-resolution visual point cloud data, directly serving the robot's grasping training.

[0062] Based on the above embodiments, the feature extraction module includes multiple stacked pressure-guided graph convolutional blocks, and the feature extraction module is used for: Based on the output of the previous pressure-guided graph convolutional block and the processing result of the current pressure-guided graph convolutional block, obtain the output of the current pressure-guided graph convolutional block; if the current pressure-guided graph convolutional block is the first pressure-guided graph convolutional block, then based on the processing result of the initial acquired data, obtain the output of the current pressure-guided graph convolutional block. Based on the output of the last pressure-guided graph convolutional block, the encoded features are obtained.

[0063] The feature extraction module comprises multiple (e.g., L) stacked pressure-guided graph convolutional blocks. This module performs feature modulation and multilayer perceptual coding on the initial data collected at each acquisition point to obtain the encoded features.

[0064] For the current pressure-guided graph convolutional block, its output is obtained by adding the residuals of the output of the previous pressure-guided graph convolutional block and the processing result of the current pressure-guided graph convolutional block. If the current pressure-guided graph convolutional block is the first pressure-guided graph convolutional block, then its output is obtained based on the processing result of the initial acquired data.

[0065] ; in, For the first The output of the graph convolution block guided by the current pressure corresponding to each sampling point. For the first The output of the graph convolution block corresponding to each sampling point, guided by the previous pressure. For the first The processing result of the graph convolution block guided by the current pressure corresponding to each collection point.

[0066] The residuals of the output results of the graph convolutional block guided by the previous pressure for all acquisition points are added together with the processing results of the graph convolutional block guided by the current pressure for all acquisition points to obtain the output results of the graph convolutional block guided by the current pressure for all acquisition points.

[0067] The multiple stacked pressure-guided graph convolutional blocks of this application enable multiple iterative processing of the initial acquired data, allowing for comprehensive and accurate feature extraction from the initial acquired data.

[0068] Based on the above embodiments, the processing result of the current pressure-guided graph convolutional block is obtained in the following manner: Based on the output of the previous pressure-guided graph convolutional block, the aggregated features and tactile pressure values ​​of each sampling point are obtained; if the current pressure-guided graph convolutional block is the first pressure-guided graph convolutional block, then the aggregated features are obtained based on the aggregation results of low-resolution visual point cloud data and tactile point cloud data. Each aggregated feature is modulated based on a modulation factor to obtain each modulated feature; the modulation factor is generated based on tactile pressure values. When the tactile pressure value is greater than or equal to the pressure threshold, the modulated feature corresponding to the tactile pressure value is used as the sensitive feature. The sensitive feature is then subjected to enhanced multilayer perceptual coding to obtain the processing result of the graph convolution block guided by the current pressure. When the tactile pressure value is less than the pressure threshold, the modulated feature corresponding to the tactile pressure value is used as a general feature. The general feature is then subjected to general multilayer perceptual encoding to obtain the processing result of the graph convolution block guided by the current pressure.

[0069] like Figure 4 As shown, the feature extraction module includes a multilayer perceptron and L stacked pressure-guided graph convolutional blocks. The multilayer perceptron performs initial feature embedding and concatenation on each initial acquisition data point to obtain the initial feature vector for each acquisition point. The initial feature vector for each acquisition point includes visual point cloud features, tactile point cloud features, and tactile pressure values.

[0070] All initial feature vectors are input into L stacked pressure-guided graph convolutional blocks. These L stacked pressure-guided graph convolutional blocks generate encoded features based on the initial feature vectors. The output of the current pressure-guided graph convolutional block is obtained by adding the residuals of the output of the previous pressure-guided graph convolutional block and the processing result of the current pressure-guided graph convolutional block. Obtaining the processing result of each current pressure-guided graph convolutional block includes the following steps.

[0071] (1) Dynamic graph construction. The graph convolutional blocks guided by the current pressure use the k-nearest neighbors (KNN) algorithm to construct the graph structure.

[0072] (2) Local feature aggregation. Based on the output of the previous pressure-guided graph convolution block, the aggregated features of each acquisition point are obtained. and its tactile pressure value If the current pressure-guided graph convolutional block is the first pressure-guided graph convolutional block, then the aggregation function in the graph convolutional block is used to aggregate the visual point cloud features and tactile point cloud features in each initial feature vector to obtain each aggregated feature. .

[0073] (3) Conditional Feature Modulation. Features are extracted from each tactile pressure value to obtain initial features. The extracted initial features are input into a small gating network to generate modulation factors, which are then modulated using element-wise multiplication. This modulation ensures that the subsequent feature extraction process (multilayer perceptual coding) is directly influenced by the grasping task and tactile pressure value from the outset. The formula for calculating the modulated features is as follows.

[0074] ; in, For the first The modulated features corresponding to each acquisition point For the first Aggregated features corresponding to each collection point is the modulation factor.

[0075] For each acquisition point, there is a corresponding modulated feature and a tactile pressure value. Based on the tactile pressure value, the modulated feature is divided into sensitive features and general features. When the tactile pressure value is greater than or equal to a pressure threshold, it indicates that the acquisition point corresponding to that tactile pressure value is located in the grasping sensitive area, and the modulated feature corresponding to that tactile pressure value is taken as the sensitive feature. Multilayer perceptual encoding is then applied to enhance the sensitive feature to obtain the encoded feature.

[0076] When the tactile pressure value is less than the pressure threshold, it indicates that the acquisition point corresponding to the tactile pressure value is located in a non-sensitive area. The modulated feature corresponding to the tactile pressure value is then used as a general feature. A general multilayer perceptron encoding is applied to the general feature to obtain the encoded feature. The pressure-guided graph convolutional block of this application modulates the tactile pressure value through a gating network and routes the modulated feature to a dedicated MLP (multilayer perceptron encoding that enhances sensitive features) or a lightweight MLP (general multilayer perceptron encoding that performs general features) based on the pressure threshold. This achieves differentiated processing of the modulated feature extraction, accurately preserving microscopic geometric details strongly related to grasping (such as flat grasping surfaces and stable edges), and avoiding the smoothing effect of deep networks on key information.

[0077] The formula for calculating the encoded features is as follows.

[0078] ; in, For the first The tactile pressure value corresponding to each sampling point The pressure threshold, For the first The encoded feature corresponding to the collection point (which is also the th ) (Processing results of graph convolution blocks guided by the current pressure corresponding to each acquisition point). For enhanced multilayer perceptual coding, For general multilayer perceptual coding, For the first The modulated features corresponding to each acquisition point.

[0079] Based on the comparison between tactile pressure value and pressure threshold, this application divides the modulated features into sensitive features and general features, and processes them using enhanced multilayer perceptron coding (MLS) and general MLS, respectively, thereby improving the task adaptability of feature extraction. This application uses enhanced MLS for feature extraction of sensitive features, enabling accurate extraction of the microscopic geometric details of these sensitive features.

[0080] This application designs pressure-guided graph convolutional blocks, modulates tactile pressure values ​​through a gating network, and distinguishes between "grasping sensitive areas" (prediction points corresponding to sensitive features) and "non-sensitive areas" (prediction points corresponding to general features) based on pressure thresholds. It then performs enhanced multilayer perceptron coding processing and general multilayer perceptron coding processing respectively, thereby improving the task adaptability of feature extraction and accurately preserving the microscopic geometric details related to grasping.

[0081] Based on the above embodiments, the fusion module is used for: Determine the contact frequency and uncertainty score for each collection point; The weights of each encoded feature are determined based on contact frequency, uncertainty score, and encoded features. Based on the weights of each encoded feature, multi-layer perceptual fusion is performed on each encoded feature to obtain fused features.

[0082] The fusion module includes a cross-modal feature fusion module. The goal of this module is to generate a high-dimensional feature representation that integrates internal point cloud information with grasping frequency information, and is highly optimized for the grasping task. The core mechanism of the fusion module lies in dynamically weighting and fusing information based on the uncertainty of the encoded features and the contact frequency of the acquisition points through an attention mechanism. The input to the fusion module includes the encoded features and contact frequency of each acquisition point. The encoded features contain rich information such as the geometry and pressure of the point cloud.

[0083] Determine the contact frequency at each collection point. A clustering algorithm is used to calculate the contact frequency of each contact region; the core principle is to group contact events that are spatially close into the same contact region. First, define the contact event dataset. Each element It is the first The three-dimensional coordinates of the first effective contact in the global coordinate system (obtained through the kinematic model of the robotic arm or the positioning of the tactile sensor). This represents the total number of contacts. The neighborhood radius is defined. This refers to the "spatial proximity" threshold, which is set based on the resolution of the tactile point cloud data and the physical precision of the robotic arm's end effector contact. ,For example, Define the minimum number of points. This refers to the minimum number of touches required for a "valid cluster". A minimum number of points is set to account for accidental touches during the grasping task (e.g., one accidental touch). For example, That is, at least two contacts are required to constitute a meaningful contact area.

[0084] Iterate through all contact events, based on and Distance in space ≤ And the quantity is ≥ Contact events are grouped into clusters, each cluster representing a "contact region". Contact events not classified into any cluster (such as isolated single touches) are marked as "noise points" (which can be assigned a default frequency of 1 in subsequent processing). After clustering, the contact frequency of each contact region (cluster) is directly determined by the number of contact events contained within the cluster. The specific rules are as follows: For each contact region identified as a "valid cluster"... ( (As cluster number), its contact frequency Equal to the total number of contact events within the cluster: For example: if a cluster contains 3 contact events, then... For isolated contact events marked as "noise" (such as a single touch), they are assumed to belong to the "low frequency region" and assigned a fundamental frequency. (Ensure that basic weights are preserved during feature fusion). After the contact frequency calculation is completed, each contact area needs to be... of Mapping to corresponding points in the haptic point cloud is specifically achieved through spatial distance matching, calculating the value of each point in the haptic point cloud data. (Global coordinates) ) and the center of all contact areas Euclidean distance (mean of coordinates within the cluster) .like Then determine This is a contact area. To give it a contact frequency Otherwise, assign the default frequency. The contact frequency of each sampling point is embedded into the same dimensional space as the encoded feature of that sampling point, so as to facilitate subsequent feature concatenation and learning.

[0085] Determine the uncertainty score for each data collection point. Perform multilayer perceptron processing and activation on the encoded features at each data collection point to obtain the uncertainty score for each point. The uncertainty score characterizes the stability of the encoded features. The lower the uncertainty score, the more reliable the information carried by the encoded feature, and the more stable the encoded feature. The higher the uncertainty score, the less reliable the information carried by the encoded feature, and the less stable the encoded feature. The formula for calculating the uncertainty score is as follows.

[0086] ; in, For the encoded features, For multi-layer sensing processing, Scoring for uncertainty, To activate.

[0087] Based on contact frequency, uncertainty score, and encoded features, the weights of each encoded feature are determined. The formulas for calculating the weights of each encoded feature are as follows.

[0088] ; in, The weights of the encoded features, For the first Contact frequency at each sampling point Scoring for uncertainty, For the encoded features, For multi-layer sensing processing, To activate, This is a feature splicing operation.

[0089] Based on the weights of each encoded feature, multi-layer perceptual fusion is performed on the encoded features to obtain fused features. The calculation formula for the fused features is as follows.

[0090] ; in, As a feature of fusion, For multi-layer sensing processing, The weights of the encoded features, For the encoded features, This is a convolution operation.

[0091] This application employs a specially designed attention mechanism to explicitly assign higher weights to features in high-contact-frequency regions, as well as more stable encoded features.

[0092] This application quantifies the number of touches in different touch areas through a clustering algorithm, embeds the touch frequency into the multi-layer perception fusion of encoded features, and gives higher weight to high-touch frequency areas (key areas for grasping).

[0093] This application combines contact frequency, uncertainty score, and encoded features to determine the weights of the encoded features, giving higher weights to high-contact-frequency regions (key grasping areas) and high-determinism features. This solves the information redundancy / dilution problem of traditional fixed-ratio or iterative fusion, and is beneficial to improving the accuracy of fusing encoded features. This application uses a clustering algorithm to calculate the contact frequency of tactile point clouds; and generates the weights of each encoded feature through an attention mechanism (fusing encoded features, uncertainty score, and contact frequency), realizing task-adaptive weighted fusion of multiple tactile features.

[0094] Based on the above embodiments, the super-resolution model is trained in the following manner: Based on the actual capture reference information and actual high-resolution visual point cloud data of each sample collection point, the initial sample collection data of each sample collection point is labeled to obtain training samples. The super-resolution model is obtained by training the pre-defined model based on the training samples and the loss function. The loss function is determined based on the chamfer distance of the global point cloud, the chamfer distance of the stable sample collection points, the chamfer distance of the dangerous sample collection points, and the loss of the grasping reference information; the chamfer distance is determined based on the error between the actual high-resolution visual point cloud data and the predicted high-resolution visual point cloud data output by the preset model.

[0095] Based on the low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​of the sample collection area, the initial sample collection data for each sample collection point is obtained.

[0096] Based on the actual capture reference information and actual high-resolution visual point cloud data of each sample collection point, the initial sample collection data of each sample collection point is labeled to obtain labeled training samples.

[0097] Input the training samples into the preset model to obtain the high-resolution visual point cloud data and the prediction and grasping reference information output by the preset model.

[0098] Based on the loss function of the preset model, and combined with actual grasping reference information, actual high-resolution visual point cloud data, predicted high-resolution visual point cloud data, and predicted grasping reference information, the chamfer distance of the global point cloud, the chamfer distance of stable sample collection points, the chamfer distance of dangerous sample collection points, and the loss of grasping reference information are calculated to obtain the loss value. The parameters of the preset model are iteratively updated based on the loss value until the loss value is less than the set loss value, thus obtaining the super-resolution model. The formula for calculating the loss value is as follows.

[0099] ; in, The loss value. The chamfer distance of the global point cloud. To stabilize the loss weight of the chamfer distance of the sample collection points, To stabilize the chamfer distance of the sample collection points, The loss weight is the chamfer distance of the dangerous sample collection points. The chamfer distance of the hazardous sample collection point. To capture the loss weights of reference information, Losses incurred in order to capture reference information.

[0100] The value is determined based on the error between the actual grasping reference information and the predicted grasping reference information of the sample collection points. Since there is no one-to-one correspondence between the predicted point cloud and the actual sample collection points, this application uses nearest neighbor matching to establish the relationship between the predicted point cloud and the actual sample collection points. Specifically, for each point in the predicted point cloud (predicted high-resolution visual point cloud data)... The KNN algorithm is used to find the K nearest points in the actual high-resolution visual point cloud data, and then the actual capture reference information of these K nearest points is taken. The average of the values ​​is used as the point. The corresponding actual capture reference information. This method yields the actual capture reference information for all predicted point clouds. , dimension Predictive capture reference information for all predicted point clouds is obtained through a preset model. ,calculate and The mean square error loss is obtained .

[0101] The chamfer distance is determined based on the error between the actual high-resolution visual point cloud data and the predicted high-resolution visual point cloud data. Specifically, taking the chamfer distance of the global point cloud as an example, the calculation formula for the chamfer distance of the global point cloud is as follows: The chamfer distance of the global point cloud is determined based on the error between the actual high-resolution visual point cloud data of all sample collection points and all predicted high-resolution visual point cloud data.

[0102] ; in, For actual high-resolution visual point cloud data, To predict high-resolution visual point cloud data, The chamfer distance of the global point cloud. For any point in the actual high-resolution visual point cloud data, To predict any point in high-resolution visual point cloud data, for any point in arrive The sum of the minimum distances, for any point in arrive The sum of the minimum distances.

[0103] Specifically, the chamfer distance of the stable sample acquisition point is determined by the error between the actual high-resolution visual point cloud data of the stable sample acquisition point and the predicted high-resolution visual point cloud data of the stable sample acquisition point.

[0104] Specifically, the chamfer distance of the hazardous sample collection point is determined by the error between the actual high-resolution visual point cloud data of the hazardous sample collection point and the predicted high-resolution visual point cloud data of the hazardous sample collection point.

[0105] Stable sample collection points are sample collection points within a stable region of the sample collection area. Stable regions include sample collection areas with flat surfaces.

[0106] Hazardous sample collection points are sample collection points located in hazardous areas within the sample collection area. Hazardous areas include sample collection areas with sharp edges or protrusions.

[0107] This application embodiment determines the loss function by considering the chamfer distance of the global point cloud, the chamfer distance of stable sample collection points, the chamfer distance of dangerous sample collection points, and the loss of grasping reference information, thereby achieving a deep binding between point cloud super-resolution accuracy and robot grasping task requirements.

[0108] This application outputs grasping reference information from high-resolution visual point cloud data through a grasping prediction module, and designs a loss function with multiple loss terms (geometric accuracy constraints that distinguish between stable and dangerous areas + grasping probability loss), so that the super-resolution results (high-resolution visual point cloud data) not only have high-precision geometric attributes, but also directly provide decision-making basis for robot grasping planning, obstacle avoidance and other tasks, thus realizing the synergistic optimization of super-resolution accuracy and robot training practicality.

[0109] Based on the above embodiments, stable sample collection points and hazardous sample collection points are determined in the following manner: Obtain at least one feature value of the neighborhood points of the target sample collection point, whereby the at least one feature value characterizes the degree of dispersion of the neighborhood points in at least one principal direction; the target sample collection point can be any sample collection point. Calculate the curvature and average curvature of the target sample collection points based on at least one feature value; When the curvature of the target sample collection point is greater than the curvature threshold, and the average curvature of the target sample collection point is greater than the average curvature threshold, the target sample collection point is determined to be a dangerous sample collection point. When the curvature of the target sample collection point is less than or equal to the curvature threshold, and the average curvature of the target sample collection point is less than or equal to the average curvature threshold, the target sample collection point is determined to be a stable sample collection point.

[0110] Any sample collection point is designated as the target sample collection point. PCA analysis is performed on the target sample collection point to obtain at least one eigenvalue of its neighboring points. For example, Principal Component Analysis (PCA) is performed on the target sample collection point to obtain three eigenvalues ​​of its neighboring points. These three feature values ​​reflect the degree of dispersion of the neighborhood points of the target sample collection point in the three main directions.

[0111] Using at least one of the feature values ​​obtained above, calculate the curvature (e.g., Gaussian curvature) and mean curvature of the target sample acquisition points. Gaussian curvature It is approximately the product of the two smallest eigenvalues. The larger the value, the more pronounced the surface curvature (e.g., sharp edges) at the target sample collection point. The closer the value is to 0, the closer the surface of the target sample collection point is to a plane. Mean curvature It is approximately the average of the two smallest eigenvalues. The larger the value, the more pronounced the overall surface curvature of the target sample collection point. The formulas for calculating the curvature and average curvature of the target sample collection point are as follows.

[0112] ; in, The first eigenvalue, The second eigenvalue, The third eigenvalue, , The curvature of the target sample collection point. The average curvature of the target sample collection points.

[0113] To facilitate comparison of curvature and average curvature in different regions, the curvature and average curvature can be normalized to the [0, 1] interval. The curvature of the target sample collection points is normalized to the [0, 1] interval based on the maximum and minimum curvature. The average curvature of the target sample collection points is normalized to the [0, 1] interval based on the maximum and minimum average curvature. The formulas for calculating the normalized curvature and normalized average curvature are as follows.

[0114] ; in, The normalized curvature, The normalized mean curvature For curvature, For minimum curvature, For maximum curvature, For the minimum mean curvature, For the maximum mean curvature, The mean curvature.

[0115] Based on the normalized curvature and the normalized average curvature, the sample collection points are divided into dangerous sample collection points and stable sample collection points.

[0116] Optionally, when the normalized curvature of the target sample collection point is greater than the curvature threshold, and the normalized average curvature of the target sample collection point is greater than the average curvature threshold, the target sample collection point is identified as a dangerous sample collection point.

[0117] Optionally, when the normalized curvature of the target sample collection point is less than or equal to the curvature threshold, and the normalized average curvature of the target sample collection point is less than or equal to the average curvature threshold, the target sample collection point is determined as a stable sample collection point.

[0118] For example, with a curvature threshold of 0.6 and an average curvature threshold of 0.6, the normalized curvature of the target sample collection points... and the normalized mean curvature ,if ,and Then, the target sample collection point is defined as a hazardous sample collection point. If , Then the target sample collection point is defined as a stable sample collection point.

[0119] This application combines the comparison results of curvature and curvature threshold, as well as the comparison results of average curvature and average curvature threshold, to accurately identify stable sample collection points and dangerous sample collection points.

[0120] Furthermore, in the loss function, the loss weight is the chamfer distance of the stable sample collection points. Assign a higher value, for example, This is achieved by forcing the super-resolution model to generate dense and uniform point clouds, meeting the detail accuracy requirements during data capture. The loss weights are adjusted based on the chamfer distance of the collection points for hazardous samples. Assign a higher value, for example, It focuses on edge sharpening to avoid contour distortion caused by excessive smoothing.

[0121] Optionally, a key region can be defined based on the rate of change of the normal vector of the point cloud in the predicted high-resolution visual point cloud data, or the local entropy value, and a loss function can be determined based on the chamfer distance of the global point cloud, the chamfer distance of the sample collection points in the key region, and the loss of grasping reference information.

[0122] Existing super-resolution models only constrain point cloud accuracy using chamfer distance (CD) during network training, without considering the uniformity of distribution and geometric accuracy of key regions (such as stable grasping areas and dangerous edges) in real-world scenarios. This application designs a multi-loss function, including chamfer distance for stable regions (high-weighted constraint on uniformity of flat areas), chamfer distance for dangerous regions (high-weighted constraint on edge sharpness), and graspability prediction loss, to adapt to the requirements of grasping tasks.

[0123] This application constructs a task-oriented loss function, which divides the stable region (the region corresponding to the sampling points of stable samples) and the dangerous region (the region corresponding to the sampling points of dangerous samples) based on Gaussian curvature and mean curvature, and assigns differentiated weights; it includes the chamfer distance of the global point cloud, the chamfer distance of the sampling points of stable samples, the chamfer distance of the sampling points of dangerous samples, and the loss of the grasping reference information, thus constraining the adaptability of the super-resolution results to the grasping task.

[0124] This application focuses on robot grasping tasks in a training environment, integrating tactile pressure values ​​and contact frequency throughout the entire data processing process to form a closed loop: tactile pressure values ​​guide feature extraction, contact frequency dynamically adjusts fusion weights, and output results directly serve grasping decisions. Through this mechanism, the high-resolution visual point cloud data output by the super-resolution model not only pursues geometric accuracy but also achieves deep adaptation with downstream tasks such as grasping planning and obstacle avoidance through graspability annotation, thus enhancing the practicality of high-resolution visual point cloud data in robot training.

[0125] Robots play a crucial role in industrial automation, logistics, and services, while the demand for high-precision point cloud data in robot training is growing. Traditional single-modal point cloud data (such as those relying solely on vision or touch) suffers from insufficient detail and limited accuracy, affecting robot performance in complex tasks. High-resolution visual point cloud data is essential for the efficient operation of robots in complex environments. This application provides a method for reconstructing high-resolution visual point clouds based on multimodal information, effectively solving this problem and meeting the market demand for high-quality robot training data.

[0126] Traditional robot training relies on high-precision scanning equipment (e.g., LiDAR, industrial CT), which is expensive and complex to operate, limiting its application in small- to medium-sized training environments. This application utilizes super-resolution fusion of low-resolution visual point cloud data and low-cost tactile point cloud data to generate high-resolution visual point cloud data that combines global structural integrity with local detail accuracy, while reducing hardware investment. It also includes grasping reference information, directly adapting to grasping training needs. This feature significantly reduces the data acquisition cost for robot training, promoting the widespread adoption of high-resolution visual point cloud data in robot training.

[0127] In fields such as industrial automation (automotive parts assembly, electronic component gripping), logistics warehousing (irregular goods sorting), and service robots (household item handling), robots have extremely high requirements for gripping accuracy and environmental adaptability. The high-resolution visual point cloud data and gripping reference information generated in this application can provide robots with more accurate object geometry information and gripping area guidance, thereby improving their success rate in complex scenarios (e.g., identifying stable gripping surfaces and avoiding sharp edges). For example, in precision assembly in the 3C industry, robots can quickly locate the optimal gripping point of parts based on gripping reference information (e.g., gripping importance scores), reducing assembly errors and improving production efficiency.

[0128] Current point cloud super-resolution technologies mostly focus on geometric accuracy optimization, lacking deep integration with downstream tasks. This application proposes a collaborative mechanism for high-resolution visual point cloud data and grasping reference information, integrating tactile pressure values, contact frequency, and other information throughout the entire process of feature extraction, fusion, and loss optimization, forming a unique technical path. The high-resolution visual point cloud reconstruction method proposed in this application can provide an integrated service of "data generation - task training - performance optimization" in robot training solutions, thereby gaining a competitive advantage in sub-sectors such as industrial robots and service robots.

[0129] Existing high-resolution visual point cloud reconstruction methods have significant shortcomings in terms of the relevance of multimodal data fusion, the task adaptability of feature extraction, the correlation between super-resolution results and downstream grasping tasks, and the balance between computational efficiency and accuracy. This application addresses the core problems of insufficient point cloud super-resolution accuracy, disconnect from grasping tasks, and poor real-time performance in robot training environments through the aforementioned improvements, achieving collaborative optimization of high-resolution visual point cloud reconstruction and grasping perception.

[0130] This application introduces attention score calculation through a dynamic weighted fusion mechanism. By fusing the encoded features, uncertainty scores, and contact frequencies using MLP, a dynamic weight matrix is ​​generated to achieve deep association of multimodal features and task-adaptive fusion. The loss function introduces density-aware constraints, dividing the stable region and the danger region by Gaussian curvature and mean curvature. The stable region is given a high weight to ensure uniform distribution, while the danger region is enhanced with edge sharpness to avoid overfitting while meeting the geometric accuracy requirements of the grasping scene.

[0131] The apparatus for reconstructing high-resolution visual point clouds provided in this application will be described below. The apparatus for reconstructing high-resolution visual point clouds described below can be referred to in correspondence with the method for reconstructing high-resolution visual point clouds described above.

[0132] like Figure 7 As shown, a high-resolution visual point cloud reconstruction device includes: The acquisition module 701 is used to acquire initial acquisition data for each acquisition point based on the low-resolution visual point cloud data, tactile point cloud data and tactile pressure value of the acquisition area. The reconstruction module 702 is used to input the initial acquisition data into the super-resolution model, obtain the high-resolution visual point cloud data of the acquisition area output by the super-resolution model, and capture reference information of each prediction point.

[0133] All relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here.

[0134] The high-resolution visual point cloud reconstruction device provided in this application acquires initial acquisition data for each acquisition point based on low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​of the acquisition area. The initial acquisition data is then input into a super-resolution model to obtain high-resolution visual point cloud data of the acquisition area output by the super-resolution model, along with grasping reference information for each predicted point. This application achieves feature enhancement of low-resolution visual point cloud data and data complementarity between full coverage and local details through tactile point cloud data and tactile pressure values, which is beneficial for improving the accuracy of the reconstructed high-resolution visual point cloud data. Simultaneously, determining the grasping reference information for each predicted point provides a decision-making basis for the robot to perform tasks such as grasping and obstacle avoidance, thus improving the practicality of the high-resolution visual point cloud data.

[0135] Figure 8 An example is a schematic diagram of the physical structure of an electronic device, such as... Figure 8 As shown, the electronic device may include a processor 810, a communications interface 820, a memory 830, and a communication bus 840. The processor 810, communications interface 820, and memory 830 communicate with each other via the communication bus 840. The processor 810 can call logical instructions in the memory 830 to execute a high-resolution visual point cloud reconstruction method. This method includes: acquiring initial acquisition data for each acquisition point based on low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​of the acquisition area; inputting the initial acquisition data into a super-resolution model to obtain high-resolution visual point cloud data of the acquisition area output by the super-resolution model, and grasping reference information for each prediction point.

[0136] Furthermore, the logical instructions in the aforementioned memory 830 can be implemented as software functional units and, when sold or used as independent products, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0137] On the other hand, this application also provides a computer program product, which includes a computer program that can be stored on a non-transitory computer-readable storage medium. When the computer program is executed by a processor, the computer can perform the high-resolution visual point cloud reconstruction method provided by the above methods. The method includes: obtaining initial acquisition data for each acquisition point based on low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​of the acquisition area; inputting each initial acquisition data into a super-resolution model to obtain high-resolution visual point cloud data of the acquisition area output by the super-resolution model, and grasping reference information for each prediction point.

[0138] In another aspect, this application also provides a non-transitory computer-readable storage medium storing a computer program thereon. When executed by a processor, the computer program implements the high-resolution visual point cloud reconstruction method provided by the above methods. The method includes: acquiring initial acquisition data for each acquisition point based on low-resolution visual point cloud data, tactile point cloud data, and tactile pressure values ​​of the acquisition area; inputting each initial acquisition data into a super-resolution model to acquire high-resolution visual point cloud data of the acquisition area output by the super-resolution model, and grasping reference information for each prediction point.

[0139] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. Those skilled in the art can understand and implement this without any creative effort.

[0140] Through the above description of the embodiments, those skilled in the art can clearly understand that each embodiment can be implemented by means of software plus necessary general-purpose hardware platforms, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solutions, in essence or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, magnetic disk, optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute the methods described in the various embodiments or some parts of the embodiments.

[0141] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of this application.

Claims

1. A method for reconstructing high-resolution visual point clouds, characterized in that, include: Based on the low-resolution visual point cloud data, tactile point cloud data and tactile pressure values ​​of the acquisition area, the initial acquisition data of each acquisition point is obtained; Each of the initial acquisition data is input into the super-resolution model to obtain high-resolution visual point cloud data of the acquisition area output by the super-resolution model, and capture reference information of each prediction point.

2. The method for reconstructing high-resolution visual point clouds according to claim 1, characterized in that, The super-resolution model includes a feature extraction module, a fusion module, an upsampling module, and a capture prediction module. The super-resolution model is used for: Based on the feature extraction module, feature modulation and multi-layer perceptual coding are performed on each of the initial collected data to obtain the encoded features of each collection point. The fusion module performs weighted fusion on each encoded feature to obtain the fused feature. Based on the upsampling module, the fused features are compressed, extracted, expanded, and rearranged to obtain the high-resolution visual point cloud data; Based on the grasping prediction module, feature extraction and activation are performed on the high-resolution visual point cloud data to generate grasping reference information for each prediction point.

3. The method for reconstructing high-resolution visual point clouds according to claim 2, characterized in that, The feature extraction module includes multiple stacked pressure-guided graph convolutional blocks, and the feature extraction module is used for: Based on the output of the previous pressure-guided graph convolutional block and the processing result of the current pressure-guided graph convolutional block, obtain the output of the current pressure-guided graph convolutional block. If the current pressure-guided graph convolutional block is the first pressure-guided graph convolutional block, then based on the processing result of the initial acquired data, obtain the output result of the current pressure-guided graph convolutional block; Based on the output of the last pressure-guided graph convolutional block, the encoded features are obtained.

4. The method for reconstructing high-resolution visual point clouds according to claim 3, characterized in that, The processing result of the current pressure-guided graph convolutional block is obtained based on the following method: Based on the output of the previous pressure-guided graph convolutional block, the aggregated features and tactile pressure values ​​of each of the sampling points are obtained; if the current pressure-guided graph convolutional block is the first pressure-guided graph convolutional block, the aggregated features are obtained based on the aggregation results of the low-resolution visual point cloud data and the tactile point cloud data. Each of the aggregated features is modulated based on a modulation factor to obtain each modulated feature. The modulation factor is generated based on the tactile pressure value; When the tactile pressure value is greater than or equal to the pressure threshold, the modulated feature corresponding to the tactile pressure value is used as a sensitive feature, and the sensitive feature is subjected to enhanced multilayer perceptual coding to obtain the processing result of the graph convolution block guided by the current pressure. When the tactile pressure value is less than the pressure threshold, the modulated feature corresponding to the tactile pressure value is used as a general feature, and a general multilayer perceptual encoding is performed on the general feature to obtain the processing result of the graph convolution block guided by the current pressure.

5. The method for reconstructing high-resolution visual point clouds according to claim 2, characterized in that, The fusion module is used for: Determine the contact frequency and uncertainty score for each of the aforementioned collection points; The weights of each encoded feature are determined based on the contact frequency, the uncertainty score, and the encoded features. Based on the weights of each encoded feature, multi-layer perceptual fusion is performed on each encoded feature to obtain the fused feature.

6. The method for reconstructing high-resolution visual point clouds according to claim 2, characterized in that, The super-resolution model was trained in the following manner: Based on the actual capture reference information and actual high-resolution visual point cloud data of each sample collection point, the initial sample collection data of each sample collection point is labeled to obtain training samples. The super-resolution model is obtained by training the preset model based on the training samples and the loss function. The loss function is determined based on the chamfer distance of the global point cloud, the chamfer distance of the stable sample collection points, the chamfer distance of the dangerous sample collection points, and the loss of grasping reference information; the chamfer distance is determined based on the error between the actual high-resolution visual point cloud data and the predicted high-resolution visual point cloud data output by the preset model.

7. The method for reconstructing high-resolution visual point clouds according to claim 6, characterized in that, The stable sample collection points and the hazardous sample collection points were determined based on the following method: Obtain at least one feature value of the neighborhood points of the target sample collection point, wherein the at least one feature value characterizes the degree of dispersion of the neighborhood points in at least one main direction; The target sample collection point can be any of the aforementioned sample collection points; Based on the at least one feature value, calculate the curvature and average curvature of the target sample collection point; When the curvature of the target sample collection point is greater than the curvature threshold, and the average curvature of the target sample collection point is greater than the average curvature threshold, the target sample collection point is determined to be the dangerous sample collection point. When the curvature of the target sample collection point is less than or equal to the curvature threshold, and the average curvature of the target sample collection point is less than or equal to the average curvature threshold, the target sample collection point is determined to be the stable sample collection point.

8. A high-resolution visual point cloud reconstruction device, characterized in that, include: The acquisition module is used to obtain the initial acquisition data of each acquisition point based on the low-resolution visual point cloud data, tactile point cloud data and tactile pressure value of the acquisition area; The reconstruction module is used to input the initial acquisition data into the super-resolution model, obtain the high-resolution visual point cloud data of the acquisition area output by the super-resolution model, and the capture reference information of each prediction point.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the high-resolution visual point cloud reconstruction method as described in any one of claims 1 to 7.

10. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the high-resolution visual point cloud reconstruction method as described in any one of claims 1 to 7.

11. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the high-resolution visual point cloud reconstruction method as described in any one of claims 1 to 7.

Images