A robot grasping method and system based on light field structure complexity driving

Abstract

This invention provides a robot grasping method driven by light field structural complexity, comprising: constructing a light field texture structure index (LFTI) based on an RGB image; constructing a manifold topological curvature field (MTC) based on a depth image; constructing a joint light field structural complexity map (LFSC) satisfying: LFSC = α·LFTI + β·MTC; processing the LFSC of the current frame using a dynamic peak-valley energy separation algorithm to output the final region of interest (ROI); constructing candidate viewpoints based on the ROI, selecting the best viewpoint (NBV) from the candidate viewpoints, controlling the robotic arm to move to the corresponding pose to perform a high-resolution secondary scan of the ROI to obtain a local high-precision point cloud; and performing coordinate system registration and fusion of the obtained local high-precision point cloud with the global background point cloud to construct a multi-resolution hybrid point cloud scene.

Description

Technical Field

[0001] This invention relates to the field of industrial automation and intelligent robot technology, and in particular to a robot grasping method and system driven by the structural complexity of light field. Background Technology

[0002] Currently, robots are widely used in automobile manufacturing, 3C electronics assembly, and logistics sorting. In mixed workpiece grasping scenarios, robots need to identify and grasp specific workpieces from scattered stacks of materials. Existing mainstream solutions typically employ a single-type visual perception architecture, such as "eye on hand" or "global camera," to acquire RGB-D point cloud data, and combine it with traditional template matching based on CAD models or deep learning-based 6D pose estimation networks (such as PoseCNN and DenseFusion) for recognition.

[0003] Although the above technologies have been implemented in some scenarios, the following problems still exist when dealing with highly complex stacking scenarios:

[0004] First, extreme lighting and material interference: Existing algorithms heavily rely on complete depth maps or clear texture maps. However, in industrial settings, the high reflectivity of metal workpieces can cause saturation at the ToF camera receiver, resulting in large areas of depth holes; while black light-absorbing objects or transparent objects can lead to missing depth data. Traditional single-modal algorithms often directly output incorrect poses in these data blind spots, resulting in ineffective captures or collisions.

[0005] Second, recognition ambiguity due to stacking occlusion: When multiple workpieces are tightly stacked, especially when similar workpieces interlock, the edge boundaries between objects are extremely blurred. Low-resolution global scanning cannot distinguish tiny physical gaps, causing the algorithm to misidentify two objects as a single entity (undersegmentation), or fail to correctly determine the stacking hierarchy, triggering bottom-pulling operations and causing the material pile to collapse.

[0006] Third, the trade-off between production speed and accuracy: To see the aforementioned minute details clearly, high-resolution scanning or multiple multi-angle shots must be used, which generates massive amounts of point cloud data. With limited computing power, processing such a large amount of data usually takes several seconds or even longer, severely slowing down the production cycle; on the other hand, reducing resolution in pursuit of speed will lead to the loss of key features.

[0007] Fourth, it lacks proactive exploration capabilities: existing visual systems are passive, like a rigid observer. Once the target cannot be seen clearly due to perspective, the system will only mechanically retry or report an error, lacking the proactive perceptual intelligence of humans, such as tilting their heads or getting closer to see, resulting in extremely poor versatility.

[0008] The above background information is provided only to assist in understanding the inventive concept and technical solution of this invention. It does not necessarily belong to the prior art of this application, nor does it necessarily provide technical teaching. In the absence of clear evidence that the above information was disclosed before the filing date of this application, the above background information should not be used to evaluate the novelty and inventiveness of this application. Summary of the Invention

[0009] The purpose of this invention is to provide a robot grasping method and system driven by the structural complexity of light field.

[0010] To achieve the above objectives, the technical solution adopted in the first aspect of the present invention is as follows:

[0011] A robot grasping method driven by light field structural complexity includes the following steps:

[0012] The Light Field Texture Structure Index (LFTI) is constructed based on RGB images to quantify the texture richness of an object's surface.

[0013] A manifold topological curvature field (MTC) based on depth maps is constructed to quantify the geometric complexity of object surfaces.

[0014] Construct the joint optical field structure complexity graph LFSC that satisfies the following relationship:

[0015] LFSC = α · LFTI + β · MTC,

[0016] Where α is the light field texture weight and β is the geometric curvature weight, satisfying α+β=1;

[0017] The Joint Light Field Structure Complexity Map (LFSC) of the current frame is processed based on the Dynamic Peak-Valley Energy Separation Algorithm. The boundary energy threshold T between the background and the stacked objects is distinguished. Regions in the LFSC with energy values ​​greater than the boundary energy threshold T are defined as high-complexity regions. Connectivity analysis or morphological processing is performed on the high-complexity regions to remove isolated noise blocks and output the final region of interest (ROI).

[0018] Candidate viewpoints are constructed based on the output final region of interest (ROI). The best viewpoint (NBV) is selected from the candidate viewpoints, and the robotic arm is controlled to move to the corresponding pose to perform a high-resolution secondary scan of the final ROI to obtain a local high-precision point cloud.

[0019] The obtained local high-precision point cloud is registered and fused with the global background point cloud to construct a multi-resolution hybrid point cloud scene. A pose estimation algorithm is run on the hybrid point cloud scene to calculate the optimal grasping pose of the workpiece to be grasped and then grasping is performed.

[0020] In some embodiments, when constructing the Joint Light Field Structure Complexity Map (LFSC), a basic weight α0 for light field texture and a basic weight β0 for geometric curvature are set to satisfy α0 + β0 = 1. An upper limit for reflectivity saturation tolerance Ts1 and an upper limit for depth hole rate tolerance Th1 are set. The reflectivity saturation S and depth map hole rate H of the current frame are calculated. When S > Ts1, the light field texture weight α0 is reduced to obtain the light field texture weight α, and the basic weight β0 for geometric curvature is increased to obtain the geometric curvature weight β. When H > Th1, the light field texture weight α0 is increased to obtain the light field texture weight α, and the basic weight β0 for geometric curvature is reduced to obtain the geometric curvature weight β.

[0021] In some embodiments, during the execution of a grasping task, the image reflectance saturation S, depth map hole rate H, and edge density change rate within the target area are considered based on the current grasping period. and historical data capture success rate One or more of the indicators are used to dynamically update the light field texture weight α and the geometric curvature weight β.

[0022] In some embodiments, when dynamically updating the light field texture weight α and the geometric curvature weight β, a compensation amount is constructed. :

[0023] ,

[0024] in, To achieve the preset target success rate, , , and The preset adjustment coefficient, The value ranges from 0.5 to 3.0. The value ranges from 0.5 to 3.0. The value range is 0.1 to 1.5. The value range is 0.01 to 0.5;

[0025] The compensation amount Light field texture weights superimposed on the current frame To obtain the updated light field texture weights for the next frame. .

[0026] In some embodiments, the dynamic peak-valley energy separation algorithm is as follows:

[0027] The global energy mean is calculated based on the Joint Light Field Structure Complexity Diagram (LFSC). with global energy standard deviation The boundary energy threshold T satisfies: ,in The complexity separation coefficient, The value ranges from 1 to 3, or the boundary energy threshold T satisfies: ,in For adjustment coefficients, The value range is 0.01 to 0.2. To achieve the preset target success rate, To improve the success rate of historical data collection.

[0028] In some embodiments, the specific method for selecting the best viewpoint NBV from the candidate viewpoints is as follows:

[0029] Visibility assessment: For each candidate viewpoint, simulate its imaging effect on the final region of interest (ROI) and calculate the visible area percentage under the candidate viewpoint. :

[0030] ,

[0031] in The total number of points in the final region of interest (ROI). The number of unobstructed visible points under the candidate viewpoint;

[0032] Quality assessment: For each candidate viewpoint, calculate the depth map hole rate H, image reflectance saturation S, and edge sharpness E to obtain a quality score Q.

[0033] ,

[0034] in For the weighting coefficients, satisfying ;

[0035] Overall score: The percentage of the visible area. The candidate viewpoint comprehensive score is obtained by fusing the quality score Q. :

[0036] ,

[0037] in ;

[0038] Optimal Viewpoint (NBV) Determination: Selection of Candidate Viewpoints Based on Comprehensive Scoring The highest candidate viewpoint is the optimal viewpoint NBV.

[0039] In some embodiments, the construction steps of the Light Field Texture Structure Index (LFTI) are as follows:

[0040] RGB image preprocessing: Convert the RGB image to a grayscale image;

[0041] Edge density extraction: Within a sliding window W centered at (x,y) and of size N xM, the density of the contour lines is statistically analyzed. To quantify the disorder of the material stacking and the difficulty of grasping:

[0042] ,

[0043] in This is the sum of the edge values ​​of all pixels in the sliding window W;

[0044] Local contrast feature extraction: in the area of ​​size Within the sliding window W, calculate the local grayscale mean values ​​respectively. Local grayscale standard deviation Based on the local grayscale mean and the local grayscale standard deviation Construct the local contrast feature C(x,y):

[0045] ;

[0046] Feature fusion constructs the light field texture structure index LFTI: the density of the contour lines. Mapping to the range of 0 to 1 using the min-max normalization method, denoted as... The local contrast feature C(x,y) is mapped to the range of 0~1 using the min-max normalization method, denoted as... Set preset weight coefficients , And satisfy Weighted fusion yields the light field texture structure index LFTI of the sliding window W, satisfying:

[0047] .

[0048] In some embodiments, the density of the contour lines When using the minimum-maximum normalization method, the following conditions must be met:

[0049] .

[0050] In some embodiments, when the local contrast feature C(x,y) is subjected to a minimum-maximum normalization method, the following conditions are met:

[0051] .

[0052] In some embodiments, the construction steps of constructing the manifold topological curvature field MTC based on the depth map are as follows:

[0053] Depth map preprocessing: After preprocessing the depth map, a smooth depth map is output, denoted as D(x,y);

[0054] Local curvature feature extraction: In the smoothed depth map D(x,y), the second-order depth divergence between the current pixel D(x,y) and its four orthogonal neighboring pixels (up, down, left, and right) is calculated using the discrete Laplacian operator. :

[0055] ,

[0056] To combat interference from single-point noise in the sensor, a method centered at (x,y) is used. Define local curvature features within the sliding window W. :

[0057] ;

[0058] Extraction of normal rate of change feature: setting points The normal vector is any point within the neighborhood The normal vector is , definition point arrive The Euclidean distance in two-dimensional space is Calculate the characteristic of the normal rate of change :

[0059] ,

[0060] in, It is the cosine similarity of the normal vectors, where e is the natural constant 2.71828;

[0061] Feature fusion to construct manifold topological curvature field MTC: for the local curvature features After normalization, we get For the normal rate of change characteristic After normalization, we get Preset geometric weight parameters ,satisfy = 1, thus obtaining the topological curvature field MTC of the manifold, satisfying:

[0062] .

[0063] In some embodiments, the local curvature features When performing normalization, the following conditions must be met:

[0064] .

[0065] In some embodiments, the normal rate of change feature When performing normalization, the following conditions must be met:

[0066] .

[0067] To achieve the above objectives, the technical solution adopted in the second aspect of the present invention is as follows:

[0068] A robot grasping system driven by light field structural complexity is configured to execute the robot grasping method driven by light field structural complexity as described above.

[0069] In some embodiments, it is applied to scenarios such as loading and unloading parts, assembling precision components, or unpacking and sorting packages.

[0070] The beneficial effects of the technical solution provided by this invention are as follows:

[0071] a. This invention addresses the contradiction between high-resolution imaging and real-time processing in existing visual perception technologies when facing complex stacked scenarios, namely the data redundancy caused by full-resolution scanning and the feature loss caused by low-resolution scanning. It provides an adaptive resolution robot randomized grasping method and system that utilizes light field structure complexity for active environmental perception, suitable for grasping randomly stacked, mutually occluded, and highly reflective materials;

[0072] b. This invention achieves a perfect balance between efficiency and accuracy. Compared with full-resolution scanning, this method only performs fine scanning on complex areas, which greatly saves processing time and solves the core perception problem of how to efficiently and accurately see complex working conditions. Attached Figure Description

[0073] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0074] Figure 1 This is a block diagram of the joint optical field structure complexity (LFSC) calculation and fusion logic in a robot grasping method driven by optical field structure complexity according to an embodiment of the present invention.

[0075] Figure 2This is a timing diagram of a robot grasping method based on light field structure complexity driven by an embodiment of the present invention. Detailed Implementation

[0076] To enable those skilled in the art to better understand the present invention, the technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort should fall within the scope of protection of the present invention.

[0077] It should be noted that the terms "comprising" and "having" and any variations thereof in this invention are intended to cover non-exclusive inclusion, for example, a process, method, apparatus, product or device that includes a series of steps or units is not necessarily limited to those steps or units that are explicitly listed, but may include other steps or units that are not explicitly listed or that are inherent to such processes, methods, products or devices.

[0078] In this invention, to facilitate understanding of the subsequent steps, the key terms involved in this invention are explained as follows:

[0079] (1) Light-field Texture Index (LFTI):

[0080] This invention presents a unique comprehensive feature index for characterizing the complexity of two-dimensional image texture within a target area. It is obtained by fusing and calculating the image edge density and local contrast features, and is used to reflect the roughness and irregularity of the target surface at the visual level.

[0081] (2) Manifold Topology Curvature Field (MTC):

[0082] This invention features a unique comprehensive characteristic index that characterizes the three-dimensional geometric complexity of the target area. It is obtained by fusing the local curvature change and normal vector change rate of the target surface and is used to reflect the degree of undulation and stacking edge strength of the target area in spatial geometry.

[0083] (3) Joint Light-field Structure Complexity Diagram (LFSC):

[0084] This invention features a unique two-dimensional distribution map obtained by fusing the Light Field Texture Structure Index (LFTI) and the Manifold Topological Curvature Field (MTC). Its pixel values ​​are used to comprehensively reflect the overall grasping difficulty level of the target area in both visual texture and spatial geometry dimensions.

[0085] (4) Sliding window:

[0086] A local computational region refers to a region that moves within the image plane at a fixed step size. It is used to perform statistical analysis on local features of the image. The window size and step size parameters can be set according to computing resources and real-time requirements.

[0087] This invention provides a robot grasping method driven by the structural complexity of light fields, specifically including the following steps:

[0088] S0 Global perception to acquire data. In some embodiments, this step mainly involves obtaining RGB images and depth maps through a 3D camera.

[0089] S1 constructs the Light Field Texture Structure Index (LFTI) based on RGB images to quantify the texture richness of an object's surface;

[0090] S2 constructs a manifold topological curvature field (MTC) based on depth maps to quantify the geometric complexity of object surfaces.

[0091] S3 constructs the joint optical field structure complexity graph LFSC, which satisfies the following relationship:

[0092] LFSC = α · LFTI + β · MTC,

[0093] Where α is the light field texture weight and β is the geometric curvature weight, satisfying α+β=1;

[0094] S4 processes the Joint Light Field Structure Complexity Map (LFSC) of the current frame based on the Dynamic Peak-Valley Energy Separation Algorithm, distinguishes the boundary energy threshold T between the background and the stacked objects, defines the regions in the LFSC with energy values ​​greater than the boundary energy threshold T as high-complexity regions, performs connected component analysis or morphological processing on the high-complexity regions to remove isolated noise blocks and output the final region of interest (ROI).

[0095] S5 constructs candidate viewpoints based on the final region of interest (ROI) output, selects the best viewpoint (NBV) from the candidate viewpoints, and controls the robotic arm to move to the corresponding pose to perform a high-resolution secondary scan of the final ROI to obtain a local high-precision point cloud.

[0096] S6 performs coordinate system registration and fusion between the obtained local high-precision point cloud and the global background point cloud to construct a multi-resolution hybrid point cloud scene. The pose estimation algorithm is run on the hybrid point cloud scene to calculate the best grasping pose of the workpiece to be grasped and then the grasping is performed.

[0097] Step S1 involves quantifying the richness of the object's surface texture. A higher Light Field Texture Structure Index (LFTI) value indicates a more chaotic material stacking in the area, making grasping more difficult and providing a 2D texture-level basis for subsequent grasping decisions. The steps for constructing the LFTI are as follows:

[0098] S1.1 RGB Image Preprocessing: Convert the RGB image to a grayscale image. Specifically, in some embodiments, the input RGB image is converted to a grayscale image, and the grayscale values ​​are normalized to reduce the impact of different light intensity variations on the subsequent texture feature extraction results. The RGB color image contains three channels: red, green, and blue. They are combined into a single-channel grayscale image brightness value (Gray) using a weighted average method. The brightness value is calculated for each pixel, and finally, a grayscale image of the same size as the original image is obtained.

[0099] in, The weights are assigned based on the fact that the human eye is most sensitive to green, followed by red, and least sensitive to blue, thus simulating the visual characteristics of the human eye and making the grayscale image more closely match the brightness perceived by the human eye.

[0100] When normalizing each pixel, use the following formula:

[0101] ,

[0102] Where X represents the original value of the current pixel. This represents the minimum value among all values ​​in the entire graph. This represents the maximum value among all values ​​in the entire graph. This represents the normalized value calculated for each pixel, and the result must be between 0 and 1.

[0103] S1.2 Edge density extraction: Within a sliding window W centered at (x,y) and of size N xM, the density of the contour lines is statistically analyzed. To quantify the disorder of the material stacking and the difficulty of grasping:

[0104] ,

[0105] in The density of the contour lines is the sum of the edge values ​​of all pixels in the sliding window W. The value ranges from 0 to 1. The larger the value, the more edges there are in the area and the more disorderly the material is stacked.

[0106] In some embodiments, this step is specifically as follows: The image is scanned pixel by pixel, and the outline of the object is extracted by calculating the brightness difference between adjacent pixels. Subsequently, the density of the outline within the sliding window W is statistically analyzed to determine whether this area is the most chaotic and difficult-to-grasp part of the material stack. The edge value E(i,j) of each pixel is calculated through the following steps:

[0107] Assume the current pixel is located at (x, y) and its brightness value is I(x, y). To compare its difference with surrounding pixels, the value is calculated in two directions:

[0108] Horizontal differences refer to left-right comparisons. The brightness of the pixel to the right of the given pixel is subtracted from the brightness of the pixel to the left of the given pixel, which satisfies the condition.

[0109] = I(x+1, y) - I(x-1, y),

[0110] Vertical differences refer to comparisons between different points in the vertical direction. : Subtract the brightness of the pixel above the pixel from the brightness of the pixel below it, satisfying the following condition:

[0111] = I(x, y+1) - I(x, y-1),

[0112] Then calculate the brightness difference between adjacent pixels. The larger this value, the more it indicates that this area resembles an edge.

[0113] ,

[0114] Let E(i,j) be the edge value for each pixel, if the total difference just calculated... If the value is greater than the preset threshold, the system determines that it is a contour line and records E(i,j) as 1. If the value is less than or equal to the preset threshold, the system determines that it is not a contour line and records E(i,j) as 0.

[0115] S1.3 Local Contrast Feature Extraction: In a size of Within the sliding window W, calculate the local grayscale mean values ​​respectively. Local grayscale standard deviation Based on local gray-scale mean and local gray-scale standard deviation We construct a local contrast feature C(x,y), which is used to characterize the severity of local texture changes in an image. Specifically:

[0116] This step reflects the degree of fluctuation in local grayscale values: high local contrast indicates drastic texture changes, such as complexly patterned parts or stacked materials; low contrast indicates smooth textures, such as a solid-color desktop. This step supplements insufficient edge density; for example, some areas may have few edges but messy textures, which can be reflected by local contrast.

[0117] S1.4 Feature fusion constructs the Light Field Texture Structure Index (LFTI): the density of contour lines. Mapping to the range of 0 to 1 using the min-max normalization method, denoted as... The local contrast feature C(x,y) is mapped to the range of 0~1 using the min-max normalization method, denoted as... Set preset weight coefficients , And satisfy Weighted fusion yields the Light Field Texture Structure Index (LFTI) of the sliding window W, satisfying:

[0118] .

[0119] Among them, the preset weight coefficient , The size can be adjusted according to the scenario; for example, when grabbing parts with obvious outlines, increase the size. Focus on edge density; when grabbing materials with complex textures but few outlines, increase the value. The higher the final Light Field Texture Structure Index (LFTI) value, the more chaotic the texture of the area, and the more difficult it is to grasp the material; the lower the value, the smoother the area, and the easier it is to grasp.

[0120] Specifically, the density of the outline When using the minimum-maximum normalization method, the following conditions must be met:

[0121] .

[0122] When the local contrast feature C(x,y) is subjected to the minimum-maximum normalization method, the following conditions are met:

[0123] The above normalization process can prevent features from being submerged due to weighting.

[0124] In step S2, a feature map extracted from the depth map, i.e. 3D geometric information, is used to quantify the geometric complexity of the object's surface. It determines whether the object's surface is flat, smooth, or a complex region with many edges or wrinkles through two dimensions: the degree of surface curvature and the degree of drastic change in orientation. This provides a 3D geometric basis for subsequent grasping decisions and complements the 2D texture complexity of LFTI.

[0125] Specifically, in some embodiments, the construction steps of the manifold topological curvature field (MTC) based on the depth map are as follows:

[0126] S2.1 Depth Map Preprocessing: After preprocessing the depth map, a smooth depth map is output, denoted as D(x,y). Specifically, the preprocessing involves performing at least one of bilateral filtering, median filtering, or guided filtering on the input depth map to reduce random noise while preserving the object edge structure, and / or interpolating or filling the neighborhood of depth hole regions.

[0127] S2.2 Local Curvature Feature Extraction: In the smoothed depth map D(x,y), the discrete Laplacian operator is used to calculate the second-order depth divergence between the current pixel D(x,y) and its four orthogonal neighboring pixels (up, down, left, and right). :

[0128] ,

[0129] For the depth second-order divergence in a flat region Approximately equal to 0, the second-order divergence of depth in the curved region. Large absolute values, such as a pixel on the surface of a ping-pong ball, result in a large value because the surrounding depth curves towards the center.

[0130] To combat interference from single-point noise in the sensor, a method centered at (x,y) is used. Define local curvature features within the sliding window W. :

[0131] .

[0132] In step S2.2 above, the local curvature features Used to quantify the curvature of an object's surface at that pixel: a large curvature means the surface is severely curved, such as a sphere, the rim of a cup, or the edges of a part; a small curvature means the surface is flat, such as a tabletop or a wall.

[0133] S2.3 Extraction of Normal Change Rate Feature: Set Point The normal vector is any point within the neighborhood The normal vector is Definition point arrive The Euclidean distance in two-dimensional space is Calculate the characteristic of the normal rate of change :

[0134] ,

[0135] in, It is the cosine similarity of the normal vectors. The closer the value is to 1, the more similar the orientations of the two normal vectors are; the closer the value is to 0, the more perpendicular the orientations are. 1 minus Represents the difference; the larger the value, the greater the difference in orientation. e is the natural constant 2.71828.

[0136] In step S2.3 above, the characteristic of the normal rate of change... This is used to quantify the difference between the normal vector of the current pixel and the normal vectors of surrounding pixels: a large difference indicates a drastic change in surface orientation, such as the corners of parts or the gaps in stacked materials; a small difference indicates a consistent surface orientation, such as a smooth cylindrical surface or a flat tabletop. This step is a supplement to the local curvature feature. Curvature describes the degree of bending, and the rate of change of normal describes the degree of orientation change. The combination of the two can more comprehensively characterize geometric complexity.

[0137] S2.4 Feature Fusion to Construct Manifold Topological Curvature Field (MTC): Local Curvature Features After normalization, we get Characteristics of the normal rate of change After normalization, we get Preset geometric weight parameters ,satisfy = 1, thus obtaining the manifold topological curvature field MTC, which satisfies:

[0138] .

[0139] Preset geometric weight parameters It can be adjusted according to the scenario. For example, when grabbing curved objects, increase γ1 to emphasize curvature, and when grabbing angular objects, increase γ2 to emphasize the rate of change of normal.

[0140] The higher the MTC value of the manifold topological curvature field, the more complex the geometry of the region, and the more difficult it is to grasp. The lower the value, the flatter the region, and the easier it is to grasp.

[0141] For local curvature features When performing normalization, the following conditions must be met:

[0142] .

[0143] Characteristics of normal rate of change When performing normalization, the following conditions must be met:

[0144] .

[0145] The above normalization process can prevent features from being submerged due to weighting.

[0146] Specifically, step S3, when constructing the joint optical field structure complexity map LFSC, is divided into the following two cases: Case 1 and Case 2.

[0147] Case 1: Set the basic weight α0 for light field texture and the basic weight β0 for geometric curvature, satisfying α0 + β0 = 1. Set the upper limit of reflective saturation tolerance Ts1 and the upper limit of depth hole rate tolerance Th1. Calculate the reflective saturation S and depth map hole rate H of the current frame. When S > Ts1, decrease the basic weight α0 for light field texture to obtain the light field texture weight α, and increase the basic weight β0 for geometric curvature to obtain the geometric curvature weight β. When H > Th1, increase the basic weight α0 for light field texture to obtain the light field texture weight α, and decrease the basic weight β0 for geometric curvature to obtain the geometric curvature weight β.

[0148] In the first scenario, based on the RGB and depth map quality of the current frame, the initial fusion weights of LFTI and MTC are adjusted. The core logic is to rely more on whichever data source has higher reliability.

[0149] The image reflectance saturation S and depth map hole rate H of the current frame are calculated as follows:

[0150] Let the total number of pixels in the currently acquired effective field of view be... .

[0151] Calculate the reflectance saturation S of the image: Extract the luminance channel matrix of the original RGB image. Set the pixel saturation brightness threshold value to Construct a saturated pixel indicator function :

[0152]

[0153] The image reflectance saturation S is obtained by global traversal and statistical calculation:

[0154] The larger the image reflectance saturation S, the more reflective areas there are in the RGB image, and the less reliable the Light Field Texture Structure Index (LFTI) is.

[0155] Calculate the hole ratio H of the depth map: Read the original depth map Pixels in blind areas that failed to acquire valid depth data are marked as specific invalid values. (e.g., specifically 0), construct the depth hole indicator function. :

[0156]

[0157] The porosity H of the depth map is calculated statistically:

[0158] The larger the hole rate H of the depth map, the more invalid regions there are in the depth map, and the less reliable the manifold topological curvature field MTC is.

[0159] Specifically, the adjustment methods for the light field texture weight α and the geometric curvature weight β based on the light field texture base weight α0 and the geometric curvature base weight β0 are as follows:

[0160] During system operation, the over-limit penalty for both is calculated based on the real-time calculated depth map hole rate H and image reflectance saturation S. and To quantify the degree of anomaly:

[0161] ,

[0162] ,

[0163] The logic here is that a penalty is only applied when H or S exceeds the tolerance limit; otherwise, the penalty is zero and the weight is not adjusted.

[0164] Introduce a preset system sensitivity coefficient and Define dynamic weight adjustment offset Perform a two-way linear game: where > 0, > 0, and the values ​​of both are set between 0.5 and 5.0. If the depth camera used in the current system performs poorly on specific materials (such as black, highly reflective metal), then the preset value is set to 0. > If the working environment has variable lighting and severe glare, then preset... > .

[0165]

[0166] Finally, to ensure the validity and mutual exclusivity of the weight values, the light field texture weight α and geometric curvature weight β of the current frame are updated using a limiting function:

[0167] The constraint α must be between 0 and 1.

[0168] .

[0169] Scenario 2: During the capture task execution, based on the image reflection saturation S, depth map hole rate H, and edge density change rate within the target area for the current capture period... and historical data capture success rate One or more of the metrics are used to dynamically update the light field texture weight α and the geometric curvature weight β.

[0170] Specifically, when dynamically updating the light field texture weight α and the geometric curvature weight β, a compensation amount is constructed. :

[0171] ,

[0172] in, To achieve the preset target success rate, , , and The preset adjustment coefficient, The value ranges from 0.5 to 3.0. The value ranges from 0.5 to 3.0. The value range is 0.1 to 1.5. The value ranges from 0.01 to 0.5. According to the above formula, when the hole rate H of the depth map increases, the compensation amount increases, and the system automatically increases its dependence on 2D texture. When the image reflectance saturation S or the edge density change rate within the target area increases, the compensation amount also increases. When the amount increases, it prompts the system to rely more on 3D geometric features. This enables adaptive closed-loop regulation based on multi-source information.

[0173] compensation amount Light field texture weights superimposed on the current frame To obtain the updated light field texture weights for the next frame. The next frame updates the geometric curvature weights. .

[0174] ,

[0175] .

[0176] The amount of compensation Light field texture weights superimposed on the current frame Above, the amplitude is limited to 0~1, and the updated light field texture weights for the next frame are obtained. This achieves a closed loop where the weight is optimized once for each crawl.

[0177] In step S3, the constructed Joint Light Field Structure Complexity Map (LFSC) is the final feature map that integrates 2D texture complexity and 3D geometric complexity. It can dynamically adjust the fusion weight of the two features according to the real-time environment (such as RGB reflections and depth holes) and grasping feedback (success rate and edge stability). This allows the system to rely more on 3D geometry when the RGB image quality is poor and more on 2D texture when the depth map quality is poor. Finally, a robust and scene-adaptive comprehensive complexity map is obtained, providing the most reliable basis for grasping decisions.

[0178] Step S4 above, the dynamic peak-valley energy separation algorithm, refers to an adaptive threshold segmentation method used to automatically distinguish between high-complexity target regions and low-complexity background regions in the joint optical field structure complexity map (LFSC). Specifically:

[0179] The global energy mean is calculated based on the Joint Light Field Structure Complexity Diagram (LFSC). with global energy standard deviation The boundary energy threshold T satisfies: ,in The complexity separation coefficient, The value ranges from 1 to 3, or the boundary energy threshold T satisfies: ,in For adjustment coefficients, The value range is 0.01 to 0.2. To achieve the preset target success rate, To improve the success rate of historical data collection.

[0180] This requires performing min-max normalization on the joint optical field structure complexity map LFSC, and outputting a normalized complexity map. This limits the value range to a preset interval of 0 to 1:

[0181]

[0182] in, and These are the minimum and maximum values ​​of the joint light field structure complexity map for the current frame, respectively, to eliminate the fluctuations in complexity amplitude caused by differences in ambient lighting or material properties in different scenarios.

[0183] Then analyze the normalized complexity graph. Smoothing is performed to obtain a smoothed complexity graph. .

[0184] Gaussian filtering is preferred:

[0185]

[0186] in This is a two-dimensional Gaussian kernel function. This step is used to weaken isolated noise points and enhance the spatial consistency of continuous high-complexity regions.

[0187] Then smooth the complexity graph Perform global statistics and calculations:

[0188] Global energy mean :

[0189]

[0190] Global energy standard deviation :

[0191]

[0192] in This represents the total number of pixels.

[0193] The mean and standard deviation are used as an overall reference benchmark to characterize the complexity distribution of the current joint optical field structure complexity map (LFSC).

[0194] By using global statistics, specifically the mean and standard deviation, we obtain the boundary energy threshold T of the complexity distribution in the current scenario. The boundary energy threshold T satisfies the following: Or, the boundary energy threshold T satisfies: Then, based on the boundary energy threshold T, a binary region is constructed. :

[0195] ,

[0196] For binary regions Perform connected component analysis and remove isolated regions with areas smaller than a preset threshold, outputting the final region of interest (ROI), where binary regions... The region is a highly complex target region, a binary region. =0 indicates the background area.

[0197] In step S5 above, the optimal viewpoint (NBV) is selected from the candidate viewpoints to address the issues of incomplete or inaccurate observations caused by occlusion, reflection, or depth holes in the final region of interest (ROI) from the current viewpoint. A robotic arm moves the camera to the optimal pose for rescanning, providing high-quality visual data for subsequent precise capture. The candidate viewpoints are generated by taking the current camera pose as the center and generating multiple candidate camera poses within a preset angle and distance range around it, forming a candidate viewpoint set.

[0198] The specific method for selecting the best perspective (NBV) from the candidate perspectives is as follows:

[0199] S5.1 Visibility Assessment: For each candidate viewpoint, simulate its imaging effect on the final region of interest (ROI) and statistically analyze the visible area percentage under the candidate viewpoint. :

[0200] ,

[0201] in The total number of points in the final region of interest (ROI). The number of unobstructed visible points and the percentage of visible area from the candidate viewpoint. The value ranges from 0 to 1. The closer the value is to 1, the more complete the observation of the ROI is from the viewpoint. The smaller the value is, the more severe the occlusion is.

[0202] S5.2 Quality Assessment: For each candidate viewpoint, calculate the depth map hole rate H, image reflectance saturation S, and edge sharpness E to obtain the quality score Q.

[0203] ,

[0204] in For the weighting coefficients, satisfying .

[0205] The purpose of setting up step S5.2 above is to measure the quality of the image under the candidate viewpoint. Even if the complete ROI can be seen, if the image has a lot of deep holes and reflections, the data is still useless. The edge sharpness E is obtained based on the statistical analysis of grayscale change intensity. The larger E is, the more obvious the texture features are, and the more accurate the subsequent feature extraction is.

[0206] Weighting coefficient The design logic is that if depth data is more important for grasping, such as grasping smooth curved materials, then the depth should be increased. If the ambient light is severe, increase the power. If the edge features of the material are key to the capture, then increase the value. .

[0207] The quality score Q ranges from 0 to 1. The closer the value is to 1, the better the image quality.

[0208] S5.3 Overall Score: Percentage of Visible Area By integrating the candidate perspective score with the quality score Q, a comprehensive score is obtained. :

[0209] ,

[0210] in .

[0211] Weight and The design logic is as follows: if the current ROI is severely occluded, increase the size to ensure visibility. If the current ROI is visible but the image quality is poor, the adjustment should be increased to prioritize clarity. .

[0212] S5.4 Determining the Optimal Viewpoint NBV: Selecting candidate viewpoints and comprehensively evaluating them. The highest candidate viewpoint is the best viewpoint NBV.

[0213] In some embodiments, step S6 above specifically uses a 6D pose estimation algorithm. 6D pose refers to the 3D position of the workpiece, such as the X-axis, Y-axis and Z-axis, and the 3D attitude, such as roll angle, pitch angle and yaw angle. The goal of this step is to accurately calculate the optimal grasping pose of the workpiece to be grasped from the mixed point cloud.

[0214] The present invention also provides a robot grasping system driven by light field structural complexity, configured to execute the robot grasping method driven by light field structural complexity of the present invention.

[0215] In some embodiments, the present invention has the following technical effects:

[0216] a. Active observation capability: Unlike traditional fixed-view scanning, this invention actively seeks the optimal viewing angle NBV, effectively solving the problems of occlusion and reflection blind spots under a single viewing angle;

[0217] b. Perfect balance between efficiency and accuracy: Compared with full-resolution scanning, this method only performs fine scanning on complex areas, significantly reducing the overall processing time.

[0218] c. Possesses online evolution capability: Through the feedback loop of the captured results, the system can automatically learn the optimal structural complexity fusion weight, and the system's robustness becomes stronger as the running time increases.

[0219] This invention relates to a robot grasping system driven by the complexity of light field structures, applicable to scenarios such as parts loading and unloading, precision component assembly, and package depalletizing and sorting. Specifically, in some embodiments, this invention is widely applicable to industrial scenarios such as loading and unloading of highly reflective metal parts in automobile manufacturing, assembly of tiny precision components in the 3C electronics field, and depalletizing and sorting of irregularly shaped / dark-colored packages in logistics warehousing. It avoids specular reflection interference from metal surfaces through an active observation mechanism, and utilizes adaptive multi-resolution scanning to accurately extract minute features while ensuring extremely fast production cycles. Furthermore, it can automatically switch to texture-driven mode for black light-absorbing or transparent packages when depth data is missing. This effectively solves the core pain points of robot visual perception—unclear visibility, inaccurate calculations, and inability to grasp—under complex working conditions, demonstrating strong industrial application value and environmental adaptability.

[0220] The above description is only a specific embodiment of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the principle of this application, and these improvements and modifications should also be considered within the scope of protection of this application.

Claims

1. A robot grasping method based on light field structure complexity driving, characterized in that, Includes the following steps: The Light Field Texture Structure Index (LFTI) is constructed based on RGB images. It is calculated by fusing image edge density and local contrast features to reflect the roughness and irregularity of the target surface at the visual level. The manifold topology curvature field (MTC) is constructed based on the depth map. It is obtained by fusing the local curvature changes and normal vector change rates of the target surface. It is used to reflect the degree of undulation and stacking edge strength of the target region in the spatial geometry. Construct the joint optical field structure complexity graph LFSC that satisfies the following relationship: LFSC = α · LFTI + β · MTC, Where α is the light field texture weight and β is the geometric curvature weight, satisfying α+β=1; The Joint Light Field Structure Complexity Map (LFSC) of the current frame is processed based on the Dynamic Peak-Valley Energy Separation Algorithm. The boundary energy threshold T between the background and the stacked objects is distinguished. Regions in the LFSC with energy values ​​greater than the boundary energy threshold T are defined as high-complexity regions. Connectivity analysis or morphological processing is performed on the high-complexity regions to remove isolated noise blocks and output the final region of interest (ROI). Candidate viewpoints are constructed based on the output final region of interest (ROI). The best viewpoint (NBV) is selected from the candidate viewpoints, and the robotic arm is controlled to move to the corresponding pose to perform a high-resolution secondary scan of the final ROI to obtain a local high-precision point cloud. The obtained local high-precision point cloud is registered and fused with the global background point cloud to construct a multi-resolution hybrid point cloud scene. A pose estimation algorithm is run on the hybrid point cloud scene to calculate the optimal grasping pose of the workpiece to be grasped and then grasping is performed.

2. The robot grasping method based on light field structure complexity driving according to claim 1, wherein: When constructing the Joint Light Field Structure Complexity Map (LFSC), a basic weight α0 for the light field texture and a basic weight β0 for the geometric curvature are set, satisfying α0 + β0 = 1. An upper limit for reflectivity saturation tolerance Ts1 and an upper limit for depth hole rate tolerance Th1 are set. The reflectivity saturation S and depth map hole rate H of the current frame are calculated. When S > Ts1, the light field texture weight α0 is reduced to obtain the light field texture weight α, and the geometric curvature weight β0 is increased to obtain the geometric curvature weight β. When H > Th1, the light field texture weight α0 is increased to obtain the light field texture weight α, and the geometric curvature weight β0 is reduced to obtain the geometric curvature weight β.

3. The robot grasping method based on light field structural complexity as described in claim 1 or 2, characterized in that: During the capture task execution, the image reflection saturation S, depth map hole rate H, and edge density change rate within the target area are used as the data for the current capture period. and historical data capture success rate One or more of the indicators are used to dynamically update the light field texture weight α and the geometric curvature weight β.

4. The robot grasping method based on light field structure complexity as described in claim 3, characterized in that: When dynamically updating the light field texture weight α and the geometric curvature weight β, a compensation amount is constructed. : ， in, To achieve the preset target success rate, , , and The preset adjustment coefficient, The value ranges from 0.5 to 3.

0. The value ranges from 0.5 to 3.

0. The value range is 0.1 to 1.

5. The value range is 0.01 to 0.5; The compensation amount Light field texture weights superimposed on the current frame To obtain the updated light field texture weights for the next frame. .

5. The robot grasping method based on light field structure complexity as described in claim 1, characterized in that: The dynamic peak-valley energy separation algorithm is as follows: The global energy mean is calculated based on the Joint Light Field Structure Complexity Diagram (LFSC). with global energy standard deviation The boundary energy threshold T satisfies: ,in The complexity separation coefficient, The value ranges from 1 to 3, or the boundary energy threshold T satisfies: ,in For adjustment coefficients, The value range is 0.01 to 0.

2. To achieve the preset target success rate, To improve the success rate of historical data collection.

6. The robot grasping method based on light field structure complexity as described in claim 1, characterized in that: The specific method for selecting the best viewpoint (NBV) from the candidate viewpoints is as follows: Visibility assessment: For each candidate viewpoint, simulate its imaging effect on the final region of interest (ROI) and calculate the visible area percentage under the candidate viewpoint. : ， in The total number of points in the final region of interest (ROI). The number of unobstructed visible points under the candidate viewpoint; Quality assessment: For each candidate viewpoint, calculate the depth map hole rate H, image reflectance saturation S, and edge sharpness E to obtain a quality score Q. ， in For the weighting coefficients, satisfying ; Overall score: The percentage of the visible area. The candidate viewpoint comprehensive score is obtained by fusing the quality score Q. : ， in ; Optimal Viewpoint (NBV) Determination: Selection of Candidate Viewpoints Based on Comprehensive Scoring The highest candidate viewpoint is the optimal viewpoint NBV.

7. The robot grasping method based on light field structural complexity as described in claim 1, characterized in that, The construction steps of the Light Field Texture Structure Index (LFTI) are as follows: RGB image preprocessing: Convert the RGB image to a grayscale image; Edge density extraction: Within a sliding window W centered at (x,y) and of size N xM, the density of the contour lines is statistically analyzed. To quantify the disorder of the material stacking and the difficulty of grasping: ， in This is the sum of the edge values ​​of all pixels in the sliding window W; Local contrast feature extraction: in the area of ​​size Within the sliding window W, calculate the local grayscale mean values ​​respectively. Local grayscale standard deviation Based on the local grayscale mean and the local grayscale standard deviation Construct the local contrast feature C(x,y): ； Feature fusion constructs the light field texture structure index LFTI: the density of the contour lines. Mapping to the range of 0 to 1 using the min-max normalization method, denoted as... The local contrast feature C(x,y) is mapped to the range of 0~1 using the min-max normalization method, denoted as... Set preset weight coefficients , And satisfy Weighted fusion yields the light field texture structure index LFTI of the sliding window W, satisfying: 。 8. The robot grasping method based on light field structural complexity as described in claim 7, characterized in that: The density of the outline When using the minimum-maximum normalization method, the following conditions must be met: 。 9. The robot grasping method based on light field structure complexity as described in claim 7, characterized in that: When the local contrast feature C(x,y) is subjected to the minimum-maximum normalization method, the following conditions are met: 。 10. The robot grasping method based on light field structural complexity as described in claim 1, characterized in that, The construction steps of the manifold topological curvature field MTC based on the depth map are as follows: Depth map preprocessing: After preprocessing the depth map, a smooth depth map is output, denoted as D(x,y); Local curvature feature extraction: In the smoothed depth map D(x,y), the second-order depth divergence between the current pixel D(x,y) and its four orthogonal neighboring pixels (up, down, left, and right) is calculated using the discrete Laplacian operator. : ， To combat interference from single-point noise in the sensor, a method centered at (x,y) is used. Define local curvature features within the sliding window W. : ； Extraction of normal rate of change feature: setting points The normal vector is any point within the neighborhood The normal vector is , definition point arrive The Euclidean distance in two-dimensional space is Calculate the characteristic of the normal rate of change : ， in, It is the cosine similarity of the normal vectors, where e is the natural constant 2.71828; Feature fusion to construct manifold topological curvature field MTC: for the local curvature features After normalization, we get For the normal rate of change characteristic After normalization, we get Preset geometric weight parameters ,satisfy = 1, thus obtaining the topological curvature field MTC of the manifold, satisfying: 。 11. The robot grasping method based on light field structure complexity as described in claim 10, characterized in that: For the local curvature features When performing normalization, the following conditions must be met: 。 12. The robot grasping method based on light field structure complexity as described in claim 10, characterized in that: For the normal rate of change characteristic When performing normalization, the following conditions must be met: 。 13. A robot grasping system driven by light field structural complexity, characterized in that: It is configured to perform the robot grasping method based on light field structure complexity as described in any one of claims 1-12.

14. The robot grasping system based on light field structural complexity as described in claim 13, characterized in that, It is used in scenarios such as loading and unloading parts, assembling precision components, or unpacking and sorting packages.

Images