Task-oriented grasping method and system for cross-hierarchy constraint reasoning

Abstract

The application provides a task-oriented grasping method and system for cross-level constraint reasoning, comprising: acquiring a natural language instruction containing a task target and a scene RGB image, and outputting a target object mask corresponding to the task target; jointly inputting the target object mask, the RGB image and the natural language instruction into a visual language large model for joint reasoning, and outputting a functional area mask indicating a suitable grasping position on the surface of an object; acquiring a depth image of the scene, registering the depth image with the RGB image to generate a scene point cloud, predicting and outputting a probability that each point in the target point cloud belongs to a high grasping confidence area; and generating a candidate grasping pose based on the obtained high grasping confidence area. The application greatly shortens the training and reasoning time of point cloud target detection, and adapts to the target recognition needs of complex stacked scenes.

Description

Technical Field

[0001] This invention belongs to the field of robot grasping control, and specifically relates to a task-oriented grasping method and system with cross-level constraint reasoning. Background Technology Enhancing the autonomous grasping capabilities of robots in complex environments is of great significance for improving production efficiency, reducing labor costs, and realizing smart logistics. The combination of large models and robots is a current research hotspot. In open-world scenarios, robot goal-oriented grasping tasks need to meet three core requirements: semantic understanding, depth perception, and accurate generation of grasping posture.

[0002] Existing point cloud object detection networks suffer from complex structures and long training times. In the field of unordered grasping robots, there is a need to solve the problems of building lightweight and efficient point cloud recognition networks and generating 6-DOF grasping postures in complex stacked environments. Furthermore, traditional methods and mainstream grasping frameworks have obvious limitations.

[0003] The current approach combines large models with robots, leveraging the analytical and reasoning capabilities of large models to understand multi-form instructions, decompose complex tasks, and integrate multimodal information to simplify the processing flow. Meanwhile, point cloud target detection networks are used for target recognition, and mainstream grasping frameworks use single-modal perception or simple multimodal stitching to handle TOG tasks.

[0004] Existing point cloud object detection networks are complex in structure and time-consuming to train; traditional fixed logic code methods have poor generalization ability, cannot handle unfamiliar multi-form instructions, and cannot integrate multimodal information; mainstream grasping frameworks are difficult to achieve deep integration of high-level knowledge and low-level geometric perception, and have defects such as disconnect between task understanding and scene perception, lack of constraints in grasping posture generation, and lack of cross-level reasoning ability. Summary of the Invention

[0005] The purpose of this invention is to provide a task-oriented grasping method and system for cross-level constraint reasoning to solve the above-mentioned problems.

[0006] To achieve the above objectives, the present invention adopts the following technical solution: In a first aspect, the present invention provides a task-oriented crawling method for cross-level constraint reasoning, comprising: Acquire RGB images of natural language instructions and scene containing task objectives. Preprocess the RGB images of natural language instructions and scene to obtain text features and object masks respectively. Calculate the similarity between text features and visual features of each object mask region, and output the target object mask corresponding to the task objective. The target object mask, RGB image and natural language instructions are jointly input into the visual language big model for joint reasoning, and the output is a functional area mask that indicates the suitable gripping position on the object surface. Acquire a depth image of the scene and register it with an RGB image to generate a scene point cloud; use a target object mask to crop out the target point cloud of the target object from the scene point cloud; construct a grasping region recognition network with a functional region mask as a spatial prior, process the target point cloud, predict and output the probability that each point in the target point cloud belongs to a high grasping confidence region. Based on the obtained high-confidence grasping region, candidate grasping poses are generated. The high-level functional priors represented by the functional region mask and the low-level geometric features of the candidate grasping poses are fused together. Collision detection and quality scoring are performed on each candidate grasping pose. The grasping pose with the best score is selected as the output result to drive the actuator to complete the grasping operation.

[0007] Furthermore, the step of acquiring RGB images of natural language instructions and scenes containing the task target, preprocessing the RGB images of natural language instructions and scenes to obtain text features and object masks respectively, calculating the similarity between the text features and the visual features of each object mask region, and outputting the target object mask corresponding to the task target includes: In the visual language encoding stage, SAM is first used to spatially encode objects in the input image, and CLIP is used to align text features with spatial codes pixel by pixel to generate a pixel-level target region mask.

[0008] Furthermore, specifically, SAM is used as the backbone of the visual encoder to spatially encode objects within the captured scene, resulting in a series of pixel-level mask regions. Then, CLIP is used as a text encoder to encode semantic instructions into feature vectors according to text categories. Simultaneously, visual features are extracted pixel-by-pixel for each mask region. Finally, the semantic category features and visual features are mapped to the same space, and a cosine similarity score is calculated, as shown in the following formula:

[0009] in, Represents cosine similarity. For the first Text feature vectors for the task. For the first The pixel-by-pixel visual feature vector of an object mask region For temperature coefficient, This represents the total number of valid object mask regions obtained through SAM encoding in the scene. For the first The normalized matching score between the object mask region and the task text features.

[0010] Furthermore, the step of jointly inputting the target object mask, RGB image, and natural language instructions into the visual language large model for joint inference, and outputting a functional region mask indicating a suitable grasping position on the object surface, includes: By leveraging the decision module of VLM as a task-oriented crawling methodology framework, and jointly reasoning about multimodal data crawling scenarios and semantic information using context-based target masks, high-level knowledge priors are generated to guide downstream crawling tasks. Specifically, the shape and structure of the target object are categorized into edge-guided or surface-guided grasping objects based on task semantic information to guide the underlying grasping perception. Secondly, starting from human grasping habits, a context-based target mask is used as the spatial geometric constraint of the Virtual Model (VLM) to perform cross-level joint reasoning on the task text and scene images, generating a functional region mask. Finally, the functional region mask and semantic pixel mask regions are unified within a shared embedding space, achieving a unified representation of geometric structure semantics and functional masks. This provides a more refined spatial prior for subsequent cross-level grasping reasoning, as expressed in the following formula:

[0011] in, For local function masks, For functional area mask, For context-based target masks.

[0012] Furthermore, the process involves acquiring a depth image of the scene and registering it with an RGB image to generate a scene point cloud; using a target object mask to crop out the target point cloud from the scene point cloud; and constructing a grasping region recognition network with a functional region mask as a spatial prior to process the target point cloud, predicting and outputting the probability that each point in the target point cloud belongs to a high grasping confidence region, including: Grasp LAKAN, a grasping region recognition network with La-GeoKan as its core, introduces pixel-level mask regions as high-level spatial priors, constraining global indiscriminate inference to make effective predictions for high-confidence grasping regions of target mask point clouds. Specifically, the La-GeoKan point cloud encoding network is used to extract features from regions of semantic target with a high probability of successful capture. Firstly, it uses points... Delineating spherical regions of different sizes around the center, the formula is expressed as:

[0013] in, Indicates the current level The surrounding set of points, The base region radius is s, where s is the scaling factor used to adjust the region size at different levels; secondly, at each point... neighborhood Within, the most critical information is extracted by concatenating the information from surrounding points. The formula is as follows:

[0014] in, This indicates the position information of surrounding points relative to the center point. Initial information for surrounding points, This indicates a feature concatenation operation. This indicates retrieving key information within a specified range. To integrate the information dimensions of each point; based on the global-local feature fusion mechanism, firstly, for each local feature... Perform non-linear processing to extract detailed shape information of the object. The formula is expressed as:

[0015] in, It is a third-order B-spline function. For adjustable parameters, The information transformation matrix is ​​designed, and adjustable weighting coefficients are constructed. This is used to highlight information in important areas, and then the local information of all points is integrated into global information. And perform linear processing to preserve overall features. The formula is expressed as:

[0016] in, For linear basis functions, For adjustable parameter matrices; finally, add global information. and processed local information By merging, the final feature information is obtained. : .

[0017] Furthermore, based on the obtained high-confidence grasping region, candidate grasping postures are generated. The high-level functional priors represented by the functional region mask are fused with the low-level geometric features of the candidate grasping postures. Collision detection and quality scoring are performed on each candidate grasping posture, and the grasping posture with the best score is selected as the output result to drive the actuator to complete the grasping operation, including: By combining high-level functional priors with low-level geometric priors, collision detection and quality scoring are performed on the candidate pose set to ensure the feasibility and stability of grasping in an open world, and output a stable 6-DOF grasping pose. Finally, a visual feedback mechanism is added to return the robot to the initial image position and capture the RGB of the scene after the grasping task is performed as visual information to be fed back to the VLM to determine whether the target object exists in the original scene, thereby determining whether the grasping task is successful or not, and constructing a closed-loop grasping framework of "perception-decision-action-feedback".

[0018] Furthermore, specifically including: First, the OBB algorithm is used to filter out center and edge points from the high-confidence region for crawling. Then, combined with existing physical knowledge, suitable crawling points are automatically determined. Then, perform geometric analysis on the determined gripping points to calculate the suitable rotation angle for the parallel grippers, using a rotation matrix. This means that by combining the rotation angle and the gripping point, a set of multiple candidate gripping postures can be obtained. Secondly, collision detection was added to the candidate poses: a cylindrical area was constructed with the gripping point as the center and the width of the gripper opening as the diameter—if there was anything other than the target object within this area, the pose was excluded, ensuring that the remaining poses were safe; subsequently, this was combined with a functional mask. The grasping posture is evaluated based on the distance from the grasping point to the object's center of gravity. From the perspective of human grasping habits, the most suitable grasping posture is selected. The scoring rules are as follows:

[0019] in, The score represents the distance between the grab point and the center of mass of the object, and reflects how easy or difficult it is to grab the object from that position. The furthest point from the centroid within the high-scratching confidence region The distance; The flag for "whether the grasping posture is in the appropriate area": ​​if the posture is in the preset "suitable grasping area", it is marked as 1; otherwise, it is marked as 0. These are weighting coefficients used to adjust the importance of "region suitability" in the scoring; finally, the posture with the highest score is selected as the final grasping posture. .

[0020] Secondly, the present invention provides a task-oriented grasping system for cross-level constraint reasoning, comprising: The data acquisition module is used to acquire RGB images of natural language instructions and scenes containing task objectives, preprocess the RGB images of natural language instructions and scenes to obtain text features and object masks respectively, calculate the similarity between text features and visual features of each object mask region, and output the target object mask corresponding to the task objective. The inference module is used to input the target object mask, RGB image and natural language instructions into the visual language big model for joint inference, and output a functional area mask that indicates the suitable gripping position on the object surface. The prediction module is used to acquire the depth image of the scene and register it with the RGB image to generate the scene point cloud; the target point cloud of the target object is cropped from the scene point cloud using the target object mask; a grasping region recognition network with the functional region mask as the spatial prior is constructed to process the target point cloud, predict and output the probability that each point in the target point cloud belongs to a high grasping confidence region. The output module is used to generate candidate grasping poses based on the obtained high grasping confidence region, fuse the high-level functional prior represented by the functional region mask and the low-level geometric features of the candidate grasping poses, perform collision detection and quality scoring on each candidate grasping pose, select the grasping pose with the best score as the output result, and drive the actuator to complete the grasping operation.

[0021] Thirdly, the present invention provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of the task-oriented grasping method for cross-level constraint reasoning.

[0022] Fourthly, the present invention provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the task-oriented grasping method for cross-level constraint reasoning.

[0023] Compared with the prior art, the present invention has the following technical effects: This invention achieves pixel-level alignment of textual and visual features through the collaboration of SAM and CLIP, enabling rapid localization of target object masks and simplifying the target recognition process. It employs a lightweight grasping region recognition network, Grasp LAKAN, with La-GeoKan at its core, leveraging a global-local feature fusion mechanism to efficiently predict high-confidence grasping regions, replacing traditional complex point cloud networks and reducing model complexity and training time. Simultaneously, it utilizes a large visual language model to perform multimodal joint inference of natural language commands, RGB images, and target masks, enabling the understanding of diverse task commands and the fusion of multimodal information, overcoming the generalization limitations of traditional fixed logic schemes. This significantly shortens the training and inference time for point cloud target detection, adapting to the target recognition needs of complex stacked scenarios. It possesses powerful multimodal fusion and unknown command adaptation capabilities, effectively meeting the core needs of robot semantic understanding and depth perception in open worlds, and simplifying the grasping task processing flow.

[0024] This invention utilizes a large visual language model to generate functional region masks as high-level spatial priors, embedding them into the grasping region recognition and pose generation stages to construct a cross-level constraint reasoning mechanism. In the pose generation stage, it integrates the high-level functional priors of the functional region masks with the low-level geometric features of candidate poses, combining collision detection and customized quality scoring to select the optimal 6-DOF grasping pose. Simultaneously, it establishes a "perception-decision-action-feedback" visual closed-loop framework, verifying the grasping completion rate through visual feedback. This achieves deep integration of high-level functional knowledge and low-level geometric perception, solving the problem of disconnect between task understanding and scene perception, providing precise spatial constraints for grasping poses, and endowing the framework with strong cross-level reasoning capabilities. It significantly improves the accuracy, stability, and executability of robot grasping poses in complex environments, forming a complete autonomous grasping closed loop, comprehensively enhancing the autonomous grasping capabilities of robots in open worlds, and providing technical support for improving production efficiency, reducing labor costs, and realizing smart logistics. Attached Figure Description

[0025] Figure 1 This is a diagram of the overall architecture of the present invention.

[0026] Figure 2 This is a schematic diagram of the LFPG of the present invention in the point cloud perception stage.

[0027] Figure 3 This is an example of the mask region of the present invention.

[0028] Figure 4 This is an example of the functional area of ​​the present invention.

[0029] Figure 5 This invention describes the process for generating the grasping posture.

[0030] Figure 6 This is a flowchart of the present invention. Detailed Implementation

[0031] The present invention will be further described below with reference to the accompanying drawings: Example 1, please refer to Figure 1 This invention provides a task-oriented crawling method for cross-level constraint reasoning, comprising: Acquire RGB images of natural language instructions and scene containing task objectives. Preprocess the RGB images of natural language instructions and scene to obtain text features and object masks respectively. Calculate the similarity between text features and visual features of each object mask region, and output the target object mask corresponding to the task objective. The target object mask, RGB image and natural language instructions are jointly input into the visual language big model for joint reasoning, and the output is a functional area mask that indicates the suitable gripping position on the object surface. Acquire a depth image of the scene and register it with an RGB image to generate a scene point cloud; use a target object mask to crop out the target point cloud of the target object from the scene point cloud; construct a grasping region recognition network with a functional region mask as a spatial prior, process the target point cloud, predict and output the probability that each point in the target point cloud belongs to a high grasping confidence region. Based on the obtained high-confidence grasping region, candidate grasping poses are generated. The high-level functional priors represented by the functional region mask and the low-level geometric features of the candidate grasping poses are fused together. Collision detection and quality scoring are performed on each candidate grasping pose. The grasping pose with the best score is selected as the output result to drive the actuator to complete the grasping operation.

[0032] This invention significantly shortens the training and inference time for point cloud target detection, adapts to the target recognition needs of complex stacked scenarios, and has powerful multimodal fusion and unknown instruction adaptation capabilities, effectively meeting the core needs of robot semantic understanding and depth perception in open worlds, and simplifying the grasping task processing flow.

[0033] Example 2: This invention provides a task-oriented crawling method for cross-level constraint reasoning, comprising: In the visual language encoding stage, SAM is first used to spatially encode objects in the input image, and CLIP is used to align text features with spatial codes pixel by pixel to generate a pixel-level target region mask. In the high-level functional area reasoning stage, VLM is used to perform joint reasoning on multimodal data including semantic information, original images and their masks to generate high-level knowledge priors to guide downstream crawling tasks. In the underlying grasping and perception stage, GraspLAKAN, a grasping region recognition network with La-GeoKan as its core, is proposed. By introducing pixel-level mask regions as high-level spatial priors, global indiscriminate reasoning is constrained to be an effective prediction of high-confidence grasping regions for target mask point clouds. During the pose generation stage, high-level functional priors and low-level geometric priors are combined to perform collision detection and quality scoring on the candidate pose set, ensuring the executability and stability of grasping in an open world, and outputting a stable 6-DOF grasping pose. Finally, a visual feedback mechanism is added to allow the robot to return to the initial photo-taking position, capture the RGB of the scene after the grasping task is completed, and use this as visual information to feed back to the VLM. This determines whether the target object exists in the original scene, thereby determining whether the grasping task was successful or not, thus constructing a closed-loop grasping framework of "perception-decision-action-feedback".

[0034] A "visual-text" encoder was constructed to parse natural semantics and visual images, mapping high-level task instructions into pixel-level mask regions, providing accurate spatial priors for downstream grasping and perception.

[0035] Specifically, SAM is used as the backbone of the visual encoder to spatially encode objects within the captured scene, resulting in a series of pixel-level mask regions. Then, CLIP is used as a text encoder to encode semantic instructions into feature vectors according to text categories. Simultaneously, visual features are extracted pixel-by-pixel for each mask region. Finally, the semantic category features and visual features are mapped to the same space, and a cosine similarity score is calculated. The formula is as follows:

[0036] in, This represents the cosine similarity.

[0037] By utilizing the decision module of VLM as a task-oriented crawling method framework, and jointly reasoning about multimodal data crawling scenarios and semantic information based on context-based target masks, high-level knowledge priors are generated to guide downstream crawling tasks.

[0038] Specifically, the shape and structure of the target object are categorized into edge-guided or surface-guided grasping objects based on task semantic information to guide the underlying grasping perception. Secondly, starting from human grasping habits, a context-based target mask is used as the spatial geometric constraint of the Virtual Model (VLM) to perform cross-level joint reasoning on the task text and scene images, generating a functional region mask. Finally, the functional region mask and semantic pixel mask regions are unified within a shared embedding space, thereby achieving a unified representation of geometric structure semantics and functional masks. This provides a more refined spatial prior for subsequent cross-level grasping reasoning, as expressed in the following formula:

[0039] in, For local function masks, For functional area mask, For context-based target masks.

[0040] LFPG in the point cloud perception stage (e.g.) Figure 2 ) This invention constructs a point cloud encoding network called La-GeoKan to extract features from regions of semantic target with a high probability of successful capture, thereby improving the system's understanding of physical laws. This network adopts a modular design approach common in point cloud processing and replaces and simplifies the traditional multilayer perceptron (MLP) stacked structure: the shallow layers use a Set Abstraction (SA) module to filter basic information and integrate spatially, eliminating redundant computational layers; a fusion-based feature enhancement module is introduced in the mid-to-deep layers to enhance the modeling ability of complex geometric shapes and alleviate the problem of weakened feature representation in deep networks.

[0041] Specifically, firstly, starting with points Delineating spherical regions of different sizes around the center, the formula is expressed as:

[0042] in, Indicates the current level The surrounding set of points, The base region radius is s, and s is a scaling factor used to adjust the region size at different levels. Secondly, at each point... neighborhood Within, the most critical information is extracted by concatenating the information from surrounding points. The formula is as follows:

[0043] in, This indicates the position information of surrounding points relative to the center point. Initial information for surrounding points, This indicates a feature concatenation operation. This indicates retrieving key information within a specified range. This relates to the information dimensions of each point after integration. Furthermore, this invention proposes a global-local feature fusion mechanism, first processing each local feature... Perform non-linear processing to extract detailed shape information of the object. The formula is expressed as:

[0044] in, It is a third-order B-spline function. For adjustable parameters, This is an information transformation matrix. Adjustable weighting coefficients are designed. This is used to highlight information in important areas, thereby enhancing the network's ability to focus on key local areas. Then, the local information from all points is integrated into global information. And perform linear processing to preserve overall features. The formula is expressed as:

[0045] in, For linear basis functions, This is an adjustable parameter matrix. Finally, the global information is... and processed local information By merging, the final feature information is obtained. :

[0046] Through the above design, nonlinear processing can preserve local details and reduce redundant calculations, while global information can be combined to enhance the recognition of the overall shape of the object, thereby improving the generalization ability and physical perception ability of the grasping system, and ultimately improving the information processing ability of the "easy grasping area".

[0047] This invention proposes a cross-level joint grasping reasoning mechanism—combining "experienced knowledge" and "actually perceived information" to comprehensively evaluate multiple candidate grasping postures and ultimately select the optimal grasping posture. This mechanism mimics the "active adjustment" characteristic of human vision, establishing a processing flow similar to the human "see first, then understand," enabling machines to understand objects and determine grasping methods like humans do.

[0048] Specifically, the OBB algorithm is first used to filter out center and edge points from the high-confidence grasping region. Then, combined with existing physics knowledge (such as how to grasp an object), the appropriate grasping points are automatically determined. Then, perform geometric analysis on the determined gripping points to calculate the appropriate rotation angle for the parallel grippers (using a rotation matrix). (This is represented as a table), and then the rotation angle and the gripping point are combined to obtain a set of multiple candidate gripping postures. Secondly, to avoid collisions with other objects during grasping, we added "collision detection" to the candidate poses: a cylindrical area is constructed with the grasping point as the center and the width of the gripper opening as the diameter—if anything other than the target object exists within this area, the pose is excluded, ensuring that the remaining poses are safe. Subsequently, this is combined with a function mask. We evaluate the grasping posture using pre-defined spatial rules and the distance from the grasping point to the object's center of gravity, selecting the most suitable posture based on human grasping habits. To select the most suitable grasping posture, we designed a scoring rule, as follows:

[0049] in, The score represents the distance between the grab point and the object's center of mass, reflecting how easy or difficult it is to grab the object from that position. The furthest point from the centroid within the high-scratching confidence region The distance; The flag indicating whether the grasping posture is within the appropriate area: if the posture is within the preset "suitable grasping area", it is marked as 1; otherwise, it is marked as 0. This is a weighting coefficient used to adjust the importance of "region suitability" in the scoring. Finally, we select the posture with the highest score as the final grasping posture, i.e. This scoring method combines considerations of "stable grip" and "conformity to functional area habits," ensuring that the machine can successfully complete the grasping action while making the grasping method similar to human grasping habits.

[0050] This invention discloses a task-oriented grasping method framework with cross-level constraint reasoning, belonging to the field of robot grasping control. The entire framework connects four different levels of tasks: "language parsing - functional reasoning - point cloud perception - pose generation" (LFPG). In the visual language encoding stage, SAM-Clip is used to jointly encode the task text and visual input, mapping high-level task instructions to pixel-level mask regions and generating target region masks, providing accurate spatial priors for lower-level tasks. In the functional region reasoning stage, VLM is used to jointly reason on multimodal data, including masks, generating high-level knowledge priors to guide downstream grasping tasks. In the point cloud perception stage, Grasp LAKAN, a grasping region recognition network with La-GeoKan as its core, is used. By introducing the target mask region as a high-level spatial prior, the globally indiscriminate reasoning is constrained to effective prediction of high-confidence grasping regions for the target mask point cloud. In the pose generation stage, the high-level functional priors are combined with the low-level geometric priors to perform collision detection and quality scoring on the candidate pose set, ensuring grasping executability and stability in an open world. Subsequently, the target grasping task is executed by combining the control function code and the grasping path. Finally, a visual feedback mechanism is added to return the robot to the initial image-taking position, capture the RGB image of the scene after the grasping task is completed, and input this RGB image into the VLM. The VLM is then used to determine whether the grasping task was successful, thus realizing a closed-loop grasping framework of "perception-decision-action-feedback".

[0051] Example 3 proposes a task-oriented grasping method framework with cross-level constraint reasoning. The entire framework consists of a two-stage collaborative design of multimodal feature reasoning and grasping posture generation, which is based on "language parsing-functional reasoning-point cloud perception-pose generation" (LFPG). This achieves deep integration of high-level knowledge priors and low-level geometric perception, improves the robot's deep perception of grasping tasks and scenarios, and ensures efficient and stable execution of open-world TOG tasks. The LFPG crawling framework uses "cross-level joint reasoning under high-level spatial constraints" as its core logic. It consists of two main modules: a multimodal feature reasoning stage and a crawling pose generation stage. These two stages achieve information exchange and constraint transfer through "target region masks." The overall architecture is as follows: Figure 1 As shown.

[0052] This embodiment consists of two parts: a multimodal feature inference stage and a grasping pose generation stage. The LFPG multimodal feature inference stage specifically includes: The target mask region generation module first performs spatial encoding on objects in the input image using SAM, and then uses CLIP to align text features with the spatial encoding pixel by pixel to generate a contour-aligned target region mask.

[0053] The visual perception module based on high-level spatial prior constraints introduces a mask region as a high-level spatial prior, constraining global indiscriminate reasoning to predictions within the target mask region. This significantly reduces the detection range, improves the attention of the crawling backbone network to semantic targets, and enhances the framework's perception of the crawling scenario.

[0054] The cross-level functional area joint reasoning module integrates cross-modal data and uses VLM to perform cross-level joint reasoning with task text and scene images, using high-level target masks as spatial geometric constraints. It encodes the geometric topology of the target object into a physically aware prefix label (edge-guided grasping object or surface-guided grasping object) and generates a local functional mask, allowing the framework to reason about the functional area and geometric topology of the target object.

[0055] The cross-level grasping and reasoning stage specifically includes: PointKan, a lightweight target recognition network combining multi-scale point cloud feature fusion, includes a data processing module, a multi-scale feature extraction module, a target local feature extraction module, and a decoding module connected in sequence. Specifically, the grasping detection model may include: The data processing module uses normalization and centering methods on the input point cloud to reduce the model's sensitivity to changes in data size and the complexity of adapting to different data distributions.

[0056] The multi-scale feature extraction module uses the FPS farthest point sampling strategy to ensure that the sampling points are evenly distributed in space. Each sampling point is fused with the feature information of its neighboring points at different scales through MLP, thereby capturing the global structural information of the input point cloud. The residual layer is used to avoid the gradient vanishing problem.

[0057] The target local feature extraction module consists of a Local Feature Aggregation (LA) layer and a KAN linear layer. The LA layer captures the local geometric features around each point, providing raw local information for subsequent deep feature extraction. The KAN linear layer performs feature transformation and point-by-point aggregation on the target point cloud. Finally, a convolutional module is used to refine the output feature dimensions.

[0058] The decoder module uses an FPS (Frequency Per Second) strategy to sample a subset of points from high-level point cloud data, serving as the basis for upsampling. Based on the coordinates of these sampled points, it interpolates the high-level feature representation to obtain feature values ​​at the corresponding locations. The interpolated feature values ​​are then concatenated with the low-level feature representation to generate a low-level feature representation that incorporates information from the high-level data.

[0059] The 6-DOF grasping module, guided by geometric topology, categorizes objects into edge-guided objects and planar-guided objects based on their internal regional geometry. It then utilizes a structure-aware prefix tagging guidance framework to perform geometric perception of the semantic target. Under the constraint of the target mask region code, the distribution of planar grasping positions follows a consistent distribution along the object's principal axis. This principal direction of the planar grasping position distribution geometrically constrains the grasping rotation direction, suppressing rotational degradation caused by the grasping posture being parallel to the target object, thus achieving a more robust 6-DoF posture.

[0060] The cross-level grasping quality joint evaluation module combines high-level functional priors with low-level geometric perception to jointly evaluate grasping quality. To ensure collision-free grasping, a geometric collision detection mechanism is introduced into the candidate pose set: a cylindrical collision model is constructed with the optimized grasping position as the center and the gripper opening distance as the diameter. If there are obstacle point clouds in the space other than the target object, the pose is identified as a collision pose and removed, eliminating grasping poses with potential collision risks.

[0061] The visual feedback module based on VLM (Visual Learning Model) constructs a closed-loop grasping framework of "perception-decision-action-feedback" by taking a picture at the initial position and inputting the RGB values ​​into VLM. VLM then determines whether the target exists in the scene.

[0062] In another embodiment of the present invention, a task-oriented crawling system for cross-level constraint reasoning is provided, which can be used to implement the above-mentioned task-oriented crawling method for cross-level constraint reasoning. Specifically, the system includes: The data acquisition module is used to acquire RGB images of natural language instructions and scenes containing task objectives, preprocess the RGB images of natural language instructions and scenes to obtain text features and object masks respectively, calculate the similarity between text features and visual features of each object mask region, and output the target object mask corresponding to the task objective. The inference module is used to input the target object mask, RGB image and natural language instructions into the visual language big model for joint inference, and output a functional area mask that indicates the suitable gripping position on the object surface. The prediction module is used to acquire the depth image of the scene and register it with the RGB image to generate the scene point cloud; the target point cloud of the target object is cropped from the scene point cloud using the target object mask; a grasping region recognition network with the functional region mask as the spatial prior is constructed to process the target point cloud, predict and output the probability that each point in the target point cloud belongs to a high grasping confidence region. The output module is used to generate candidate grasping poses based on the obtained high grasping confidence region, fuse the high-level functional prior represented by the functional region mask and the low-level geometric features of the candidate grasping poses, perform collision detection and quality scoring on each candidate grasping pose, select the grasping pose with the best score as the output result, and drive the actuator to complete the grasping operation.

[0063] The module division in this embodiment of the invention is illustrative and represents only one logical functional division. In actual implementation, other division methods may be used. Furthermore, the functional modules in the various embodiments of the invention can be integrated into a single processor, exist as separate physical entities, or be integrated into a single module. The integrated modules described above can be implemented in hardware or as software functional modules.

[0064] In another embodiment of the present invention, a computer device is provided, comprising a processor and a memory. The memory stores a computer program, which includes program instructions. The processor executes the program instructions stored in the computer storage medium. The processor may be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. It is the computing and control core of the terminal, suitable for implementing one or more instructions, specifically suitable for loading and executing one or more instructions from the computer storage medium to achieve a corresponding method flow or corresponding function. The processor described in this embodiment of the present invention can be used for the operation of a task-oriented grasping method with cross-level constraint reasoning.

[0065] In another embodiment of the present invention, a storage medium is provided, specifically a computer-readable storage medium (Memory), which is a memory device in a computer device used to store programs and data. It is understood that the computer-readable storage medium here can include both the built-in storage medium in the computer device and extended storage media supported by the computer device. The computer-readable storage medium provides storage space that stores the terminal's operating system. Furthermore, the storage space also stores one or more instructions suitable for loading and execution by a processor. These instructions can be one or more computer programs (including program code). It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device. The processor can load and execute one or more instructions stored in the computer-readable storage medium to implement the corresponding steps of the task-oriented grasping method for cross-level constraint reasoning in the above embodiments.

[0066] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, systems, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0067] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0068] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0069] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0070] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention and not to limit it. Although the present invention has been described in detail with reference to the above embodiments, those skilled in the art should understand that modifications or equivalent substitutions can still be made to the specific implementation of the present invention. Any modifications or equivalent substitutions that do not depart from the spirit and scope of the present invention should be covered within the scope of protection of the claims of the present invention.

Claims

1. A task-oriented grasping method for cross-level constraint reasoning, characterized in that, include: Acquire RGB images of natural language instructions and scene containing task objectives. Preprocess the RGB images of natural language instructions and scene to obtain text features and object masks respectively. Calculate the similarity between text features and visual features of each object mask region, and output the target object mask corresponding to the task objective. The target object mask, RGB image and natural language instructions are jointly input into the visual language big model for joint reasoning, and the output is a functional area mask that indicates the suitable gripping position on the object surface. Acquire a depth image of the scene and register it with an RGB image to generate a scene point cloud; The target point cloud of the target object is cropped from the scene point cloud using the target object mask; a grasping region recognition network with functional region mask as spatial prior is constructed to process the target point cloud, predict and output the probability that each point in the target point cloud belongs to a high grasping confidence region. Based on the obtained high-confidence grasping region, candidate grasping poses are generated. The high-level functional priors represented by the functional region mask and the low-level geometric features of the candidate grasping poses are fused together. Collision detection and quality scoring are performed on each candidate grasping pose. The grasping pose with the best score is selected as the output result to drive the actuator to complete the grasping operation.

2. The task-oriented grasping method for cross-level constraint reasoning according to claim 1, characterized in that, The process of acquiring RGB images of natural language instructions and scenes containing the task target, preprocessing the RGB images of natural language instructions and scenes to obtain text features and object masks, calculating the similarity between the text features and the visual features of each object mask region, and outputting the target object mask corresponding to the task target includes: In the visual language encoding stage, SAM is first used to spatially encode objects in the input image, and CLIP is used to align text features with spatial codes pixel by pixel to generate a pixel-level target region mask.

3. The task-oriented grasping method for cross-level constraint reasoning according to claim 2, characterized in that, Specifically, SAM is used as the visual encoder backbone to spatially encode objects within the captured scene, resulting in a series of pixel-level mask regions. Then, CLIP is used as the text encoder to encode semantic instructions into feature vectors according to text categories. Simultaneously, visual features are extracted pixel-by-pixel for each mask region. Finally, the semantic category features and visual features are mapped to the same space, and a cosine similarity score is calculated using the following formula: in, Represents cosine similarity. For the first Text feature vectors for the task. For the first The pixel-by-pixel visual feature vector of an object mask region For temperature coefficient, This represents the total number of valid object mask regions obtained through SAM encoding in the scene. For the first The normalized matching score between the object mask region and the task text features.

4. The task-oriented grasping method for cross-level constraint reasoning according to claim 1, characterized in that, The process of jointly inputting the target object mask, RGB image, and natural language instructions into a large visual language model for joint inference, and outputting a functional region mask indicating a suitable grasping position on the object surface, includes: By leveraging the decision module of VLM as a task-oriented crawling methodology framework, and jointly reasoning about multimodal data crawling scenarios and semantic information using context-based target masks, high-level knowledge priors are generated to guide downstream crawling tasks. Specifically, the shape and structure of the target object are categorized into edge-guided or surface-guided grasping objects based on task semantic information to guide the underlying grasping perception. Secondly, starting from human grasping habits, a context-based target mask is used as the spatial geometric constraint of the Virtual Model (VLM) to perform cross-level joint reasoning on the task text and scene images, generating a functional region mask. Finally, the functional region mask and semantic pixel mask regions are unified within a shared embedding space, achieving a unified representation of geometric structure semantics and functional masks. This provides a more refined spatial prior for subsequent cross-level grasping reasoning, as expressed in the following formula: in, For local function masks, For functional area mask, For context-based target masks.

5. The task-oriented grasping method for cross-level constraint reasoning according to claim 1, characterized in that, The process involves acquiring a depth image of the scene, registering it with an RGB image to generate a scene point cloud, cropping a target point cloud of the target object from the scene point cloud using a target object mask, and constructing a grasping region recognition network with a functional region mask as a spatial prior to process the target point cloud, predicting and outputting the probability that each point in the target point cloud belongs to a high grasping confidence region, including: Grasp LAKAN, a grasping region recognition network with La-GeoKan as its core, introduces pixel-level mask regions as high-level spatial priors, constraining global indiscriminate inference to make effective predictions for high-confidence grasping regions of target mask point clouds. Specifically, the La-GeoKan point cloud encoding network is used to extract features from regions of semantic target with a high probability of successful capture. Firstly, it uses points... Delineating spherical regions of different sizes around the center, the formula is expressed as: in, Indicates the current level The surrounding set of points, The base region radius is s, where s is the scaling factor used to adjust the region size at different levels; secondly, at each point... neighborhood Within, the most critical information is extracted by concatenating the information from surrounding points. The formula is as follows: in, This indicates the position information of surrounding points relative to the center point. Initial information for surrounding points, This indicates a feature concatenation operation. This indicates retrieving key information within a specified range. To integrate the information dimensions of each point; based on the global-local feature fusion mechanism, firstly, for each local feature... Perform non-linear processing to extract detailed shape information of the object. The formula is expressed as: in, It is a third-order B-spline function. For adjustable parameters, The information transformation matrix is ​​designed, and adjustable weighting coefficients are constructed. This is used to highlight information in important areas, and then the local information of all points is integrated into global information. And perform linear processing to preserve overall features. The formula is expressed as: in, For linear basis functions, For adjustable parameter matrices; finally, add global information. and processed local information By merging, the final feature information is obtained. : 。 6. The task-oriented grasping method for cross-level constraint reasoning according to claim 1, characterized in that, Based on the obtained high-confidence grasping region, candidate grasping postures are generated. The high-level functional priors represented by the functional region mask and the low-level geometric features of the candidate grasping postures are fused. Collision detection and quality scoring are performed on each candidate grasping posture, and the grasping posture with the best score is selected as the output result to drive the actuator to complete the grasping operation, including: By combining high-level functional priors with low-level geometric priors, collision detection and quality scoring are performed on the candidate pose set to ensure the feasibility and stability of grasping in an open world, and output a stable 6-DOF grasping pose. Finally, a visual feedback mechanism is added to allow the robot to return to the initial image position and capture the RGB of the scene after the grasping task is performed as visual information to be fed back to the VLM to determine whether the target object exists in the original scene, thereby determining whether the grasping task is successful or not, and constructing a closed-loop grasping framework of "perception-decision-action-feedback".

7. The task-oriented grasping method for cross-level constraint reasoning according to claim 6, characterized in that, By combining high-level functional priors with low-level geometric priors, collision detection and quality scoring are performed on the candidate pose set to ensure the executability and stability of grasping in an open world, and to output stable 6-DOF grasping poses, specifically including: First, the OBB algorithm is used to filter out center and edge points from the high-confidence region for crawling. Then, combined with existing physical knowledge, suitable crawling points are automatically determined. Then, perform geometric analysis on the determined gripping points to calculate the suitable rotation angle for the parallel grippers, using a rotation matrix. This means that by combining the rotation angle and the gripping point, a set of multiple candidate gripping postures can be obtained. Secondly, collision detection was added to the candidate grasping postures: a cylindrical area was constructed with the grasping point as the center and the width of the gripper opening as the diameter. If there was anything other than the target object within this area, the grasping posture was excluded, ensuring that the remaining postures were safe. Subsequently, this was combined with a function mask. The grasping posture is evaluated based on the "distance from the grasping point to the object's center of gravity," selecting the most suitable posture from the perspective of human grasping habits; the "scoring rules" are as follows: in, The score represents the distance between the grab point and the center of mass of the object, and reflects how easy or difficult it is to grab the object from that position. The furthest point from the centroid within the high-scratching confidence region The distance; The flag for "whether the grasping posture is in the appropriate area": ​​if the grasping posture is in the preset "appropriate grasping area", it is marked as 1; otherwise, it is marked as 0. This is a weighting coefficient used to adjust the importance of "region suitability" in the scoring; finally, the posture with the highest score is selected as the final grasping posture, i.e. .

8. A task-oriented grasping system for cross-level constraint reasoning, characterized in that, include: The data acquisition module is used to acquire RGB images of natural language instructions and scenes containing task objectives, preprocess the RGB images of natural language instructions and scenes to obtain text features and object masks respectively, calculate the similarity between text features and visual features of each object mask region, and output the target object mask corresponding to the task objective. The inference module is used to input the target object mask, RGB image and natural language instructions into the visual language big model for joint inference, and output a functional area mask that indicates the suitable gripping position on the object surface. The prediction module is used to acquire the depth image of the scene and register it with the RGB image to generate the scene point cloud; The target point cloud of the target object is cropped from the scene point cloud using the target object mask; a grasping region recognition network with functional region mask as spatial prior is constructed to process the target point cloud, predict and output the probability that each point in the target point cloud belongs to a high grasping confidence region. The output module is used to generate candidate grasping poses based on the obtained high grasping confidence region, fuse the high-level functional prior represented by the functional region mask and the low-level geometric features of the candidate grasping poses, perform collision detection and quality scoring on each candidate grasping pose, select the grasping pose with the best score as the output result, and drive the actuator to complete the grasping operation.

9. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the task-oriented grasping method for cross-level constraint reasoning as described in any one of claims 1 to 7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements the steps of the task-oriented grasping method for cross-level constraint reasoning as described in any one of claims 1 to 7.

Images