Part grasping positioning method based on RGB-D and visual model

Abstract

The application discloses a part grasping positioning method based on RGB-D and a visual model, and belongs to the technical field of robot visual positioning and man-machine cooperation assembly. The method acquires a color image, a depth image, camera internal parameters and an external parameter matrix of a camera to a robot base of an RGB-D camera, and performs effectiveness detection on the color image; generates a visual enhanced description according to a target part description, and calls a visual large model to obtain a two-dimensional boundary box or a center point; obtains target depth in combination with an aligned depth map, back projects to obtain camera coordinates and converts the camera coordinates into robot base coordinates; and outputs a grasping target point after work area and motion safety verification. The application can reduce the dependence on labeled samples, and improve the adaptability and safety of small industrial part grasping positioning.

Description

Technical Field

[0001] This invention relates to the fields of robot vision positioning, human-machine collaborative assembly, and industrial parts grasping control technology, and particularly to a parts grasping and positioning method based on RGB-D and a visual model. Background Technology

[0002] In industrial assembly, human-robot collaboration, and robot-assisted operations, robots often need to identify, locate, and grasp small industrial parts such as bearings, gears, washers, and nuts according to operator instructions. Traditional robot vision grasping systems typically rely on pre-trained target detection models, training the detection network based on a large number of labeled images. This type of method is effective in scenarios with fixed target categories and sufficient image samples, but in scenarios involving multiple product types, small batches, educational assembly, or temporary tasks, it is necessary to frequently add new categories of parts, resulting in high costs for re-collecting and labeling data, and insufficient deployment flexibility.

[0003] Object detection methods based solely on RGB images typically only provide two-dimensional image coordinates, failing to directly obtain the three-dimensional spatial coordinates required for robot grasping. While RGB-D cameras can acquire depth information, depth images themselves lack target semantics, making it difficult to directly determine whether a specific depth region corresponds to the part to be grasped. Therefore, organically combining visual semantic recognition capabilities with RGB-D depth measurement capabilities is one of the core issues in achieving autonomous robot grasping and localization.

[0004] Large-scale visual models possess strong open-vocabulary image understanding capabilities, enabling them to locate target objects in images based on natural language descriptions. However, these models typically only output two-dimensional bounding boxes or two-dimensional center points, failing to directly output the accurate three-dimensional grasping coordinates required for robot control. Furthermore, small industrial parts often present challenges such as small size, metallic reflectivity, complex backgrounds, and similar appearances, easily leading to visual localization failures. In addition, real-world systems may encounter issues like black frames in RGB-D camera colors, invalid depth information, and incorrect extrinsic parameters from the camera to the robot base. Directly sending erroneous coordinates to the robot could result in motion failures or even safety risks.

[0005] In existing technologies, some solutions attempt to combine large visual models with 3D perception for robot manipulation, but most are based on 2D RGB images and lack depth filtering for reflective metal surfaces, resulting in insufficient reliability when directly applied to industrial parts grasping. Some deep learning-based visual localization solutions rely on specific datasets for training and deformation prediction, making them unsuitable for the zero-shot rapid deployment requirements of multi-variety, small-batch scenarios. Related solutions in academia either rely on precise CAD models and point cloud registration, leading to high deployment costs; or they are designed for general scenarios, failing to enhance visual features for small industrial parts and lacking pre-emptive safety checks on robot joint boundaries and work areas. These solutions fail to systematically address engineering problems in industrial assembly scenarios, such as black frames in images, depth holes, and abnormal extrinsic parameters causing abnormal robot movement.

[0006] Therefore, there is an urgent need for an industrial part grasping and positioning solution that can integrate the 2D localization capabilities of large visual models with RGB-D depth measurement capabilities, and simultaneously possess image validity detection, enhanced visual description of industrial parts, depth validity screening, coordinate transformation, and robot motion safety verification capabilities. Summary of the Invention

[0007] The purpose of this invention is to overcome the above-mentioned shortcomings in the prior art and provide a part grasping and positioning method based on RGB-D and visual models. This method reduces the dependence on labeled samples and can effectively improve the adaptability, reliability and motion safety of grasping and positioning small industrial parts, and is suitable for assembly operation scenarios with multiple varieties and small batches.

[0008] To achieve the above-mentioned technical features, the objective of this invention is as follows: The first aspect of this invention provides a part grasping and positioning method based on RGB-D and a visual model, comprising the following steps: S1, acquire the color image and depth image captured by the RGB-D camera, and acquire the camera intrinsic parameters of the RGB-D camera, as well as the extrinsic parameter matrix from the RGB-D camera coordinate system to the robot base coordinate system; S2, perform image validity detection on the color image; if the detection fails, reacquire the color image. S3, Obtain the target description of the part to be grasped, and generate a visually enhanced description based on the target description; S4, input the color image and the visual enhancement description into the large visual model to obtain the two-dimensional positioning result of the part to be grasped in the color image; S5, based on the two-dimensional positioning result, obtain the target region depth value in the depth image aligned with the color image, and perform validity filtering on the depth value to obtain the target depth value; S6. Based on the camera intrinsic parameters, two-dimensional positioning results and target depth value, back-project to obtain the target camera coordinates in the RGB-D camera coordinate system; S7. Convert the target camera coordinates into target base coordinates in the robot base coordinate system according to the external parameter matrix; S8, perform a grasping work area verification and motion safety verification on the target base coordinates. If the verification passes, output the robot grasping target point; if the verification fails, output positioning abnormal information and prohibit the issuance of robot grasping motion commands.

[0009] Preferably, the image validity detection in step S2 includes black frame detection, brightness detection, and non-zero pixel ratio detection; specifically: Calculate the pixel mean, pixel standard deviation, and proportion of non-zero pixels in a color image; When the average pixel value is less than the first threshold, the standard deviation of pixels is less than the second threshold, or the proportion of non-zero pixels is less than the third threshold, the color image is deemed invalid and a new color image is acquired.

[0010] Preferably, the visual enhancement description in step S3 includes the shape features, hole features, material features, and / or category synonyms of the part to be grasped; The step of generating a visually enhanced description based on the target description specifically includes: mapping the target description into an enhanced description containing visual attributes based on a preset industrial part visual feature dictionary.

[0011] Preferably, when the target description includes a bearing, a visual feature description is generated including at least one of the following: annular, having a central hole, metal part, inner ring, and outer ring. When the target description contains a gear, a visual feature description is generated that includes at least one of the following: circular, toothed outer ring, hole in the center, and metal part. When the target description includes a gasket, a visual feature description is generated that includes at least one of the following: a thin ring, a central hole, and a sheet-like metal part. When the target description includes a nut, a visual feature description is generated that includes at least one of the following: hexagonal, centrally located hole, and threaded hole.

[0012] Preferably, the two-dimensional positioning result in step S4 includes a two-dimensional bounding box and / or a two-dimensional center point, and also includes confidence level; After obtaining the two-dimensional localization results, the validity of the area, aspect ratio, whether the two-dimensional center point is located at the edge of the image, and the confidence level of the two-dimensional bounding box are verified. If the visual model does not return a valid two-dimensional localization result, multiple candidate visual descriptions are generated based on the target description and input into the visual model in sequence for retry until a valid result is obtained or the preset number of retries is reached.

[0013] Preferably, the validity screening of depth values ​​in step S5 includes: Using the two-dimensional center point in the two-dimensional positioning result as the center, select a neighborhood window of a preset size in the depth image; Remove zero-value depths, depths exceeding the preset depth range, and abrupt changes from the neighborhood window; The target depth value is obtained by taking the median, mean, or weighted mean of the retained effective depths.

[0014] Preferably, in step S6, the target camera coordinates are calculated according to the following formula: ; ; ; in, and The pixel coordinates of the two-dimensional center point. For the target depth value, , , , This refers to the camera intrinsics of an RGB-D camera. , , The coordinates are the target camera coordinates.

[0015] Preferably, in step S7, the coordinates of the target base are calculated according to the following formula: ; in, The target camera coordinates are in homogeneous form. This is the extrinsic parameter matrix from the RGB-D camera coordinate system to the robot base coordinate system. The coordinates of the target base are in homogeneous form.

[0016] Preferably, the work area verification in step S8 includes: Determine whether the target base coordinates are within the preset grasping working area, which is defined by the X-axis range, Y-axis range and Z-axis range under the robot base coordinate system; If the target base coordinates are located behind the robot base, close to the center line of the robot base, or outside the preset grasping work area, it is considered invalid; The motion safety verification in step S8 includes: Calculate the robot base plane orientation angle based on the target base coordinates; When the difference between the orientation angle of the robot base plane and the motion boundary of the robot's first joint is less than a preset threshold, it is determined that there is a joint boundary or singular risk, and the issuance of robot motion commands is prohibited.

[0017] Preferably, the method further includes a failed sample recording step: When the visual large model fails to find the target, the depth value is invalid, the extrinsic parameter matrix is ​​unavailable, the target base coordinates are outside the working area, or the motion safety check fails, save the current color image, target description, visual large model return result, 2D positioning result, depth value, target camera coordinates, target base coordinates, and failure type.

[0018] Preferably, the target point to be grasped by the robot includes the target base coordinates, the pre-grasping point, and the grasping execution point; The pre-grabbing point is obtained by adding a preset safety height along the vertical direction of the robot base coordinate system based on the target base coordinates. The grab execution point is obtained based on the offset between the target base coordinates and the center point of the end effector tool.

[0019] A second aspect of the present invention provides a part grasping and positioning system based on RGB-D and a visual model, comprising: The RGB-D image acquisition module is used to acquire color images, depth images, and camera intrinsic parameters. The image validity detection module is used to perform image validity detection on the color image, and triggers re-acquisition when the detection fails. The target description enhancement module is used to generate a visually enhanced description based on the target description of the part to be grasped. The visual large model localization module is used to input color images and visually enhanced descriptions into the visual large model to obtain the two-dimensional localization results of the parts to be grasped. The depth fusion module is used to obtain the depth value of the target region in the aligned depth image based on the two-dimensional positioning results, and to obtain the target depth value after validity filtering; The camera coordinate calculation module is used to back-project the target camera coordinates based on the camera intrinsic parameters, 2D positioning results and target depth values. The base coordinate transformation module is used to convert the target camera coordinates into the target base coordinates based on the extrinsic parameter matrix from the RGB-D camera coordinate system to the robot base coordinate system. The safety verification module is used to verify the target base coordinates, including the working area for grasping and motion safety. The target grabbing output module is used to output the target point that the robot grabs after the verification is passed, and to output positioning error information and prohibit the issuance of robot grabbing motion commands when the verification fails.

[0020] Preferably, the target description enhancement module has a built-in industrial part visual feature dictionary, which includes at least the shape features, hole features, material features, and category synonyms for bearings, gears, gaskets, nuts, bolts, and shafts.

[0021] Preferably, the security verification module includes: The work area judgment unit is used to determine whether the target base coordinates are located within the preset grasping work area; The joint boundary judgment unit is used to calculate the robot base plane orientation angle based on the target base coordinates and determine whether it is close to the robot's first joint motion boundary; The anomaly interception unit is used to prevent motion commands from being sent to the robot controller when the target base coordinates exceed the preset grasping work area or approach the motion boundary of the robot's first joint.

[0022] Preferably, it also includes a failure sample recording module, which is used to save the color image, target description, visual large model return result, two-dimensional positioning result, depth value, target camera coordinates, target base coordinates and failure type when positioning fails.

[0023] A third aspect of the present invention provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable by the processor, wherein the computer program, when executed by the processor, implements the aforementioned part gripping and positioning method based on RGB-D and a visual model.

[0024] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the part grasping and positioning method based on RGB-D and a visual model.

[0025] A fourth aspect of the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the part grasping and positioning method based on RGB-D and a visual model.

[0026] The present invention has the following beneficial effects: 1. By leveraging the open vocabulary capabilities of the large visual model, two-dimensional localization of parts can be achieved without retraining the detection model for new types of parts. This significantly reduces the dependence on labeled samples, making it suitable for industrial assembly scenarios with multiple varieties and small batches, and lowering the deployment threshold.

[0027] 2. By converting the two-dimensional semantic localization results into three-dimensional spatial coordinates through RGB-D depth images, the localization results can be directly used for robot grasping planning, thus opening up the link from visual semantics to robot execution.

[0028] 3. An enhanced visual description mechanism for industrial parts has been added, supplementing visual features such as shape, material, and holes for typical small parts such as bearings, gears, washers, and nuts, significantly improving the success rate of visual positioning of small metal parts.

[0029] 4. The pre-processing color image validity detection and deep neighborhood screening mechanism can avoid the problems of black frames and low brightness image input models, while suppressing depth noise caused by metal reflections and deep holes, thus improving the robustness of localization.

[0030] 5. A new grasping work area verification and motion safety verification step has been added. This step can intercept motion commands when there are errors in external parameters, the target goes out of bounds, or approaches a singular joint configuration, thereby preventing the robot from moving abnormally backward or to the boundary area and improving the safety of system operation.

[0031] 6. Configure the ability to record failure samples throughout the entire process, which can retain data such as images, model output, coordinates, external parameters and failure types, so as to quickly locate the cause of failure and improve the efficiency of system debugging and calibration.

[0032] 7. It does not rely on part CAD models or pre-built 3D model libraries, simplifying the on-site implementation process and making the project more feasible. Attached Figure Description

[0033] The present invention will be further described below with reference to the accompanying drawings and embodiments.

[0034] Figure 1 This is a schematic diagram of the part gripping and positioning system described in this invention.

[0035] Figure 2 This is an overall flowchart of the part gripping and positioning method described in this invention.

[0036] Figure 3 This is a flowchart illustrating the localization process of deep fusion of the visual large model and RGB-D in this invention.

[0037] Figure 4 This is a flowchart illustrating the generation process of the visual enhancement description of industrial parts according to the present invention.

[0038] Figure 5 This is a schematic diagram illustrating the conversion from camera coordinates to robot base coordinates according to the present invention.

[0039] Figure 6 This is a schematic diagram of the robot's grasping work area and motion safety verification according to the present invention.

[0040] Figure 7 This is a flowchart illustrating the recording process for locating failed samples in this invention. Detailed Implementation

[0041] The present invention will now be described in detail with reference to the accompanying drawings and specific embodiments. It should be understood that the specific embodiments described herein are for illustrative purposes only and are not intended to limit the scope of the invention.

[0042] The purpose of this invention is to address the shortcomings of existing technologies by providing a part grasping and positioning method based on RGB-D and a visual model, thus solving the following technical problems: (1) Traditional target detection models rely on a large number of labeled samples, which makes it difficult to adapt to the rapid deployment needs of various types of industrial small parts; (2) The visual large model can only provide two-dimensional positioning results and cannot directly output the three-dimensional grasping coordinates that the robot can execute; (3) RGB-D cameras are prone to black frames, invalid depth or poor image quality, which affects the reliability of positioning; (4) When the camera coordinates are converted to the robot base coordinates, errors in external parameters or the target exceeding the working area can easily lead to motion failure or safety risks; (5) The robot lacks a pre-verification mechanism for the target base coordinates and motion accessibility before grasping.

[0043] Terminology Definition Visual large model: A visual semantic localization model with cross-modal understanding capabilities of images and text, capable of outputting two-dimensional target localization results in images based on natural language descriptions.

[0044] RGB-D images: Color and depth images that are synchronously acquired and registered by an RGB-D camera. The color image is used for target semantic recognition, and the depth image is used to obtain the spatial depth information of the target.

[0045] Visual Enhancement Description: A target description generated based on the name of the part to be grasped, including shape, holes, material, category synonyms, and usage features, used to improve the positioning stability of large visual models for small industrial parts.

[0046] Aligning depth images: Depth data obtained after pixel-level registration of the depth image and the color image, so that the two-dimensional coordinates in the color image can be directly mapped to the depth values ​​in the depth image.

[0047] Target camera coordinates: Three-dimensional coordinates in the RGB-D camera coordinate system obtained by back-projection of the two-dimensional center point, target depth value and camera intrinsic parameters.

[0048] Target base coordinates: The three-dimensional coordinates in the robot base coordinate system obtained by transforming the target camera coordinates through the extrinsic parameter matrix.

[0049] Extrinsic parameter matrix: A 4×4 homogeneous transformation matrix that describes the rotation and translation relationship between the RGB-D camera coordinate system and the robot base coordinate system.

[0050] Grasping work area verification: The verification process to determine whether the coordinates of the target base are within the robot's preset grasping space range.

[0051] Motion safety verification: Before issuing motion commands, the process of verifying whether there are risks such as joint boundaries, singular configurations, targets behind the robot, or out-of-bounds work areas in the target base coordinates.

[0052] Example 1: Complete Example of Part Gripping and Positioning Method This embodiment provides a part grasping and positioning method based on RGB-D and a visual model, including the following steps: Step S1: Image and Parameter Acquisition The system acquires a single frame of color image captured by an RGB-D camera. With a depth image Simultaneously read the camera's factory-calibrated intrinsic parameter matrix. And the extrinsic parameter matrix obtained in advance through hand-eye calibration, from the RGB-D camera coordinate system to the robot base coordinate system. .

[0053] Among them, camera intrinsic parameters include focal length. , coordinates of the principal point , ;External parameter matrix It is a 4×4 homogeneous transformation matrix containing a 3×3 rotation matrix. With 3×1 translation vector .

[0054] Step S2: Color Image Validity Detection For the input color image Perform validity checks and calculate the average pixel value of the image. Pixel standard deviation Non-zero pixel ratio .

[0055] When satisfied , or Under any condition ( For brightness threshold, For texture change threshold, If the frame is determined to be a black frame, a low-brightness frame, or an invalid frame (using a non-zero pixel ratio threshold), the current frame is discarded and the image is re-acquired from the RGB-D camera. If multiple re-acquisitions are unsuccessful, a camera image anomaly alarm is output.

[0056] Step S3: Visual Enhancement Description Generation The system acquires the target description of the part to be grasped, such as "bearing" or "gear," input by the operator. Based on a pre-defined dictionary of visual features of industrial parts, the brief target description is expanded into a visually enhanced description that includes shape, holes, material, and category synonyms, thereby strengthening the visual model's recognition and perception of industrial parts.

[0057] Step S4: Two-dimensional localization of the large visual model Color image Along with the visual enhancement description, the data is input into the large visual model. The model outputs the two-dimensional localization result of the target, including a marker indicating whether the target was found and a two-dimensional bounding box. Two-dimensional center point And confidence.

[0058] If the model only outputs bounding boxes, then the center point is calculated using the bounding box coordinates: ; ; After obtaining the two-dimensional center point, the validity of the bounding box area, aspect ratio, whether the center point is located at the edge of the image, and confidence value are verified. If the verification fails, a retry mechanism is triggered, generating multiple sets of candidate descriptions and calling the model in sequence until a valid result is obtained or the maximum number of retries is reached.

[0059] Step S5: Depth Acquisition and Neighborhood Filtering Using the 2D center point [u,v] as an index, the depth value at the corresponding position is read from the depth image aligned with the color image. To suppress the effects of metallic reflections and depth holes, a neighborhood window of size N×N (e.g., 5×5, 7×7) is selected centered on the center point. Zero-value depths, depths exceeding the preset effective depth range, and abrupt depths with excessive deviations from the neighborhood median are removed from the window. The median of the remaining effective depths is then taken to obtain the final target depth value. .

[0060] Step S6: Camera coordinate back projection Based on the target depth value within the camera, the two-dimensional pixel coordinates are back-projected into three-dimensional coordinates in the RGB-D camera coordinate system. The calculation formula is as follows: ; ; ; in, , The pixel coordinates of the two-dimensional center point. For the target depth value, , The horizontal and vertical focal lengths of the camera. , The coordinates of the principal point in the image; , , The target's three-dimensional coordinates in the camera coordinate system.

[0061] Represent it in homogeneous coordinate form: ; Step S7: Robot base coordinate transformation Using a pre-calibrated extrinsic matrix The target camera coordinates are converted to coordinates in the robot's base coordinate system using the following formula: ; in, The target base coordinates are in homogeneous form. , , This refers to the three-dimensional spatial position of the target in the robot's base coordinate system.

[0062] Step S8: Security Verification and Capture Point Output First, perform a work area verification: determine the target base coordinates. Whether it falls within the preset grasping work area, the work area is determined by , , Limitations: If the target is located behind the robot base or outside the work area, the coordinates are considered invalid.

[0063] Then, motion safety verification is performed: the robot base plane orientation angle is calculated based on the target base coordinates. ; When the direction angle If the difference between the value of the first joint motion boundary and the value of the first joint motion boundary of the robot is less than the preset safety threshold, it is determined that there is a risk of joint boundary or singular configuration.

[0064] If both checks pass, the robot will generate a target point for grasping, including a pre-grasp point and a grasp execution point. Pre-grabbing points: Safety height superimposed vertically along the base coordinate system ,Right now ; Capture execution point: Overlay the offset of the end effector tool center point ,Right now .

[0065] If the verification fails, a positioning error message will be output, and motion commands will be prohibited from being sent to the robot controller.

[0066] Example 2: Complete Implementation of Parts Gripping and Positioning System This embodiment illustrates the complete architecture and module composition of the system of the present invention, such as... Figure 1As shown, this implementation provides an industrial parts gripping and positioning system based on the fusion of a large visual model and RGB-D images. The system includes an RGB-D image acquisition module, an image validity detection module 200, a target description enhancement module 300, a large visual model positioning module 400, a deep fusion module 500, a coordinate transformation module 600, a security verification module 700, a gripping target output module 800, and a robot controller 900.

[0067] The functions and connections of each module are as follows: 1. RGB-D Image Acquisition Module The corresponding RGB-D camera hardware includes an RGB-D camera 100 used to acquire color and depth images of industrial parts on an assembly workbench, while simultaneously outputting camera intrinsic parameter data. Structured light cameras, binocular depth cameras, or time-of-flight depth cameras can be used, supporting synchronous acquisition and hardware-level alignment of color and depth frames. The image data output by this module is transmitted to an image validity detection module, and the depth data is synchronously transmitted to a depth fusion module.

[0068] 2. Image validity detection module It receives color image data and has three built-in detection units: black frame detection, brightness detection, and non-zero pixel ratio detection. These units calculate the pixel mean, standard deviation, and non-zero pixel ratio, respectively, and compare these values ​​with preset thresholds to output a validity determination result. If the image is invalid, a resampling command is sent to the RGB-D image acquisition module; if the image is valid, the color image is forwarded to the visual large-scale model localization module.

[0069] 3. Target Description Enhancement Module The module receives a target part description input by the operator and includes a built-in visual feature dictionary for industrial parts and a candidate description generation unit. The visual feature dictionary stores visual attributes such as shape, holes, material, and synonyms for various typical industrial parts. The candidate description generation unit generates multiple sets of differentiated descriptive text when the initial localization fails. The visually enhanced description output by this module is then transmitted to the visual large-scale model localization module.

[0070] 4. Visual Large Model Localization Module The module receives a color image and visual enhancement description, calls a large visual model for inference, and outputs the 2D bounding box, 2D center point, and confidence score of the target part. The module has a built-in result normalization unit that can convert the normalized coordinates output by the model into pixel coordinates, or calculate the center point based solely on the bounding box. It also has a built-in validity verification unit that verifies the bounding box size, position, and confidence score. Verified 2D positioning results are transmitted to the deep fusion module.

[0071] 5. Deep Fusion Module It receives the 2D positioning results and the aligned depth image, and has built-in neighborhood sampling unit, invalid depth removal unit, and depth calculation unit. The neighborhood sampling unit extracts a depth window of a specified size centered on the 2D center point; the invalid removal unit filters out zero values, out-of-range and abrupt depth changes; the depth calculation unit takes the median or mean of the valid depths to obtain the target depth value and transmits it to the coordinate transformation module.

[0072] 6. Coordinate Transformation Module It includes a camera coordinate calculation unit and a base coordinate transformation unit. The camera coordinate calculation unit back-projects the target camera coordinates based on the camera's intrinsic parameters, the 2D center point, and the target depth value. The base coordinate transformation unit calls the extrinsic parameter matrix to convert the camera coordinates into robot base coordinates. The transformed target base coordinates are then transmitted to the safety verification module.

[0073] 7. Security Verification Module It incorporates a working area judgment unit, a joint boundary judgment unit, and an anomaly interception unit. The working area judgment unit verifies whether the target base coordinates are within the preset working space; the joint boundary judgment unit calculates the base plane orientation angle to determine whether it is close to the joint motion boundary; and the anomaly interception unit sends an interception signal to the target capture output module when the verification fails.

[0074] 8. Target capture output module Upon receiving the verified target base coordinates, the module generates a complete grasping target point including pre-grasp points and grasping execution points, and sends it to the robot controller. If an interception signal is received, the module outputs positioning anomaly information and does not issue motion commands. The module has a built-in failure sample recording unit, which can retain full-process data in case of failure at any stage.

[0075] 9. Robot Controller The robot controller 900 is communicatively connected to the robotic arm 901 and the end effector 902, and is used to control the movement of the robotic arm and the closing of the end effector based on the grasping target point. It receives the grasping target point and controls the robotic arm and end effector to complete the grasping action, serving as the system's execution terminal.

[0076] Example 3: Detailed Example of Image Validity Detection This embodiment further refines the execution logic of image validity detection. In this embodiment, after the system acquires the color image from the RGB-D camera 100, it first performs image validity detection. Let the input color image resolution be W×H, and the pixel value range be 0~255.

[0077] The system iterates through all pixels and calculates the global pixel mean. : ; Calculate pixel standard deviation : ; Count the number of pixels with a value greater than 0 and calculate the proportion of non-zero pixels. .

[0078] Set threshold , , ,when When it is judged as a black frame, when When the image is judged to have no texture (e.g., the lens is obstructed), If the frame is deemed to have insufficient valid pixels, the current frame will be discarded and image acquisition will be retried if any of these conditions are met. If five consecutive frames are deemed invalid, an RGB-D camera color channel fault will be reported to the host computer.

[0079] This method avoids sending a completely black image to a large visual model, thereby reducing invalid inference results when the target is not found.

[0080] Example 4: Example of Visual Enhancement Description Generation for Industrial Parts This embodiment provides rules for generating visually enhanced descriptions and candidate descriptions for typical industrial parts. In this embodiment, the target description enhancement module 300 receives the target description input by the operator. For example, the operator inputs "bearing," "gear," "shim," or "nut." Since the visual large model's understanding of simple mechanical part names may be unstable, the system converts the target description into a visually enhanced description based on the industrial part visual feature dictionary 301.

[0081] The system's built-in visual feature dictionary for industrial parts includes mapping rules for four typical parts: bearings, gears, washers, and nuts. Bearing → A ring-shaped metal part with a central hole, consisting of an inner ring and an outer ring. Gear → a round metal part with teeth on the outer ring and a hole in the center. Gasket → Thin circular metal sheet with a hole in the center, washer Nut → a hexagonal metal part with a threaded hole in the center. When the initial localization fails, the candidate description generation unit 302 generates multiple sets of candidate descriptions according to the logic of feature progression and interference elimination. Taking a bearing as an example, the generated candidate descriptions are as follows: 1. Bearing: A ring-shaped metal part with a hole in the center; 2. The part in the image that most resembles a bearing: a silver or black ring with a hole in the center; it is neither a gear nor a nut. 3. The circular metal piece on the tabletop has an outer circle and an inner hole, resembling a bear. 4. Bearings in the reducer assembly are usually small, ring-shaped metal parts.

[0082] The system inputs candidate descriptions into the large visual model in sequence until a valid localization result is obtained or all candidate descriptions have been traversed.

[0083] The above methods can improve the success rate of visual positioning of small industrial parts in complex assembly scenarios.

[0084] Example 5: Two-Dimensional Localization and Retry Implementation of a Large Visual Model This embodiment illustrates the verification and retry logic for two-dimensional positioning results.

[0085] After the visual large model returns the results, the system first checks the "found" flag. If the target is not found, a retry is triggered directly; if the target is found, further verification is performed. Bounding box area: Areas that occupy less than 1% or more than 80% of the image are considered invalid. Aspect ratio: Aspect ratios exceeding 1:5 to 5:1 are considered invalid; Edge distance: A center point less than 5 pixels from the image edge is considered invalid; Confidence level: Confidence level below 0.5 is considered invalid.

[0086] If any item fails, the location result is deemed invalid, and a candidate description retry mechanism is initiated, with a maximum of 4 retries. If all retries fail, the failed sample is recorded, and the location process is exited.

[0087] Example 6: Deep Neighborhood Filtering and Target Depth Calculation Example This embodiment details the execution process of deep validity screening.

[0088] After obtaining the two-dimensional center point [u,v], the depth fusion module 500 searches for the corresponding depth in the depth image aligned with the color image.

[0089] Because industrial parts may have metallic reflections, deep voids, or partial occlusions, the system does not directly use the depth of a single pixel. Instead, it selects a neighborhood window near the center point. The system defaults to a 7×7 neighborhood window, centered on the two-dimensional center point to extract the depth region. The filtering rules are as follows: 1. Remove pixels with a depth value of 0, corresponding to the depth hole region; 2. Remove pixels with depth values ​​less than 300mm or greater than 2000mm, corresponding to data outside the effective ranging range of the camera; 3. Calculate the median of all non-zero depths within the window, and remove abrupt changes in depth that deviate from the median by more than 20%, corresponding to abnormal jumps caused by metallic reflection.

[0090] The median of the remaining valid depth pixels is taken as the final target depth value; if the percentage of valid depth pixels is less than 30% of the total pixels in the window, the depth is determined to be invalid, the failed sample is recorded and the process is exited.

[0091] Example 7: Example of Grasping Work Area and Motion Safety Verification This embodiment provides specific parameter examples for security verification.

[0092] Taking a six-axis industrial robot as an example, the preset gripping work area is: X-axis range: 200mm~800mm (the positive X-axis direction is directly in front of the robot); Y-axis range: 400mm~400mm; Z-axis range: 100mm~300mm.

[0093] During workspace verification, if If the target is located behind or inside the robot base, the determination is invalid; if the coordinates exceed the range of the three axes, it is determined as an out-of-bounds work area.

[0094] During motion safety verification, the robot's first joint has a range of motion of 170° to +170°, with a preset safety threshold of 10°. Calculate the base plane orientation angle. Afterwards, if If the target is close to the joint movement boundary, it is determined that there is a singular risk, and movement commands are prohibited.

[0095] This step can prevent the robot from moving backward or to the joint boundary area due to errors in the extrinsic parameter matrix, misposition of the target, or reversal of the coordinate axis direction.

[0096] Example 8: Example of Recording Failed Location Samples This embodiment illustrates the recording content and purpose of failed samples.

[0097] When any step in the location process fails, the failure sample recording module automatically saves the following data: 1. Current frame color image and depth image; 2. Original target description and generated visually enhanced description, candidate description; 3. The complete results returned by the large visual model include bounding boxes, center points, confidence scores, and failure explanations; 4. The original depth data, effective depth ratio, and filtered target depth value of the depth neighborhood window; 5. Calculated target camera coordinates and target base coordinates; 6. The source and version of the extrinsic parameter matrix currently in use; 7. Failure type and failure reason labels.

[0098] The recorded samples can be used for offline analysis to locate the cause of failure, optimize the visual feature dictionary, adjust the depth screening parameters, or recalibrate the hand and eye, and continuously iterate the system performance.

[0099] Example 9: Example of an electronic device and storage medium This embodiment provides an electronic device, including a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements all the steps of the part gripping and positioning method described in Embodiment 1.

[0100] This electronic device can be an industrial control computer, an embedded industrial control board, or a robot controller. It can communicate with RGB-D cameras and robot controllers via Ethernet to complete visual positioning calculations and command issuance.

[0101] This embodiment also provides a computer-readable storage medium storing a computer program. When the program is executed by a processor, it implements the aforementioned part grasping and positioning method. The storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory, random access memory, magnetic disks, or optical disks.

[0102] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A part grasping and positioning method based on RGB-D and a visual model, characterized in that, Includes the following steps: S1, acquire the color image and depth image captured by the RGB-D camera, and acquire the camera intrinsic parameters of the RGB-D camera, as well as the extrinsic parameter matrix from the RGB-D camera coordinate system to the robot base coordinate system; S2, perform image validity detection on the color image; if the detection fails, reacquire the color image. S3, Obtain the target description of the part to be grasped, and generate a visually enhanced description based on the target description; S4, input the color image and the visual enhancement description into the large visual model to obtain the two-dimensional positioning result of the part to be grasped in the color image; S5, based on the two-dimensional positioning result, obtain the target region depth value in the depth image aligned with the color image, and perform validity filtering on the depth value to obtain the target depth value; S6. Based on the camera intrinsic parameters, two-dimensional positioning results and target depth value, back-project to obtain the target camera coordinates in the RGB-D camera coordinate system; S7. Convert the target camera coordinates into target base coordinates in the robot base coordinate system according to the external parameter matrix; S8, perform a grasping work area verification and motion safety verification on the target base coordinates. If the verification passes, output the robot grasping target point; if the verification fails, output positioning abnormal information and prohibit the issuance of robot grasping motion commands.

2. The part grasping and positioning method based on RGB-D and visual models according to claim 1, characterized in that, The image validity detection in step S2 includes black frame detection, brightness detection, and non-zero pixel ratio detection; specifically: Calculate the pixel mean, pixel standard deviation, and proportion of non-zero pixels in a color image; When the average pixel value is less than the first threshold, the standard deviation of pixels is less than the second threshold, or the proportion of non-zero pixels is less than the third threshold, the color image is deemed invalid, and a new color image is acquired.

3. The part grasping and positioning method based on RGB-D and visual models according to claim 1, characterized in that, The visual enhancement description in step S3 includes the shape features, hole features, material features, and / or category synonyms of the part to be grasped; The step of generating a visually enhanced description based on the target description specifically includes: mapping the target description into an enhanced description containing visual attributes based on a preset industrial part visual feature dictionary.

4. The part grasping and positioning method based on RGB-D and visual model according to claim 3, characterized in that: When the target description includes a bearing, a visual feature description is generated that includes at least one of the following: annular, with a central hole, metal parts, inner ring, and outer ring. When the target description contains a gear, a visual feature description is generated that includes at least one of the following: circular, toothed outer ring, hole in the center, and metal part. When the target description includes a gasket, a visual feature description is generated that includes at least one of the following: a thin ring, a central hole, and a sheet-like metal part. When the target description includes a nut, a visual feature description is generated that includes at least one of the following: hexagonal, centrally located hole, and threaded hole.

5. The part grasping and positioning method based on RGB-D and visual models according to claim 1, characterized in that, The two-dimensional positioning result in step S4 includes a two-dimensional bounding box and / or a two-dimensional center point, and also includes confidence level; After obtaining the two-dimensional localization results, the validity of the area, aspect ratio, whether the two-dimensional center point is located at the edge of the image, and the confidence level of the two-dimensional bounding box are verified. If the visual model does not return a valid two-dimensional localization result, multiple candidate visual descriptions are generated based on the target description and input into the visual model in sequence for retry until a valid result is obtained or the preset number of retries is reached.

6. The part grasping and positioning method based on RGB-D and visual models according to claim 1, characterized in that, Step S5 involves validating the depth values, including: Using the two-dimensional center point in the two-dimensional positioning result as the center, select a neighborhood window of a preset size in the depth image; Remove zero-value depths, depths exceeding the preset depth range, and abrupt changes from the neighborhood window; The target depth value is obtained by taking the median, mean, or weighted mean of the retained effective depths.

7. The part grasping and positioning method based on RGB-D and visual models according to claim 1, characterized in that, In step S6, the target camera coordinates are calculated according to the following formula: ； ； ； in, and The pixel coordinates of the two-dimensional center point. For the target depth value, , , , This refers to the camera intrinsics of an RGB-D camera. , , The target camera coordinates; In step S7, the coordinates of the target base are calculated according to the following formula: ； in, The target camera coordinates are in homogeneous form. This is the extrinsic parameter matrix from the RGB-D camera coordinate system to the robot base coordinate system. The coordinates of the target base are in homogeneous form.

8. The part grasping and positioning method based on RGB-D and visual model according to claim 1, characterized in that, The work area verification in step S8 includes: Determine whether the target base coordinates are within the preset grasping working area, which is defined by the X-axis range, Y-axis range and Z-axis range under the robot base coordinate system; If the target base coordinates are located behind the robot base, close to the center line of the robot base, or outside the preset grasping work area, it is considered invalid; The motion safety verification in step S8 includes: Calculate the robot base plane orientation angle based on the target base coordinates; When the difference between the orientation angle of the robot base plane and the motion boundary of the robot's first joint is less than a preset threshold, it is determined that there is a joint boundary or singular risk, and the issuance of robot motion commands is prohibited.

9. The part grasping and positioning method based on RGB-D and visual models according to claim 1, characterized in that, It also includes the step of recording failed samples: When the visual large model fails to find the target, the depth value is invalid, the extrinsic parameter matrix is ​​unavailable, the target base coordinates are outside the working area, or the motion safety check fails, save the current color image, target description, visual large model return result, 2D positioning result, depth value, target camera coordinates, target base coordinates, and failure type.

10. The part grasping and positioning method based on RGB-D and visual models according to claim 1, characterized in that, The target point that the robot grasps includes the target base coordinates, the pre-grab point, and the grasp execution point; The pre-grabbing point is obtained by adding a preset safety height along the vertical direction of the robot base coordinate system based on the target base coordinates. The grab execution point is obtained based on the offset between the target base coordinates and the center point of the end effector tool.

Images