Robot vision object semantic understanding and posture generation method based on large model

Abstract

The invention provides a robot vision object semantic understanding and posture generation method based on a large model, and belongs to the technical field of image processing. Comprising the following steps: S1, inputting an image and a 3D model; s2, object detection; s3, multi-modal feature alignment is carried out; s4, diffusion model sampling; s5, performing geometric screening; s6, carrying out NeRF (New Random Field) morphological modeling; s7, optimizing the optical flow; s8, joint loss calculation; s9, confidence coefficient analysis; and S10, outputting the attitude junction. According to the invention, through an innovative single-view rendering-optical flow optimization closed loop strategy, the calculation overhead is significantly reduced and the estimation precision is improved. According to the method, the algorithm performance in a complex scene is remarkably improved through multi-scale feature fusion and a confidence decomposition strategy.

Description

Technical Field

[0001] This invention provides a method for robot visual object semantic understanding and pose generation based on a large model, belonging to the field of image processing technology. Background Technology

[0002] In applications such as industrial automation, robotics, and augmented reality, 6D pose estimation technology (including 3D position and 3D rotation) plays a crucial role. This technology enables robots to accurately identify and grasp objects, achieving precise operations on automated production lines, while providing realistic visual effects in augmented reality applications. Traditional 6D pose estimation methods primarily rely on deep learning models, learning feature representations of specific objects through extensive training data. These methods perform well when handling known objects and have been widely applied in various fields such as industrial production, warehousing and logistics, and smart manufacturing. However, with the continuous expansion of application scenarios and the increasing demands of technology, traditional methods have encountered new challenges in practical applications, urgently requiring more flexible and intelligent solutions.

[0003] Existing 6D pose estimation methods suffer from severe limitations when dealing with novel objects. First, these methods require retraining the model for each new object, which is not only time-consuming and labor-intensive in practical applications but also significantly increases deployment costs. Second, existing methods often struggle to effectively utilize the object's 3D geometric information, resulting in less than ideal estimation accuracy. This is particularly problematic in industrial settings where the variety of new objects is constantly increasing, and current technologies cannot adapt quickly enough, severely hindering improvements in production efficiency. Furthermore, existing methods exhibit poor generalization ability when handling objects with complex shapes or surface features, making them prone to estimation bias.

[0004] For example, existing technologies include pose refinement methods based on multi-view rendering (Labbé, Y., Manuelli, L., Mousavian, A., Tyree, S., Birchfield, S., Tremblay, J., Carpentier, J., Aubry, M., Fox, D., & Sivic, J. (2023). MegaPose: 6D Pose Estimation of Novel Objects via Render & Compare. In Conference on Robot Learning (CoRL).). The core idea of ​​this method is to achieve 6D pose estimation of new objects through a rendering and comparison strategy. Specifically, this method first uses a pre-trained classification model to make a coarse estimate of the initial pose, and then improves the estimation accuracy through an iterative refinement process. During the refinement process, the system generates multiple rendered images from different perspectives based on the currently estimated pose. These images contain rich geometric information such as the object's surface normals. The drawback of this technical solution is that it requires rendering images from multiple perspectives during each pose refinement iteration, which significantly increases computational overhead. Furthermore, due to the need to store and process rendered images from multiple perspectives, this method has high memory requirements and is prone to memory bottlenecks when processing a large number of objects. Moreover, when processing objects with complex geometry, the computational cost will further increase because more perspectives are needed to capture complete shape information.

[0005] Another example is the existing technology, a single-stage pose estimation method based on 2D-2D feature matching (Shugurov, I., Li, F., Busam, B., & Ilic, S. (2022). OSOP: A Multi-Stage One Shot Object Pose EstimationFramework. *arXiv Preprint*, arXiv:2203.15533 [cs.CV]. Retrieved from [https: / / arxiv.org / abs / 2203.15533](https: / / arxiv.org / abs / 2203.15533)). This method employs a multi-stage end-to-end framework, inferring the 6D pose of an object by establishing correspondences between 2D features. The entire process is performed entirely in the 2D image space, avoiding direct processing of complex 3D geometric information. Because the method relies solely on 2D-2D feature matching for training, it ignores the 3D geometric constraints of the object, resulting in less than ideal accuracy in the final pose estimation. When dealing with objects with limited surface texture features, the estimation performance drops significantly due to the difficulty in extracting effective 2D features. Furthermore, the method is highly sensitive to image quality and lighting conditions, and is easily affected by interference in real-world industrial applications, impacting the stability of the estimation results.

[0006] Therefore, developing a new method that can simultaneously achieve both accuracy and generalization, enabling rapid and accurate estimation of the 6D pose of new objects, has become an important research direction in this field. This will not only improve the efficiency of industrial automation but also promote the development of emerging technologies such as augmented reality. Summary of the Invention

[0007] This invention provides a method for robot visual object semantic understanding and pose generation based on a large model, solving the following technical problems:

[0008] (1) The primary technical problem addressed by this invention is how to effectively utilize the 3D shape information of an object while reducing computational costs. Existing technologies either require rendering a large number of multi-view images to obtain shape information, resulting in huge computational overhead; or they rely entirely on 2D feature matching while ignoring 3D geometric constraints, leading to insufficient accuracy. This trade-off between computational efficiency and estimation accuracy severely restricts the practical application of existing methods in industrial scenarios. Therefore, there is an urgent need for a new technical solution that can fully utilize 3D shape information to improve the accuracy of pose estimation while maintaining low computational costs.

[0009] (2) Another key issue that this invention also needs to address is how to improve the robustness and generalization ability of the algorithm in complex scenes. Existing technologies perform poorly when dealing with interference factors such as occlusion and changes in lighting, especially for objects with limited surface texture, where the estimation performance drops significantly due to the difficulty in extracting reliable features. Furthermore, many methods require retraining the model for new objects, which is both time-consuming and impractical in real-world applications. Therefore, a technical solution is needed that can adaptively handle various complex scenes and has good generalization ability for new objects.

[0010] The specific technical solution provided by this invention is as follows:

[0011] A method for robot visual object semantic understanding and pose generation based on a large model includes the following steps:

[0012] S1. Input image and 3D model;

[0013] Input the 3D model file of the target object and the RGB image of the target object whose pose needs to be estimated;

[0014] For input image calibration, the camera intrinsic parameter matrix K is in the form of:

[0015]

[0016] Where f x f y These are focal length parameters, (cx, c) y () are the coordinates of the principal point;

[0017] S2. Object detection;

[0018] S2.1. Object detection is performed using a CLIP-DETR detector:

[0019] Use pre-trained large models (such as ViT-Giant) to extract deep features from images;

[0020] The detection head network consists of two branches:

[0021] Bounding box regression branch: Outputs the 2D bounding box coordinates (x, y, w, h) of the object.

[0022] Category classification branch: the probability distribution for predicting object categories.

[0023] The detection results can be represented as a list: D = {di}, where each detection di contains:

[0024] Bounding box coordinates: b i =(x i ,y i ,w i ,h i );

[0025] Category tag: c i Detection confidence score: s i ;

[0026] S2.2. Image cropping transformation:

[0027] I crop =Crop(I, b) i (1)

[0028] The Crop function is based on the bounding box coordinates b. i Extract the target region from the original image I.

[0029] S2.3. Adjust the cropped image to a fixed resolution (H, W):

[0030] I resize =Resize(I crop (H, W)) (2) S2.4. Adjust camera parameters:

[0031] The estimation results of scene depth information from a large model are introduced to assist in adjusting camera intrinsic parameters. Using the depth map generated by the large model, the focal length parameters fx and fy, as well as the principal point coordinates (cx, cy) in the camera intrinsic parameter matrix K are optimized.

[0032] S2.5. Standardize the adjusted image:

[0033] I norm =(I resize -μ) / σ (3)

[0034] Where μ is the mean of the image intensity values ​​and σ is the standard deviation of the image intensity values.

[0035] S2.6. Attitude representation and predefined rotations;

[0036] The system uses Euler angles to represent 3D rotations, with each rotation defined by three angles. Where ψ is the yaw angle and θ is the pitch angle. It is the roll angle;

[0037] N uniformly distributed sets of rotated samples are predefined: {R} i}(i=1to N);

[0038] S3. Multimodal feature alignment;

[0039] The input image extracts features through a visual encoder of a large model (such as ViT-Giant);

[0040] Generate multiple sets of semantic cues (such as "object is placed upright" or "handle is facing right"), and generate embeddings using a large model (such as a multimodal large model).

[0041] Joint probability modeling:

[0042]

[0043] Diffusion t This is the reverse process of the DDIM diffusion model.

[0044] S4. Diffusion model sampling;

[0045] The UNet structure is adopted, with the number of channels ranging from 64 to 128 to 256 to 512, and the embedding dimension of the time step being 256.

[0046] A linear noise scheduling strategy is adopted, with a total time step T = 1000;

[0047] Generate N = 256 pose candidates {P k =[R k , t k ]} N k＝1 , where R k For a 3×3 rotation matrix, t k It is a 3×1 translation vector; S5. Geometric screening;

[0048] Projection function:

[0049]

[0050] Reprojection error calculation:

[0051]

[0052] Gradient descent optimization: minimizing ∈ using the Adam optimizer k The Top-64 candidate poses with the smallest error are retained;

[0053] S6.NeRF morphological modeling;

[0054] S6.1 Implicit Field Modeling:

[0055] Network architecture: MLP network, input is 3D coordinate x and view direction d, output is voxel density σ and color c:

[0056] σ,c=MLP(X,d) (7)

[0057] Regularization term:

[0058]

[0059] Training objectives:

[0060]

[0061] S6.2 Real-time Rendering:

[0062] Generate N r A ray of light r j Sample N s Points {X j ,k};

[0063] Volumetric rendering:

[0064]

[0065] in Output rendering depth map D render and semantic mask M render ;

[0066] S6.3 Shape Consistency Constraint:

[0067] Deep consistency loss:

[0068]

[0069] Surface normal consistency:

[0070]

[0071] S7. Optical flow optimization;

[0072] Feature extraction: Input image I norm and rendering image I render Features are extracted using ViT, and sine / cosine position encodings are added to the features:

[0073]

[0074] Correlation volume construction:

[0075]

[0076] Where i and j are spatial indices, and k is the depth index;

[0077] Optical flow refinement: Input the optical flow Φ from the previous moment t-1 And related correlation features C ijk Output the current optical flow Φ t :

[0078] Φ t+1 =GUR(Φ t C ijk (15)

[0079] Set the maximum number of iterations T = 10, and the learning rate η = 0.01 per step;

[0080] S8. Joint loss calculation;

[0081] Joint optimization objective function:

[0082]

[0083] Projection loss:

[0084]

[0085] Semantic consistency loss:

[0086]

[0087] S9. Confidence analysis;

[0088] Multi-scale confidence assessment:

[0089] Feature pyramid construction: Extract features from layers P3-P7 of ViT and calculate projection errors at each scale.

[0090]

[0091] Uncertainty estimation:

[0092]

[0093] Where σ is the Softplus activation function.

[0094] S10. Output attitude

[0095] The output includes:

[0096] Final stance: Obtained by minimization get.

[0097] Confidence level: in For rendering mask.

[0098] The technical effects of the technical solution provided by this invention are as follows:

[0099] (1) This invention significantly reduces computational overhead and improves estimation accuracy through an innovative single-view rendering-optical flow optimization closed-loop strategy. Compared with existing technologies, this scheme only needs to generate a single-view rendering image in each iteration, greatly reducing computational overhead. By combining a differentiable PnP solver with optical flow prediction, the system can fully utilize 3D geometric constraint information, ensuring both efficient processing and accuracy of estimation results. This efficient shape-aware mechanism not only solves the problem of excessive computational resource consumption in traditional methods but also maintains high accuracy in attitude estimation, providing a more practical technical solution for industrial automation applications.

[0100] (2) This invention significantly improves algorithm performance in complex scenarios through multi-scale feature fusion and confidence decomposition strategies. A cascaded architecture of ViT-Pose (ViT-B / 16) and GRU optical flow network is adopted, and coarse-to-fine pose optimization is achieved through a multi-scale feature pyramid (P3-P7 layers). The confidence score is decomposed into deterministic weights (based on feature response intensity) and pose-sensitive weights (based on optical flow consistency), effectively suppressing noise interference in occluded areas. This design not only significantly improves the reliability of the estimation results but also enables this invention to maintain stable high performance in different industrial scenarios, demonstrating excellent environmental adaptability. Attached Figure Description

[0101] Figure 1 This is a flowchart of the present invention. Detailed Implementation

[0102] The specific technical solutions of the present invention will be described in conjunction with the accompanying drawings and embodiments.

[0103] Example 1

[0104] This embodiment uses 6D pose estimation for an industrial robotic arm grasping an unknown part as an example: In an automobile manufacturing workshop, the robotic arm needs to grasp a textureless metal gear (a new object with no prior training data) from a jumble of parts. The gear surface is highly reflective and partially occluded (due to overlap with other parts).

[0105] According to such Figure 1 The process and implementation steps are as follows:

[0106] S1. Input the CAD model of the gear (STL format), RGB image (1920×1080, taken by a camera fixed above the worktable), and camera intrinsics.

[0107] The S2.CLIP-DETR detector identifies the gear bounding box (confidence 0.92), which is then cropped and adjusted to a resolution of 224×224. Intrinsic parameters are optimized using a depth map estimated by a large model (correcting focal length deviation caused by reflections).

[0108] S3. Generate semantic cues “metal gear, tooth face up”, and generate embeddings through multimodal large model generation to jointly probabilistically model the symmetry constraints of the gear.

[0109] S4. Generate 256 pose candidates, where the initial distribution of the gear's 6D pose (rotation + translation) is sampled around the axis of symmetry of the CAD model.

[0110] S5. Calculate the reprojection error using the projection function and retain the Top-64 candidates (error threshold 0.8 pixels).

[0111] S6. Implicit Field Network (MLP) is used to model the gear shape. Regularization terms suppress reflective noise, and a depth map is generated in real time. Compared with the depth map, the depth consistency loss is <1mm.

[0112] S7.ViT extracts features from the image and rendered image, constructs a correlation volume, iterates 10 times to optimize optical flow, and corrects edge deviations caused by occlusion.

[0113] S8. Weighted joint optimization of projection loss (2D key point error) and semantic consistency loss (gear center symmetry).

[0114] S9. The confidence level was assessed using a multi-scale feature pyramid (P3-P7), with a final confidence level of 0.95 (92% of the region was of high confidence).

[0115] S10. The final 6D posture of the output gear: rotation matrix R (rotation around the Z-axis by 5°), translation vector t = (120mm, -30mm, 45mm). The robotic arm uses this to perform precise grasping with an error of <2mm.

[0116] Example 2

[0117] This embodiment takes the attitude estimation of an obstructed object in AR glasses-assisted maintenance as an example: When an engineer wears AR glasses to repair an aircraft engine, he needs to identify a fuel nozzle (a new object) that is partially obstructed by a pipeline and overlay virtual installation guidance on the AR interface.

[0118] according to Figure 1 The process and implementation steps are as follows:

[0119] S1. Input the 3D model of the nozzle (GLB format), the RGB image (640×480, dynamic viewpoint) captured by the AR glasses camera, and the initial values ​​of the camera intrinsic parameters.

[0120] S2.CLIP-DETR nozzle detection (confidence level 0.88), after cropping and adjustment to 224×224, large model depth map to assist in correcting principal point coordinates to compensate for lens distortion.

[0121] S3. Semantic hints "nozzle down, interface forward", embedding vectors are combined with the context (pipeline geometry) of the occluded area to adjust the prior distribution.

[0122] S4. Generate 256 attitude candidates, focusing on sampling the rotating subspace with the nozzle facing downwards (in the direction of gravity).

[0123] S5. Retain Top-64 candidates by using the occlusion-aware projection function (ignoring key points covered by the pipeline).

[0124] S6. Implicit field network modeling of the visible part of the nozzle, surface normal consistency constraint forces the unobserved area (the occluded part) to conform to the geometry of the CAD model.

[0125] S7.ViT feature extraction enhances the response to occlusion edges, and optical flow iteratively corrects the attitude deviation of the nozzle base (converging after 5 iterations).

[0126] S8. Semantic consistency loss prioritizes matching the visible edges of the nozzle interface, while projection loss is dynamically weighted (weight of occluded areas is reduced).

[0127] S9. Multiscale confidence decomposition: deterministic weight (visible area) accounts for 0.85, attitude sensitivity weight (optical flow consistency) accounts for 0.78, and the overall confidence is 0.81.

[0128] S10. Output nozzle 6D attitude: rotation matrix R (pitch angle -90°, no yaw), translation vector t = (50mm, 20mm, 200mm). AR glasses overlay virtual guidance in real time with an error of <3mm. Obstructed areas are displayed by completing the display using CAD models.

[0129] This invention achieves breakthroughs in efficiency and accuracy for 6D pose estimation in industrial scenarios by integrating optical flow shape perception mechanisms with dynamic NeRF-large model collaborative optimization. Its core innovation lies in proposing an iterative optimization strategy based on single-view rendering. A differentiable PnP solver directly maps 2D-3D projection errors to 6D pose parameters, avoiding the computational redundancy of traditional multi-view rendering. Combined with dynamic NeRF shape modeling, geometric consistency loss (such as depth error) is used to constrain the pose space, resulting in lower pose estimation errors for textureless objects compared to traditional methods. This design not only avoids the computational burden of extensive view rendering required in traditional methods but also achieves efficient utilization of 3D shape information, fundamentally solving the trade-off between computational efficiency and estimation accuracy.

[0130] This invention designs a multi-scale feature pyramid and an optical flow refinement network, achieving coarse-to-fine pose optimization through a cascaded architecture. It innovatively splits the confidence score into deterministic weights (based on feature response intensity) and pose sensitivity weights (based on optical flow consistency), improving the success rate in occluded scenarios. Furthermore, it uses GPT-4V to control NeRF dynamic modeling, supporting real-time shape reconstruction in scenarios without 3D models. Combined with CLIP text prompts to generate semantic constraints (such as "handle to the left"), it reduces the misjudgment rate of symmetrical object poses, breaking through the dependence of traditional methods on predefined shape priors.

Claims

1. A method for semantic understanding and pose generation of robot visual objects based on a large model, characterized in that, Includes the following steps: S1. Input image and 3D model; Input the 3D model file of the target object and the RGB image of the target object whose pose needs to be estimated; For input image calibration, the camera intrinsic parameter matrix K is in the form of: Where f x ,f y It is the focal length parameter, (c x ,c y () are the coordinates of the principal point; S2. Object Detection: Object detection is performed using a CLIP-DETR detector, and the image is cropped and transformed to adjust the cropped image to a fixed resolution; the estimation results of scene depth information from a large model are introduced to assist in adjusting camera intrinsic parameters; the adjusted image is standardized and then pose representation and predefined rotation are performed; S3. Multimodal feature alignment; The input image extracts features through the visual encoder of a large model, forms multiple sets of semantic cues, and generates embeddings through a multimodal large model, jointly performing probabilistic modeling. S4. Diffusion model sampling; Using the UNet architecture and a linear noise scheduling strategy, pose candidates are generated. S5. Geometric screening: Optimize and retain candidate poses with the minimum error using projection functions and gradient descent. S6.NeRF morphological modeling; employs implicit field modeling and real-time rendering; S7. Optical flow optimization: Features of the input and rendered images are extracted using the ViT model, and the correlation volume is constructed and continuously optimized through optical flow refinement. S8. Joint loss calculation; calculate the projection loss and semantic consistency loss separately; S9. Confidence analysis; Construct a feature pyramid and employ multi-scale confidence assessment; S10. Output the attitude result.

2. The method for semantic understanding and pose generation of robot visual objects based on a large model according to claim 1, characterized in that, S2 specifically includes the following sub-steps: S2.

1. Object detection is performed using a CLIP-DETR detector: Use a pre-trained large model to extract deep features from the image; The detection head network consists of two branches: Bounding box regression branch: Outputs the 2D bounding box coordinates (x, y, w, h) of the object; Category classification branch: Predicts the probability distribution of object categories; The detection results are represented as a list: D = {di}, where each detection di contains: Bounding box coordinates: b i =(x i ,y i ,w i ,h i ); Category tag: c i Detection confidence score: s i ; S2.

2. Image cropping transformation: I crop =Crop(I,b i ) (1) The Crop function is based on the bounding box coordinates b. i Extract the target region from the original image I; S2.

3. Adjust the cropped image to a fixed resolution (H, W): I resize =Resize(I crop ,(H,W)) (2) S2.

4. Adjust camera parameters: The estimation results of scene depth information by a large model are introduced to help adjust the camera intrinsic parameters; the focal length parameters fx and fy in the camera intrinsic parameter matrix K, as well as the principal point coordinates (cx, cy) are optimized using the depth map generated by the large model. S2.

5. Standardize the adjusted image: I norm =(I resize -m) / s (3) Where μ is the mean of the image intensity values, and σ is the standard deviation of the image intensity values; S2.

6. Attitude representation and predefined rotations; The system uses Euler angles to represent 3D rotations, with each rotation defined by three angles. Where ψ is the yaw angle and θ is the pitch angle. It is the roll angle; N uniformly distributed sets of rotated samples are predefined: {R} i (i = 1 to N).

3. The method for robot visual object semantic understanding and pose generation based on a large model according to claim 2, characterized in that, The specific method for S3 is as follows: The input image has features extracted by a visual encoder of a large model; Generate multiple sets of semantic prompts and generate embeddings through a large model. Joint probability modeling: Diffusion t This is the reverse process of the DDIM diffusion model.

4. The method for semantic understanding and pose generation of robot visual objects based on a large model according to claim 3, characterized in that, The specific method for S4 is as follows: The UNet structure is adopted, with the number of channels ranging from 64 to 128 to 256 to 512, and the embedding dimension of the time step being 256. A linear noise scheduling strategy is adopted, with a total time step T = 1000; Generate N = 256 pose candidates {P k =[R k , t k ]} N k＝1 , where R k For a 3×3 rotation matrix, t k It is a 3×1 translation vector.

5. The robot visual object semantic understanding and pose generation method based on a large model according to claim 4, characterized in that, The specific method for S5 is as follows: Projection function: Reprojection error calculation: Gradient descent optimization: minimizing ∈ using the Adam optimizer k The Top-64 candidate poses with the smallest error are retained.

6. The robot visual object semantic understanding and pose generation method based on a large model according to claim 5, characterized in that, The specific method for S6 is as follows: S6.1 Implicit Field Modeling: Network architecture: MLP network, input is 3D coordinate x and view direction d, output is voxel density σ and color c: σ,c=MLP(X,d) (7) Regularization term: Training objectives: S6.2 Real-time Rendering: Generate N r A ray of light r j Sample N s Points {X j ,k}; Volumetric rendering: in Output rendering depth map D render and semantic mask M render ; S6.3 Shape Consistency Constraint: Deep consistency loss: Surface normal consistency:

7. The method for semantic understanding and pose generation of robot visual objects based on a large model according to claim 6, characterized in that, The specific method for S7 is as follows: Feature extraction: Input image I norm and rendering image I render Features are extracted using ViT, and sine / cosine position encodings are added to the features: F I =ViT(I norm )∈R H / 16×W / 16×D F R =ViT(I render )∈R H / 16×W / 16×D (13) Correlation volume construction: Where i and j are spatial indices, and k is the depth index; Optical flow refinement: Input the optical flow Φ from the previous moment. t-1 and correlation feature C ijk Output the current optical flow Φ t : F t+1 =GUR(Φ t ,C ijk ) (15) Set the maximum number of iterations T = 10, and the learning rate η = 0.01 per step.

8. The method for semantic understanding and pose generation of robot visual objects based on a large model according to claim 7, characterized in that, The specific method for S8 is as follows: Joint optimization objective function: Projection loss: Semantic consistency loss:

9. The method for robot visual object semantic understanding and pose generation based on a large model according to claim 8, characterized in that, The specific method for S9 is as follows: Feature pyramid construction: Extract features from layers P3-P7 of ViT and calculate projection errors at each scale. Uncertainty estimation: Where σ is the Softplus activation function.

10. The method for semantic understanding and pose generation of robot visual objects based on a large model according to claim 1, characterized in that, The output of S10 includes: Final stance: Obtained by minimization get; Confidence level: in For rendering mask.

Images