Three-dimensional assembly method based on single-view image and large language model guidance

Abstract

The application discloses a three-dimensional assembly method based on a single-view image and a large language model guide, which is used for automatic assembly of a service robot. A semantic segmentation model and an edge detection algorithm are used to segment components in a single-view image and obtain corresponding semantic class labels, so that the single-view image is accurately analyzed. Fixed prompt words are designed to activate assembly prior knowledge contained in the large language model, form component selection rules, guide the component selection process of three-dimensional assembly, and solve the partial occlusion problem in the single-view image. Finally, the component semantic class information analyzed from the single-view image and the assembly sequence information generated by the large language model are encoded into a component assembly network based on a Transformer architecture, the selected components are assembled, and the correct assembly pose is predicted. The generalization ability of the algorithm is greatly improved, and the precision of three-dimensional assembly is improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the field of three-dimensional assembly in computer vision, and particularly relates to a three-dimensional assembly method based on a single-view image and a large language model. BACKGROUND

[0002] Up to now, robots have participated in various fields of human life, and the role of robots has gradually extended from production and manufacturing to life service, completing the upgrade from traditional industrial robots to service robots. Three-dimensional assembly is one of the typical application scenarios of service robots, which mainly includes two stages of component selection and component assembly. The component selection stage requires the robot to select all the components required for assembling a specific assembly body in combination with the specific assembly scene; the component assembly stage requires the robot to predict the pose of each selected component and assemble it completely. The three-dimensional assembly task involves a very large solution space, which is a tedious and time-consuming process even for humans, and is of high difficulty for service robots.

[0003] The existing method often trains a three-dimensional assembly model based on a data set with a specific distribution, so as to remember fixed assembly patterns. Such method has serious limitations. Given a set of selected components, the model can only provide a fixed style of assembly scheme that meets the specific distribution of the training data set, and the generalization is difficult to meet the actual demand. SUMMARY

[0004] The technical problem to be solved by the application is to provide a three-dimensional assembly method based on a single-view image and a large language model, which solves the problem of the prior art that the assembly method only learns fixed assembly patterns within the distribution of the training data set.

[0005] The application adopts the following technical solutions to solve the above technical problems:

[0006] The three-dimensional assembly method based on a single-view image and a large language model comprises the following steps:

[0007] Step S1, using a trained semantic segmentation model and an edge detection algorithm to segment the assembly body in the single-view image at the component level and obtain the corresponding semantic class label;

[0008] Step S2, using the assembly prior knowledge contained in the large language model to guide the component selection stage of three-dimensional assembly, and forming certain component selection rules;

[0009] Step S3, combining the component selection rules generated by the large language model, matching the segmented components with the components in the selected component set one by one, and selecting the component in the selected component set with the highest similarity to the segmentation result as the assembly component;

[0010] Step S4, encode the part semantic category information parsed from the single-view image and the assembly sequence information generated by the large language model into the part assembly network based on the Transformer architecture, predict the correct assembly pose, and assemble the selected parts.

[0011] The specific process of step S1 is as follows:

[0012] Step S1.1, construct a dataset containing assembly finished product images;

[0013] Step S1.2, select the assembly finished product image in the dataset as the single-view image for guiding three-dimensional assembly, and use the Canny operator to detect and identify the edge lines of the parts of the assembly, and obtain the optimal edge contour of the single-view image;

[0014] Step S1.3, perform part-level segmentation and semantic category detection on the single-view image, identify the corresponding category objects in the picture based on the text description, and segment the corresponding object mask;

[0015] Step S1.4, segment all the parts contained in the image, and obtain the semantic category label of each part according to the pre-input text containing all the categories of the parts to be detected.

[0016] The specific process of using the Canny operator to detect and identify the edge lines of the parts of the assembly and obtain the optimal edge contour of the single-view image is as follows:

[0017] Step 1.2.1, filter out possible high-frequency noise in the image using a Gaussian filter;

[0018] Step 1.2.2, calculate the gradient G(x,y) and angle θ(x,y) of any pixel point in the image;

[0019] Step 1.2.3, suppress non-maximum points of the gradient image, eliminate non-edge pixel points, and find those points as edge points that are local maximum values in the gradient image;

[0020] Step 1.2.4, use a double-threshold algorithm to detect and connect the edges, set the size of the two thresholds to 1:2, and determine the false edge for the pixel points less than the low threshold; and determine the strong edge for the pixel points greater than the high threshold.

[0021] The specific process of step S2 is as follows:

[0022] Step S2.1, design a fixed prompt word template to activate the assembly prior knowledge contained in the large language model, and parse the part semantic category information according to the input task scene description;

[0023] Step S2.2, fill in the prompt word template with the component semantic category label obtained in step S2.1 to form a specific prompt word corresponding to the current single-view image, and input it into the large language model;

[0024] Step S2.3, the large language model obtains the prompt information of the components that are not successfully segmented and the correct assembly order of all components according to the summarized assembly component selection rule.

[0025] The specific process of step S3 is as follows:

[0026] Step S3.1, project the three-dimensional components in the set of components to be selected to form a set of two-dimensional images of the components to be selected;

[0027] Step S3.2, calculate the cosine similarity of the component mask image in the segmentation result and the images in the set of two-dimensional images of the components to be selected one by one, and the combination with the highest cosine similarity is regarded as a successful match, and the successfully matched components are removed from the set of components to be selected;

[0028] Step S3.3, according to the prompt information given by the large language model in step 2 and the summarized assembly component selection rule, correct and complete the component selection result.

[0029] In step S3.2, fix the images in the set of two-dimensional images of the components to be selected to rotate 8 angles, respectively calculate the cosine similarity of each image with the component mask image in the segmentation result, and take the average value as the final result.

[0030] The specific process of step S4 is as follows:

[0031] Step S4.1, build a component assembly network based on the Transformer architecture;

[0032] Step S4.2, predict the correct assembly pose of the component according to the component semantic category encoding and the component assembly order encoding.

[0033] The component assembly network based on the Transformer architecture includes three parts, namely the PointNet module, the encoder and the predictor.

[0034] The specific steps of predicting the correct assembly pose of the component include:

[0035] Step 1, apply the PointNet module to extract the three-dimensional point cloud features of the component, map the three-dimensional point cloud features of the component to the same dimension feature space, and learn a unified vector representation;

[0036] Step 2, apply the encoder to encode the component, and use the self-attention module to infer and model the geometric and spatial position relationship between the components;

[0037] Step 3, the predictor is composed of a 3-layer feedforward network, and the pose prediction is based on the part feature output by the encoder.

[0038] By fusing the part semantic category encoding and the part assembly order encoding, the part point cloud feature is added with semantic category information and assembly order information.

[0039] Compared with the prior art, the present application has the following beneficial effects:

[0040] 1. Part semantic information is clear: the three-dimensional assembly method proposed in the present application realizes the understanding of part semantic information in the process of analyzing single-view images through fine-tuning training of the Grounded-SAM model on specific tasks, and the rationality and purpose of assembly are stronger compared with the traditional method without explicit analysis of part semantic information.

[0041] 2. High accuracy: the three-dimensional assembly method proposed in the present application improves the precision of the three-dimensional assembly method by introducing the assembly prior knowledge contained in the large language model and fusing the part semantic category information and the assembly order information.

[0042] 3. Good generalization performance: the three-dimensional assembly method proposed in the present application solves the defect that the existing assembly method can only remember the fixed assembly mode within the distribution of the training data set to a certain extent by introducing the guidance of single-view images and large language models, greatly improving the generalization ability of the algorithm. BRIEF DESCRIPTION OF DRAWINGS

[0043] Figure 1 An example of a single-view image used in the present application is shown in the figure.

[0044] Figure 2 A schematic diagram of the workflow of the present application is shown in the figure.

[0045] Figure 3 A structural schematic diagram of the part assembly network of the present application is shown in the figure. DETAILED DESCRIPTION

[0046] The structure and working process of the present application will be further described below with reference to the accompanying drawings.

[0047] The application aims to provide a three-dimensional assembly method based on a single-view image and a large language model guide, for automatic assembly of a service robot. The application uses a semantic segmentation model and an edge detection algorithm to segment the assembly body in the single-view image at the component level and obtain the corresponding semantic class label, to realize accurate analysis of the single-view image; a fixed prompt word is designed to activate the assembly prior knowledge contained in the large language model, to form a component selection rule, thereby guiding the component selection process of three-dimensional assembly, and solving the partial occlusion problem in the single-view image; finally, the component semantic class information analyzed from the single-view image and the assembly sequence information generated by the large language model are encoded into a component assembly network based on the Transformer architecture, to assemble the selected components and predict their correct assembly poses. By introducing the guidance of the single-view image and the large language model, the application solves the defect that the existing assembly method only learns the fixed assembly mode within the training data set distribution to a certain extent, greatly improves the generalization ability of the algorithm. At the same time, the fusion of component semantic class information and assembly sequence information improves the precision of three-dimensional assembly.

[0048] With the wide development and application of the current large language model, large language models such as GPT-4 and PaLM-E have shown strong reasoning ability, chain thinking ability and zero-shot generalization ability, and because a large-scale corpus is used in the training process, the large language model itself also contains rich prior knowledge, which can provide sufficient guidance for the three-dimensional assembly task, thereby realizing three-dimensional assembly with stronger generalization and higher accuracy.

[0049] The three-dimensional assembly method based on a single-view image and a large language model guide comprises the following steps:

[0050] Step S1, using a trained semantic segmentation model and an edge detection algorithm to segment the assembly body in the single-view image at the component level and obtain the corresponding semantic class label;

[0051] Step S2, using the assembly prior knowledge contained in the large language model to guide the component selection stage of three-dimensional assembly, to form certain component selection rules;

[0052] Step S3, combining the component selection rules generated by the large language model, matching the segmented components with the components in the candidate component set one by one, and selecting the component in the candidate component set with the highest similarity to the segmentation result as the assembly component;

[0053] Step S4, encoding the component semantic class information analyzed from the single-view image and the assembly sequence information generated by the large language model into a component assembly network based on the Transformer architecture, to predict the correct assembly pose and assemble the selected components.

[0054] In particular embodiments, as shown in Figures 1 to 3

[0055] The three-dimensional assembly method based on a single-view image and a large language model guidance comprises the following steps:

[0056] Step S1: using a trained semantic segmentation model and an edge detection algorithm to perform component-level segmentation on the assembly body in the single-view image and obtain the corresponding semantic class label. In this embodiment, the assembly finished product picture in the furniture assembly instruction is used as the single-view image for guiding three-dimensional assembly, as shown in the accompanying Figure 1

[0057] Such pictures are all single-color line drawings outlined by black lines. In order to obtain accurate segmentation and recognition results, a Canny operator is first used to detect and recognize the edge lines of the assembly body components, providing assistance for subsequent component segmentation.

[0058] The Canny operator is an edge detection operator that integrates filtering, enhancement, detection and other stages, and its goal is to find an optimal edge contour. The specific steps of the algorithm are as follows:

[0059] 1) Use a Gaussian filter operator to filter out possible high-frequency noise in the image.

[0060] The specific Gaussian filter operator used in this embodiment is:

[0061]

[0062] 2) Calculate the gradient G(x, y) and angle θ(x, y) of any pixel point in the image according to the following formula.

[0063]

[0064] where,

[0065] 3) Suppress non-maximum points of the gradient image to eliminate non-edge pixel points, and find those points with local maximum values in the gradient image as edge points.

[0066] 4) Use a double-threshold algorithm to detect and connect the edges, and set the size of the two thresholds to 1:2. The pixel points less than the low threshold are determined as false edges; while the pixel points greater than the high threshold are determined as strong edges.

[0067] Further, on the basis of the edge detection result, this embodiment uses a Grounded-SAM model to perform component-level segmentation and semantic class detection on the single-view image.

[0068] ​​The Grounded-SAM model can be regarded as a combination of the image segmentation base model SAM and the powerful zero-shot object detector Grounding DINO. With the powerful zero-shot detection capability of Grounding DINO, the Grounded-SAM model can identify the objects of the corresponding category in the picture through the text description, and then finely segment the mask of the corresponding object through the powerful segmentation capability of SAM.

[0069] For the furniture assembly scene in the three-dimensional assembly task, the embodiment constructs a dataset containing furniture assembly finished pictures, and fine-tunes the Grounded-SAM general model using the dataset to make it better meet the needs of specific tasks. Finally, the Grounded-SAM model can segment all the components contained in the assembly body in the image and obtain the semantic category label of each component according to the pre-input text prompt containing all the categories of the components to be detected.

[0070] Step S2: Use the assembly prior knowledge contained in the large language model to guide the component selection stage of three-dimensional assembly, form certain component selection rules, and solve the partial occlusion problem existing in the single-view image. The embodiment designs a fixed prompt word template to activate the assembly prior knowledge contained in the large language model, thereby guiding the component selection process of three-dimensional assembly. Specifically, the large language model prompt word template designed by the embodiment is composed of the following parts:

[0071] ①Task introduction: task scene description, prompting the large language model to call the relevant prior knowledge obtained in the pre-training process, and explaining to the large language model what task to perform.

[0072] ②Specific example: successful examples covering various situations in actual task scenarios, fine-tuning and correcting the relevant prior knowledge possessed by the large language model in specific scenarios.

[0073] ③Component semantics: component semantic category information parsed in the previous step.

[0074] Further, the component semantic category label obtained in step S1 is used to fill the above prompt word template to form a specific prompt corresponding to the current single-view image, which is input into the large language model. The large language model will prompt the missing components that are not successfully segmented in the current segmentation result and the correct assembly order of these components according to the assembly component selection rules summarized, thereby solving the problem of missing segmented components caused by the partial occlusion phenomenon existing in the single-view image. The large language model used in the embodiment is GPT-4, which currently performs outstanding performance in various tasks. It is used in the program in the form of API interface, and the parameter setting uses the default configuration.

[0075] Step S3: Combine the part selection rules generated by the large language model, match the segmented parts with the parts in the candidate part set one by one, and select the part with the highest similarity to the segmentation result in the candidate part set as the assembly part. This step specifically includes three parts:

[0076] 3.1) Project the three-dimensional parts in the candidate part set to form a set of two-dimensional images of the candidate parts.

[0077] 3.2) Calculate the cosine similarity between the part mask image in the segmentation result and the images in the two-dimensional image set of the candidate parts pair by pair. The combination with the highest similarity is considered a successful match, and the successfully matched parts will be removed from the candidate part set.

[0078] To solve the problem of inconsistent rotation angles between the segmented parts and the corresponding parts in the candidate part set, the images in the two-dimensional image set of the candidate parts are fixedly rotated by 8 angles, and the cosine similarity between each image and the segmented part mask image is calculated. The average value is taken as the final result. The specific calculation formula is as follows:

[0079]

[0080] where Rot k (I) represents rotating the image I clockwise by k°.

[0081] 3.3) According to the missing information of the segmentation result prompted by the large language model in step 2 and the assembly part selection rules summarized, correct and complete the part selection result.

[0082] Step S4: Encode the part semantic category information parsed from the single-view image and the assembly order information generated by the large language model into the part assembly network based on the Transformer architecture, and assemble the parts selected in the previous step to predict their correct assembly poses. The goal of part assembly is to find the correct assembly pose for the selected parts Predict the corresponding assembly pose where is the three-dimensional part point cloud, N is the total number of parts of the assembly, (R, t) is the 6-DOF pose in SE(3) space, representing the rotation quaternion and translation transformation respectively, and finally all parts will be assembled into a complete assembly according to their predicted poses. For the part assembly problem, this embodiment proposes a part assembly framework based on Transformer, which takes a set of three-dimensional point clouds of parts as input and estimates the part poses in SE(3) space as output. As shown in the accompanying Figure 3 , the part assembly framework specifically includes three components:

[0083] 1) PointNet extracts the point cloud features of the part, maps the three-dimensional point cloud of the part to the feature space of the same dimension, and learns a unified vector representation.

[0084] 2) The encoder is responsible for encoding the part, and uses the self-attention module to infer the geometric and spatial position relationship between the parts.

[0085] 3) The predictor consists of a 3-layer feedforward network, which performs pose prediction based on the part features output by the encoder.

[0086] At the same time, by fusing the part semantic category encoding and the part assembly order encoding, the part point cloud feature is added with semantic category information and assembly order information, solving the semantic ambiguity problem between parts.

[0087] A specific application example is provided below:

[0088] This application example takes the chair assembly in a three-dimensional assembly task as an example, installs Ubuntu 20.04 system on the control workstation, and installs Intel Core i9-10920x, 3.60GHz CPU, and GPU NVIDIA RTX 3090Ti*2. Based on these devices, the specific implementation method is introduced:

[0089] Use 1700 pieces of single-view image dataset as shown in the furniture assembly instruction manual Figure 1 The assembly finished product picture is constructed, and the training set and test set are divided according to the ratio of 7:3. About 50 kinds of typical parts in the Partnet chair dataset are selected to construct the candidate part set.

[0090] After the dataset is constructed, the following process is executed, that is:

[0091] S1: Perform part-level segmentation on the chair in the single-view image and obtain the corresponding semantic category label. First, use the Canny operator to detect and identify the edge lines of the parts in the image, which provides assistance for subsequent part segmentation. On the basis of the edge detection result, this embodiment uses the Grounded-SAM model to perform part-level segmentation and semantic category detection on the single-view image. The single-view image training set is used to fine-tune the Grounded-SAM model on the chair part segmentation task. The prompt containing all chair part categories is set as the text input of the Grounded-SAM model, and finally the Grounded-SAM model can segment all parts contained in the image and obtain the corresponding semantic category label.

[0092] S2: Utilize the assembly prior knowledge in the large language model to guide the part selection stage of three-dimensional assembly. In this embodiment, the GPT-4 large language model is accessed in the program by calling the API interface, and the parameters use the default configuration. The obtained part semantic category label is filled into the pre-designed prompt word template and input into the large language model. The large language model summarizes the part selection rules of chair assembly, prompts the missing parts that are not successfully segmented in the current segmentation result, and the correct assembly order of all parts.

[0093] S3: Combine the part selection rules to match the segmented parts and the selected parts one by one. Project the three-dimensional parts in the selected part set to form a two-dimensional image set of the selected parts. Fix the rotation of the images in the two-dimensional image set of the selected parts at 8 angles, respectively, and calculate the cosine similarity with the segmented part mask image. Take the average value as the final result. According to the calculation result, the combination with the highest similarity is considered as successful matching, and the successfully matched parts will be removed from the selected part set. According to the prompt information of the large language model for the missing segmentation result and the summarized assembly part selection rules, the part selection result is corrected and completed to obtain the final part selection result.

[0094] S4: Fusion of part semantic category information and assembly order information based on Transformer part assembly. The part semantic category information and the part assembly order information are fused by using the Transformer-based part assembly framework proposed in the application to realize the correct assembly pose prediction of the parts.

[0095] Those skilled in the art should understand that the above-mentioned modules or steps of the present application can be realized by a general portable device, and alternatively, they can be realized by program codes executable by a computing device. The present application is not limited to any specific combination of hardware and software. The above description is only the preferred embodiment of the present application and is not intended to limit the present application. Those skilled in the art can make various changes and modifications to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principles of the present application shall be included in the protection scope of the present application.

Claims

1. A 3D assembly method based on single-view images and guided by a large language model, characterized by: The steps include: Step S1: Use the trained semantic segmentation model Grounded-SAM and edge detection algorithm to segment the assembly at the component level in the single-view image and obtain the corresponding semantic category labels; construct a dataset containing assembly finished product images; select the assembly finished product images in the dataset as the single-view images to guide the three-dimensional assembly; Step S2: Use the assembly prior knowledge contained in the large language model to guide the component selection stage of the three-dimensional assembly and form certain component selection rules. The specific process of step S2 is as follows: Step S2.1: Design a fixed prompt word template to activate the assembly prior knowledge contained in the large language model; Step S2.2: Fill the prompt word template with the component semantic category labels obtained in step S1 to form a specific prompt word corresponding to the current single-view image, and input it into the large language model; Step S2.3: The large language model obtains the prompt information of the missing parts that were not successfully segmented in the current segmentation result and the correct assembly order of all parts based on the summarized assembly part selection rules; Step S3: Combine the component selection rules generated by the large language model to match the segmented components with the components in the set of components to be selected one by one, and select the component in the set of components to be selected that has the highest similarity with the segmentation result as the assembly component; Step S4: Encode the component semantic category labels parsed from the single-view image and the assembly order information generated by the large language model into a component assembly network based on the Transformer architecture, predict its correct assembly pose, and assemble the selected components.

2. The 3D assembly method based on single-view image and large language model guidance according to claim 1, characterized in that: The specific process of step S1 is as follows: Step S1.1, construct a dataset containing assembly finished product images; Step S1.2: Select the finished assembly image in the dataset as the single-view image to guide the 3D assembly, use the Canny operator to detect and identify the edge lines of the assembly parts, and obtain the optimal edge contour of the single-view image; Step S1.3: Perform component-level segmentation and semantic category detection on the single-view image, identify the components of the corresponding category in the image based on the text description, and segment the masks of the corresponding components; Step S1.4: Segment all components contained in the assembly in the image, and obtain the semantic category label of each component based on the pre-input text containing the categories of all components to be inspected.

3. The 3D assembly method based on single-view image and large language model guidance according to claim 2, characterized in that: The specific process of using the Canny operator to detect and identify the edge lines of assembly parts and obtain the optimal edge contour of a single-view image is as follows: Step 1.2.1: Use the Gaussian filter operator to filter out any high-frequency noise that may exist in the image. Step 1.2.

2. Calculate the gradient G(x,y) and angle θ(x,y) of any pixel in the image. Step 1.2.3: Suppress the non-maximum points of the gradient image, remove non-edge pixels, and find the points with local maximum values ​​in the gradient image as edge points; Step 1.2.4: Use a dual-threshold algorithm to detect and connect edges. Set the ratio of the two thresholds to 1:

2. Pixels smaller than the lower threshold are considered false edges, while pixels larger than the higher threshold are considered strong edges.

4. The 3D assembly method based on single-view image and large language model guidance according to claim 1, characterized in that: The specific process of step S3 is as follows: Step S3.1, projecting the three-dimensional components in the set of components to be selected to form a two-dimensional image set of the components to be selected; Step S3.2: Calculate the cosine similarity between the component mask image in the segmentation result and the images in the set of two-dimensional images of the selected components one by one. The combination with the highest cosine similarity is considered a successful match, and the successfully matched component is removed from the set of selected components. Step S3.3: Correct and complete the component selection results based on the prompt information provided by the large language model in step S2 and the summarized assembly component selection rules.

5. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 4, characterized in that: In step 3.2, the images in the two-dimensional image set of the part to be selected are rotated by a fixed angle of 8, and the cosine similarity between them and the mask image of the part in the segmentation result is calculated respectively, and the average value is taken as the final result.

6. The 3D assembly method based on single-view image and large language model guidance according to claim 1, characterized in that: The specific process of step S4 is as follows: Step S4.1: Build a component assembly network based on the Transformer architecture; Step S4.2: Predict the correct assembly pose of the component based on the component semantic category label encoding and the component assembly order encoding.

7. The 3D assembly method based on single-view image and large language model guidance according to claim 6, characterized in that: The component assembly network based on the Transformer architecture consists of three parts: the PointNet module, the encoder, and the predictor.

8. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 7, characterized in that: Predicting the correct assembly pose of a component includes the following steps: Step 1: Apply the PointNet module to extract the 3D point cloud features of the component, map the 3D point cloud features of the component to a feature space of the same dimension, and learn a unified vector representation; Step 2: Encode the components using the encoder and use the self-attention module to infer the geometric and spatial position relationships between the modeled components. Step 3: The predictor consists of a 3-layer feedforward network and performs pose prediction based on the component features output by the encoder.

9. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 8, characterized in that: By fusing component semantic category label encoding and component assembly order encoding, semantic category information and assembly order information are added to component point cloud features.

Images