Three-dimensional assembly method based on single-view image and large language model guidance

Abstract

The invention discloses a three-dimensional assembly method based on single-view image and large language model guidance, which is used for automatic assembly of a service robot. Performing part-level segmentation on the assembly in the single-view image by using a semantic segmentation model and an edge detection algorithm, and obtaining a corresponding semantic category label to realize accurate analysis of the single-view image; a fixed cue word is designed to activate assembly priori knowledge contained in a large language model, and a component selection rule is formed, so that the component selection process of three-dimensional assembly is guided, and the problem of partial shielding existing in a single-view image is solved; and finally, encoding the component semantic category information analyzed from the single-view image and the assembly sequence information generated by the large language model into a component assembly network based on a Transform architecture, assembling the selected component, and predicting a correct assembly pose of the selected component. The generalization ability of the algorithm is greatly improved, and the precision of three-dimensional assembly is improved.

Description

Technical Field

[0001] The invention belongs to the field of three-dimensional assembly in computer vision, and in particular relates to a three-dimensional assembly method guided by a single-view image and a large language model. Background Art

[0002] Today, robots have been involved in all areas of human life, and the role of robots has gradually extended from manufacturing to life services, completing the upgrade from traditional industrial robots to service robots. Three-dimensional assembly is one of the typical application scenarios of service robots. This task can be mainly divided into two stages: component selection and component assembly. The component selection stage requires the robot to select all the components required to assemble a specific assembly based on the specific assembly scenario; the component assembly stage requires the robot to predict the position and posture of each selected component and assemble it completely. The three-dimensional assembly task involves a very large solution space. Even for humans, it is a tedious and time-consuming process, and it is even more difficult for service robots.

[0003] Existing practices often train 3D assembly models based on data sets with specific distributions, so that they remember fixed assembly patterns. This approach has serious limitations. Given a set of candidate parts, the model can only provide fixed-style assembly solutions that conform to the specific distribution of the training data set, and generalization is difficult to meet actual needs. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a three-dimensional assembly method based on single-view images and large language model guidance, which solves the problem that the assembly method in the prior art only learns the defect of fixed assembly patterns within the distribution of training data sets.

[0005] The present invention adopts the following technical solutions to solve the above technical problems: The three-dimensional assembly method based on single-view image and large language model guidance includes the following steps: Step S1, using the trained semantic segmentation model and edge detection algorithm to perform component-level segmentation on the assembly in the single-view image and obtain the corresponding semantic category labels; Step S2: using the assembly prior knowledge contained in the large language model to guide the component selection stage of the three-dimensional assembly and form certain component selection rules; Step S3: Combine the component selection rules generated by the large language model, match the segmented components with the components in the set of components to be selected one by one, and select the component in the set of components to be selected that has the highest similarity with the segmentation result as the assembly component; Step S4: Encode the component semantic category information parsed from the single-view image and the assembly order information generated by the large language model into a component assembly network based on the Transformer architecture, predict its correct assembly posture, and assemble the selected components.

[0006] The specific process of step S1 is as follows: Step S1.1, construct a data set containing assembly finished product images; Step S1.2, selecting the finished assembly image in the data set as the single-view image for guiding the three-dimensional assembly, using the Canny operator to detect and identify the edge lines of the assembly parts, and obtaining the optimal edge contour of the single-view image; Step S1.3, performing component-level segmentation and semantic category detection on the single-view image, identifying objects of corresponding categories in the image based on the text description, and segmenting the masks of the corresponding objects; Step S1.4: Segment all the components contained in the assembly in the image, and obtain the semantic category label of each component based on the pre-input text containing the categories of all the components to be inspected.

[0007] The specific process of using the Canny operator to detect and identify the edge lines of assembly parts and obtain the optimal edge contour of the single-view image is as follows: Step 1.2.1, use Gaussian filter operator to filter out the high-frequency noise that may exist in the image; Step 1.2.2: Calculate the gradient of any pixel in the image and angle ; Step 1.2.3, suppress the non-maximum points of the gradient image, remove non-edge pixels, and find the points with local maximum values ​​in the gradient image as edge points; Step 1.2.4: Use a dual threshold algorithm to detect and connect edges. Set the size of the two thresholds to 1:2. Pixels smaller than the low threshold are determined as false edges, while pixels larger than the high threshold are determined as strong edges.

[0008] The specific process of step S2 is as follows: Step S2.1, design a fixed prompt word template to activate the assembly prior knowledge contained in the large language model, and parse the component semantic category information according to the input task scenario description; Step S2.2, using the component semantic category labels obtained in step S2.1 to fill in the prompt word template, forming a specific prompt word corresponding to the current single-view image, and inputting it into the large language model; Step S2.3: The large language model obtains the prompt information of the missing parts that have not been successfully segmented in the current segmentation result and the correct assembly order of all parts according to the summarized assembly part selection rules.

[0009] The specific process of step S3 is as follows: Step S3.1, projecting the three-dimensional parts in the set of parts to be selected to form a two-dimensional image set of the parts to be selected; Step S3.2, calculating the cosine similarity between the component mask image in the segmentation result and the image in the two-dimensional image set of the component to be selected one by one, the combination with the highest cosine similarity is regarded as a successful match, and the successfully matched component is removed from the set of components to be selected; Step S3.3: According to the prompt information given by the large language model in step 2 and the summarized assembly component selection rules, the component selection results are corrected and completed.

[0010] In step S3.2, the images in the two-dimensional image set of the part to be selected are rotated at fixed angles of 8, and their cosine similarities with the mask images of the parts in the segmentation results are calculated respectively, and the average value is taken as the final result.

[0011] The specific process of step S4 is as follows: Step S4.1, building a component assembly network based on the Transformer architecture; Step S4.2: predict the correct assembly posture of the component based on the component semantic category coding and the component assembly order coding.

[0012] The component assembly network based on the Transformer architecture consists of three parts: PointNet module, encoder and predictor.

[0013] Predicting the correct assembly pose of a component includes the following steps: Step 1: Use the PointNet module to extract the 3D point cloud features of the components, map the 3D point cloud features of the components to the feature space of the same dimension, and learn a unified vector representation; Step 2: Encode the components using the encoder, and use the self-attention module to infer the geometric and spatial position relationships between the modeled components; Step 3: The predictor consists of a 3-layer feedforward network and performs pose prediction based on the component features output by the encoder.

[0014] By fusing the component semantic category coding and the component assembly order coding, semantic category information and assembly order information are added to the component point cloud features.

[0015] Compared with the prior art, the present invention has the following beneficial effects: 1. Clear semantic information of components: The three-dimensional assembly method proposed in the present invention realizes the understanding of component semantic information in the process of parsing single-view images by fine-tuning the Grounded-SAM model on specific tasks. Compared with the traditional method that does not clearly parse the semantic information of components, the rationality and purpose of assembly are stronger.

[0016] 2. High accuracy: The three-dimensional assembly method proposed in the present invention improves the accuracy of the three-dimensional assembly method by introducing the assembly prior knowledge contained in the large language model and integrating the component semantic category information and assembly order information.

[0017] 3. Good generalization performance: The three-dimensional assembly method proposed in the present invention solves the defect that the existing assembly method can only remember fixed assembly patterns within the distribution of the training data set to a certain extent by introducing the guidance of single-view images and large language models, thereby greatly improving the generalization ability of the algorithm. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] Figure 1 This is an example of a single-view image used in the present invention.

[0019] Figure 2 It is a schematic diagram of the workflow of the present invention.

[0020] Figure 3 It is a schematic diagram of the structure of the component assembly network of the present invention. DETAILED DESCRIPTION

[0021] The structure and working process of the present invention will be further described below in conjunction with the accompanying drawings.

[0022] The purpose of the present invention is to provide a three-dimensional assembly method guided by a single-view image and a large language model, which is used for the automatic assembly of a service robot. The present invention uses a semantic segmentation model and an edge detection algorithm to segment the assembly in the single-view image at the component level and obtain the corresponding semantic category label, so as to achieve accurate analysis of the single-view image; a fixed prompt word is designed to activate the assembly prior knowledge contained in the large language model, form a component selection rule, thereby guiding the component selection process of the three-dimensional assembly and solving the partial occlusion problem existing in the single-view image; finally, the component semantic category information parsed from the single-view image and the assembly order information correspondingly generated by the large language model are encoded into a component assembly network based on the Transformer architecture, the selected components are assembled, and their correct assembly posture is predicted. By introducing the guidance of the single-view image and the large language model, the present invention solves the defect that the existing assembly method only learns the fixed assembly mode within the distribution of the training data set to a certain extent, and greatly improves the generalization ability of the algorithm. At the same time, the component semantic category information and the assembly order information are integrated to improve the accuracy of the three-dimensional assembly.

[0023] With the widespread development and application of large language models, large language models such as GPT-4 and PaLM-E have demonstrated strong reasoning ability, chain thinking ability and zero-sample generalization ability. In addition, due to the use of large-scale corpora in the training process, the large language models themselves also contain rich prior knowledge, which can provide sufficient guidance for 3D assembly tasks, thereby achieving more generalized and accurate 3D assembly.

[0024] The three-dimensional assembly method based on single-view image and large language model guidance includes the following steps: Step S1, using the trained semantic segmentation model and edge detection algorithm to perform component-level segmentation on the assembly in the single-view image and obtain the corresponding semantic category labels; Step S2: using the assembly prior knowledge contained in the large language model to guide the component selection stage of the three-dimensional assembly and form certain component selection rules; Step S3: Combine the component selection rules generated by the large language model, match the segmented components with the components in the set of components to be selected one by one, and select the component in the set of components to be selected that has the highest similarity with the segmentation result as the assembly component; Step S4: Encode the component semantic category information parsed from the single-view image and the assembly order information generated by the large language model into a component assembly network based on the Transformer architecture, predict its correct assembly posture, and assemble the selected components.

[0025] Specific embodiments, such as Figures 1 to 3 As shown, The three-dimensional assembly method based on single-view image and large language model guidance includes the following steps: Step S1: Use the trained semantic segmentation model and edge detection algorithm to segment the assembly in the single-view image at the component level and obtain the corresponding semantic category label. This embodiment uses the assembly finished product picture in the IKEA furniture assembly manual as the single-view image to guide the three-dimensional assembly, as shown in the attached figure. Figure 1 shown.

[0026] These images are all monochrome line drawings outlined by black lines. In order to obtain accurate segmentation and recognition effects, the Canny operator is first used to detect and recognize the edge lines of assembly parts to provide assistance for subsequent part segmentation.

[0027] The Canny operator is an edge detection operator that combines filtering, enhancement, detection and other stages. Its goal is to find an optimal edge contour. The specific steps of the algorithm are as follows: 1) Use the Gaussian filter operator to filter out the high-frequency noise that may exist in the image.

[0028] The specific Gaussian filter operator used in this embodiment is:

[0029] The gradient of any pixel in the image is calculated according to the following formula and angle ,

[0030]

[0031] in, ,

[0032] 3) Suppress the non-maximum points of the gradient image to remove non-edge pixels, and find the points with local maximum values ​​in the gradient image as edge points.

[0033] 4) Use the dual threshold algorithm to detect and connect edges, set the size of the two thresholds to 1:2, and the pixels less than the low threshold will be determined as false edges; while the pixels greater than the high threshold will be determined as strong edges.

[0034] Furthermore, based on the edge detection results, this embodiment uses the Grounded-SAM model to perform component-level segmentation and semantic category detection on the single-view image.

[0035] The Grounded-SAM model can be regarded as a combination of the basic image segmentation model SAM and the powerful zero-sample target detector Grounding DINO. With the help of Grounding DINO's powerful zero-sample detection capability, the Grounded-SAM model can identify objects of the corresponding category in the image through text descriptions, and then use SAM's powerful segmentation capability to fine-grainedly segment the masks of the corresponding objects.

[0036] For the furniture assembly scenario in the three-dimensional assembly task, this embodiment constructs a dataset containing IKEA furniture assembly finished product images, and uses the dataset to fine-tune the Grounded-SAM general model to better meet the needs of specific tasks. Finally, the Grounded-SAM model can segment all the components contained in the assembly in the image, and obtain the semantic category label of each component based on the pre-input text prompt containing all the categories of the components to be inspected.

[0037] Step S2: Use the assembly prior knowledge contained in the large language model to guide the component selection stage of the three-dimensional assembly, form certain component selection rules, and solve the partial occlusion problem in the single-view image. This embodiment designs a fixed prompt word template to activate the assembly prior knowledge contained in the large language model, thereby guiding the component selection process of the three-dimensional assembly. Specifically, the large language model prompt word template designed in this embodiment consists of the following parts: ①Task introduction: Description of the task scenario, prompting the large language model to call on the relevant prior knowledge acquired during the pre-training process, and explaining to the large language model what task to perform.

[0038] ② Specific examples: Cover successful examples of various situations in actual task scenarios, and fine-tune and correct the relevant prior knowledge of the large language model in specific scenarios.

[0039] ③ Component semantics: component semantic category information parsed in the previous step.

[0040] Furthermore, the above-mentioned prompt word template is filled with the component semantic category label obtained in step S1 to form a specific prompt word corresponding to the current single-view image, which is input into the large language model. The large language model will prompt the missing parts that have not been successfully segmented in the current segmentation result and the correct assembly order of these parts according to the summarized assembly component selection rules, thereby well solving the problem of missing segmented parts caused by partial occlusion in the single-view image. The large language model used in this embodiment is GPT-4, which currently shows excellent performance in various tasks. It is used in the program by calling the API interface, and the parameter settings use the default configuration.

[0041] Step S3: Combined with the component selection rules generated by the large language model, the segmented components are matched one by one with the components in the set of components to be selected, and the components in the set of components to be selected that have the highest similarity with the segmentation results are selected as the assembly components. This step specifically includes three parts: 3.1) Project the three-dimensional parts in the set of parts to be selected to form a two-dimensional image set of the parts to be selected.

[0042] 3.2) Calculate the cosine similarity between the component mask image in the segmentation result and the image in the two-dimensional image set of the component to be selected pair by pair. The combination with the highest similarity is regarded as a successful match, and the successfully matched components will be removed from the set of components to be selected.

[0043] In order to solve the problem of inconsistent rotation angles between the segmented parts and the corresponding parts in the set of parts to be selected, this embodiment rotates the images in the two-dimensional image set of the parts to be selected by a fixed angle of 8, calculates the cosine similarity between them and the mask image of the segmented parts, and takes the average value as the final result. The specific calculation formula is as follows:

[0044] in, Indicates that the image Rotate clockwise .

[0045] 3.3) Based on the missing prompt information of the segmentation results and the summarized assembly component selection rules of the large language model in step 2, the component selection results are corrected and completed.

[0046] Step S4: Encode the component semantic category information parsed from the single-view image and the assembly order information generated by the large language model into the component assembly network based on the Transformer architecture, assemble the components selected in the previous step, and predict their correct assembly pose. The goal of component assembly is to Predict the corresponding assembly pose ,in, is the 3D component point cloud, and N is the total number of actual components in the assembly. (R, t) is the 6-DOF pose in SE(3) space, which represents the rotation quaternion and translation transformation respectively. Finally, all components will be assembled into a complete assembly according to their predicted poses. In view of the component assembly problem, this embodiment proposes a Transformer-based component assembly framework, which takes a set of 3D point clouds of components as input and outputs the jointly estimated component pose in SE(3) space. Figure 3 As shown, the component assembly framework specifically includes three components: 1) PointNet extracts the point cloud features of the components, maps the 3D point cloud of the components to a feature space of the same dimension, and learns a unified vector representation.

[0047] 2) The encoder is responsible for encoding the components and using the self-attention module to infer the geometric and spatial position relationships between modeling components.

[0048] 3) The predictor consists of a 3-layer feedforward network and performs pose prediction based on the component features output by the encoder.

[0049] At the same time, by fusing the component semantic category coding and the component assembly order coding, semantic category information and assembly order information are added to the component point cloud features, solving the semantic ambiguity problem between components.

[0050] The following is a specific application example: This application example takes the chair assembly in the 3D assembly task as an example. The control workstation is installed with Ubuntu 20.04 system, equipped with Intel Core i9-10920x, 3.60GHz CPU, and NVIDIA RTX 3090Ti*2 GPU. Based on these devices, the specific implementation method is introduced: Use 1700 sheets as attached Figure 1 The finished assembly pictures in the IKEA furniture assembly instructions shown in the figure are used to construct a single-view image dataset, and the training set and test set are divided into a ratio of 7:3. About 50 typical parts of various types in the Partnet chair dataset are selected to construct a set of candidate parts.

[0051] After the data set is constructed, the following process is executed, namely: S1: Perform component-level segmentation on the chair in the single-view image and obtain the corresponding semantic category labels. First, the Canny operator is used to detect and identify the edge lines of the components in the image to assist in the subsequent component segmentation. Based on the edge detection results, this embodiment uses the Grounded-SAM model to perform component-level segmentation and semantic category detection on the single-view image. The Grounded-SAM model is fine-tuned on the chair component segmentation task using the single-view image training set, and a prompt containing all chair component categories is set as the text input of the Grounded-SAM model. Finally, the Grounded-SAM model can segment out all the components contained in the chair in the image and obtain their corresponding semantic category labels.

[0052] S2: Use the assembly prior knowledge in the large language model to guide the component selection stage of three-dimensional assembly. This embodiment accesses the GPT-4 large language model in the program by calling the API interface, and the parameters use the default configuration. Fill the acquired component semantic category labels into the pre-designed prompt word template and input it to the large language model. The large language model will summarize the component selection rules for chair assembly, prompt the missing parts that have not been successfully segmented in the current segmentation results, and the correct assembly order of all parts.

[0053] S3: Combined with the component selection rules, the segmented components are matched one by one with the components to be selected. The three-dimensional components in the set of components to be selected are projected to form a two-dimensional image set of the components to be selected. The images in the two-dimensional image set of the components to be selected are fixedly rotated by 8 angles, and their cosine similarities with the mask images of the segmented components are calculated respectively, and the average value is taken as the final result. According to the calculation results, the combination with the highest similarity is regarded as a successful match, and the successfully matched components will be removed from the set of components to be selected. According to the missing prompt information of the segmentation results of the large language model and the summarized assembly component selection rules, the component selection results are corrected and supplemented to obtain the final component selection results.

[0054] S4: Transformer-based component assembly that integrates component semantic category information and assembly order information. The Transformer-based component assembly framework proposed in the present invention integrates component semantic category information and component assembly order information to achieve correct component assembly posture prediction.

[0055] Those skilled in the art should understand that each module or each step of the above-mentioned present application can be implemented with a general portable device, and optionally, they can be implemented with a program code executable by a computing device. The present application is not limited to any specific combination of hardware and software. The above is only a preferred embodiment of the present application and is not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present application should be included in the protection scope of the present application.

Claims

1. A three-dimensional assembly method based on single-view images and guided by a large language model, characterized by: The steps include: Step S1, using the trained semantic segmentation model and edge detection algorithm to perform component-level segmentation on the assembly in the single-view image and obtain the corresponding semantic category labels; Step S2, using the assembly prior knowledge contained in the large language model to guide the component selection stage of the three-dimensional assembly, and forming certain component selection rules; Step S3: Combine the component selection rules generated by the large language model, match the segmented components with the components in the set of components to be selected one by one, and select the component in the set of components to be selected that has the highest similarity with the segmentation result as the assembly component; Step S4: Encode the component semantic category information parsed from the single-view image and the assembly order information generated by the large language model into a component assembly network based on the Transformer architecture, predict its correct assembly posture, and assemble the selected components.

2. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 1, characterized in that: The specific process of step S1 is as follows: Step S1.1, construct a data set containing assembly finished product images; Step S1.2, selecting the finished assembly image in the data set as the single-view image for guiding the three-dimensional assembly, using the Canny operator to detect and identify the edge lines of the assembly parts, and obtaining the optimal edge contour of the single-view image; Step S1.3, performing component-level segmentation and semantic category detection on the single-view image, identifying objects of corresponding categories in the image based on the text description, and segmenting the masks of the corresponding objects; Step S1.4: Segment all the components contained in the assembly in the image, and obtain the semantic category label of each component based on the pre-input text containing the categories of all the components to be inspected.

3. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 2, characterized in that: The specific process of using the Canny operator to detect and identify the edge lines of assembly parts and obtain the optimal edge contour of the single-view image is as follows: Step 1.2.1, use Gaussian filter operator to filter out the high-frequency noise that may exist in the image; Step 1.2.2: Calculate the gradient of any pixel in the image and angle ; Step 1.2.3, suppress the non-maximum points of the gradient image, remove non-edge pixels, and find the points with local maximum values ​​in the gradient image as edge points; Step 1.2.4: Use a dual threshold algorithm to detect and connect edges. Set the size of the two thresholds to 1:

2. Pixels smaller than the low threshold are determined as false edges, while pixels larger than the high threshold are determined as strong edges.

4. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 1, characterized in that: The specific process of step S2 is as follows: Step S2.1, design a fixed prompt word template to activate the assembly prior knowledge contained in the large language model, and parse the component semantic category information according to the input task scenario description; Step S2.2, using the component semantic category labels obtained in step S2.1 to fill in the prompt word template, forming a specific prompt word corresponding to the current single-view image, and inputting it into the large language model; Step S2.3: The large language model obtains the prompt information of the missing parts that have not been successfully segmented in the current segmentation result and the correct assembly order of all parts according to the summarized assembly part selection rules.

5. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 1, characterized in that: The specific process of step S3 is as follows: Step S3.1, projecting the three-dimensional parts in the set of parts to be selected to form a two-dimensional image set of the parts to be selected; Step S3.2, calculating the cosine similarity between the component mask image in the segmentation result and the image in the two-dimensional image set of the component to be selected one by one, the combination with the highest cosine similarity is regarded as a successful match, and the successfully matched component is removed from the set of components to be selected; Step S3.3: According to the prompt information given by the large language model in step 2 and the summarized assembly component selection rules, the component selection results are corrected and completed.

6. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 5, characterized in that: In step 3.2, the images in the two-dimensional image set of the part to be selected are rotated at a fixed angle of 8, and the cosine similarity between them and the mask image of the part in the segmentation result is calculated respectively, and the average value is taken as the final result.

7. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 1, characterized in that: The specific process of step S4 is as follows: Step S4.1, building a component assembly network based on the Transformer architecture; Step S4.2: predict the correct assembly posture of the component based on the component semantic category coding and the component assembly order coding.

8. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 7, characterized in that: The component assembly network based on the Transformer architecture consists of three parts: PointNet module, encoder and predictor.

9. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 8, characterized in that: Predicting the correct assembly pose of a component includes the following steps: Step 1: Use the PointNet module to extract the 3D point cloud features of the components, map the 3D point cloud features of the components to the feature space of the same dimension, and learn a unified vector representation; Step 2: Encode the components using the encoder, and use the self-attention module to infer the geometric and spatial position relationships between the modeled components; Step 3: The predictor consists of a 3-layer feedforward network and performs pose prediction based on the component features output by the encoder.

10. The three-dimensional assembly method based on single-view image and large language model guidance according to claim 9, characterized in that: By fusing the component semantic category coding and the component assembly order coding, semantic category information and assembly order information are added to the component point cloud features.

Images