A system and method for constructing and automatically annotating three-dimensional scene generation

Abstract

The present invention discloses a system and method for constructing and automatically annotating a three-dimensional scene generation, and relates to the technical field. A system for constructing and automatically annotating a three-dimensional scene generation includes an item classification module, an item retrieval module, a scene integration module, a key distinguishing point recognition module, and a scene annotation integration module; a method for constructing and automatically annotating a three-dimensional scene generation includes: S100, scene separation; S200, item classification; S300, item retrieval; S400, scene integration; S500, key distinguishing point recognition; S600, scene annotation integration. The present invention expands the scale of existing three-dimensional data sets, provides a wider range of object categories and richer item variants, constructs an automated annotation process, can capture the subtle differences between target objects and similar distractors, and provides more sophisticated text annotations.

Description

Technical Field

[0001] The present invention relates to the technical fields of computer vision, artificial intelligence and three-dimensional point cloud data processing, and in particular to a system and method for constructing and automatically annotating a three-dimensional scene generation system. Background Art

[0002] In recent years, building machine perception systems that can understand the three-dimensional world has become a hot research topic. Existing research has mainly focused on learning three-dimensional representations through point clouds, and has made significant progress in object classification. With the success of large language models, the focus of research has gradually expanded beyond traditional object classification tasks, such as PointLLM and 3D-LLM, which aim to align the latent representation of text descriptions with three-dimensional point clouds, enabling machine perception systems to understand and interact with the physical world more efficiently through text-based instructions. If the objects in the scene can be accurately identified based on the text description, it will greatly enhance the intelligence of robotic applications, enabling them to perform tasks through text instructions or promote the optimization of human-machine collaboration.

[0003] However, the realization of the above vision of 3D scene understanding is currently challenged by the lack of a comprehensive 3D benchmark dataset. The existing scene-level benchmark datasets are mainly based on the ScanNet dataset, which is a 3D dataset containing scan data of more than 1,600 real scenes. It mainly collects indoor scene information, including RGB, depth, and 3D point cloud ply data, and annotates semantic labels and instance labels. Subsequently, improvements were made on the basis of the ScanNet dataset to generate the ScanRef dataset, Multi3DRef dataset, and ScanQA dataset, which improved and enriched the text annotations of ScanNet to facilitate the evaluation and development of 3D models. Despite this, the existing 3D benchmark datasets still cannot fully meet the needs of the development of 3D models, as shown in the following:

[0004] 1. The dataset size is limited. Compared with the large-scale text-image pair dataset required for training large 2D vision-text models, the 3D scene dataset is relatively small because it mainly comes from scans of real-world scenes.

[0005] 2. The annotations of the dataset are not fine enough. The fine granularity of the annotations is crucial in objectively evaluating the quality of 3D models and promoting their improvement. The localization text itself may be ambiguous and refer to objects in multiple scenes, which causes the objects extracted by the model in the point cloud scene to be inconsistent with the labels;

[0006] 3. Lack of flexible and customized challenge levels. Current 3D scene data is mainly collected from a limited number of scene layouts and object combinations and placement settings. Although these datasets may contain some challenging scenes, their non-customizability may lead to model overfitting. Having diverse scene data and data with adjustable difficulty will help to more comprehensively evaluate the performance of the model.

[0007] Based on the above limitations and the urgent need for a more comprehensive three-dimensional scene dataset, technicians in this field are committed to developing a system and method for constructing and automatically annotating three-dimensional scene generation. Summary of the invention

[0008] In view of the above-mentioned defects of the prior art, the technical problem to be solved by the present invention is how to expand the scale of the data set, refine the data set annotation and provide a flexible scene construction method to construct more diverse scenes and build an automated and refined annotation process.

[0009] This application proposes to build diversified scenes by integrating item-level point cloud data and scene-level point cloud data. Item-level point cloud data include ModelNet dataset, ShapeNet dataset, ScanObjectNN dataset and OmniObject dataset; scene-level point cloud data include ScanNet dataset. Compared with scene-level point cloud data, item-level point cloud data can provide a wider range of object categories and richer object variants, especially in challenging scenes that need to include subtle object distinctions. In addition to scene construction, this application further integrates large language models and two-dimensional visual language models to build an automated and refined annotation process. Specifically, the Large Language Model (LLM) is instructed to guide the Vision Language Model (VLM) to pay attention to the differences between the target object and the distractor, and summarize their differences in multiple aspects such as color, shape, and position as dataset annotations.

[0010] In one embodiment of the present invention, a three-dimensional scene generation system for construction and automatic annotation is provided, comprising:

[0011] The item classification module organizes the item-level point cloud data and classifies them according to the physical characteristics using the item classification method to obtain a candidate item pool;

[0012] The item retrieval module searches the candidate item pool according to the physical characteristics of the target object and the retrieval specifications to obtain similar items;

[0013] The scene integration module adjusts the size of similar objects to the target object, displaces them according to the predefined spatial position, synthesizes the integrated object point cloud with the target object, and puts it into the background segmented by the scene-level point cloud data according to the label to obtain a three-dimensional scene;

[0014] The key distinguishing point recognition module, which resides in the large language model and the visual language model, uses the distinguishing point recognition method to capture the difference between the target object and similar items, and obtains a text description of the difference between the target object and similar items;

[0015] The scene annotation integration module stores the large language model, summarizes the difference text descriptions between the target object and similar objects, inputs the large language model for summary, extracts the key difference texts, and obtains the annotations of the 3D scene;

[0016] The item classification module, the item retrieval module, the scene integration module, the key distinguishing point recognition module and the scene annotation integration module are sequentially connected in communication;

[0017] The scene is separated into background and target object according to the segmentation labels of the scene-level point cloud. The item classification module organizes the item-level point cloud data and classifies it according to the physical features using the item classification method to obtain a candidate item pool. The item retrieval module retrieves similar items from the candidate item pool according to the physical features of the target object and the retrieval specifications. The scene integration module displaces and splices similar items according to pre-defined spatial positions, synthesizes and integrates the item point cloud with the target object, and puts it into the background segmented by the scene-level point cloud data according to the labels to obtain a three-dimensional scene. The key distinguishing point recognition module resides in the large language model and the visual language model, and uses the distinguishing point recognition method to capture the difference between the target object and similar items, and obtains a text description of the difference between the target object and similar items. The scene annotation integration module summarizes the text description of the difference between the target object and similar items, inputs the large language model for summary, and extracts the key difference text to obtain the annotation of the three-dimensional scene.

[0018] Optionally, in the three-dimensional scene generation system for construction and automatic annotation in the above embodiment, the physical features include the category of the object, the color of the object and the shape of the object.

[0019] Optionally, in the constructed and automatically labeled three-dimensional scene generation system in any of the above embodiments, the object classification method includes first organizing the item-level point cloud data according to the category of the object, then using the standard RGB triplet to represent the color of the object, quantifying the color similarity through the L2 norm, and finally limiting the shape of the object to a standard shape.

[0020] Furthermore, in the three-dimensional scene generation system for construction and automatic annotation in the above embodiment, the standard shapes include a cube, an L-shape, and a sphere.

[0021] Optionally, in the three-dimensional scene generation system for construction and automatic annotation in any of the above embodiments, the classification of objects includes common indoor objects.

[0022] Furthermore, in the three-dimensional scene generation system for construction and automatic annotation in the above embodiment, common indoor objects include tables, sofas, chairs, beds, cabinets, doors, and windows.

[0023] Optionally, in the three-dimensional scene generation system for construction and automatic annotation in any of the above embodiments, the search criteria include satisfying at least one physical feature being the same and the combined physical features being different.

[0024] Optionally, in the three-dimensional scene generation system for construction and automatic annotation in any of the above embodiments, the predefined spatial positions include up, down, left, right, front, and back.

[0025] Optionally, in the three-dimensional scene generation system for construction and automatic annotation in any of the above embodiments, the distinguishing point recognition method includes:

[0026] Multi-view recognition, rendering the target object and similar items from multiple angles to avoid missing key differences and using visual language models to identify differences;

[0027] Iterative distinguishing point capture prompts the large language model to continuously ask the visual language model for new potential distinguishing point dimensions, prompting the visual language model to capture more differences. Iterative question-and-answering is performed between the visual language model and the large language model to enhance the comprehensiveness of difference recognition.

[0028] Preferably, in the three-dimensional scene generation system for construction and automatic annotation in the above embodiment, the large language model uses GPT.

[0029] Preferably, in the three-dimensional scene generation system for construction and automatic annotation in the above embodiment, the visual language model uses LLaVA.

[0030] Furthermore, in the three-dimensional scene generation system for construction and automatic annotation in the above embodiment, iterative question and answering is performed for at least four rounds.

[0031] Based on any of the above embodiments, in another embodiment of the present invention, a method for constructing and automatically annotating a three-dimensional scene generation is provided, comprising the following steps:

[0032] S100, scene separation, separating the scene into a target object and a background according to the segmentation labels of the scene-level point cloud;

[0033] S200, item classification, sorting the item-level point cloud data, and classifying them using an item classification method according to physical features to obtain a candidate item pool;

[0034] S300, item retrieval, searching in a candidate item pool according to the physical characteristics of the target object and the retrieval specification to obtain similar items;

[0035] S400, scene integration, adjusting the size of similar objects to the target object, displacing them according to a predefined spatial position, synthesizing and integrating the object point cloud with the target object, and placing it in the background segmented by the scene-level point cloud data according to the label, to obtain a three-dimensional scene;

[0036] S500, key distinguishing point recognition, using a distinguishing point recognition method to capture the difference between the target object and similar items, and obtain a text description of the difference between the target object and the similar items;

[0037] S600: integrate scene annotations, summarize difference text descriptions, input a large language model for summarization, extract key difference texts, and obtain annotations for the three-dimensional scene.

[0038] Optionally, in the three-dimensional scene generation method of construction and automatic annotation in the above embodiment, the physical features include the category of the object, the color of the object and the shape of the object.

[0039] Furthermore, in the method for constructing and automatically annotating a three-dimensional scene in the above embodiment, step S200 includes:

[0040] S210, sorting category information, sorting according to the category of items;

[0041] S220, quantifying color similarity, using standard RGB triplet to represent the color of the object, and quantifying color similarity through L2 norm;

[0042] S230, sorting shape information, limiting the shape range of the object to a standard shape;

[0043] S240: Obtain a candidate pool of similar items.

[0044] Furthermore, in the three-dimensional scene generation system for construction and automatic annotation in the above embodiment, the standard shapes include a cube, an L-shape, and a sphere.

[0045] Optionally, in the method for constructing and automatically annotating a three-dimensional scene in the above embodiment, step S300 includes:

[0046] S310, determining the physical characteristics of the target object;

[0047] S320: Search in a candidate item pool according to the physical features of the target object and the search criteria to obtain similar items.

[0048] Furthermore, in the method for constructing and automatically annotating a three-dimensional scene in the above embodiment, the search criteria include satisfying at least one physical feature being the same and the physical features of the combination being different.

[0049] Furthermore, in the three-dimensional scene generation method of construction and automatic annotation in the above embodiment, the predefined spatial positions include up, down, left, right, front, and back.

[0050] Optionally, in the method for generating a three-dimensional scene by construction and automatic annotation in the above embodiment, step S400 includes:

[0051] S410, adjusting the size of similar objects, adjusting the size of similar objects to the size of the target object;

[0052] S420, synthesizing and integrating the object point cloud, displacing the resized similar objects according to the predefined spatial position, and synthesizing and integrating the object point cloud with the target object;

[0053] S430, background integration, putting the integrated object point cloud into the background segmented by the scene-level point cloud data according to the label to obtain a three-dimensional scene.

[0054] Optionally, in the method for generating a three-dimensional scene by construction and automatic annotation in any of the above embodiments, step S500 includes:

[0055] S510, multi-view recognition, rendering the target object and similar items from multiple angles to avoid missing key differences and using visual language models to identify differences;

[0056] S520, iteratively capturing distinguishing points, prompting the large language model to continuously ask the visual language model for new potential distinguishing point dimensions, prompting the visual language model to capture more differences, and iteratively asking and answering questions between the visual language model and the large language model to enhance the comprehensiveness of difference recognition;

[0057] S530: Obtain a text description of the difference between the target object and the details of similar items.

[0058] Furthermore, in the method for constructing and automatically annotating a three-dimensional scene in the above embodiment, the iterative question-answering is performed for at least four rounds.

[0059] Furthermore, in the method for constructing and automatically annotating a three-dimensional scene in the above embodiment, step S600 includes:

[0060] S610, text description summary, summarizing the difference text descriptions;

[0061] S620, summarizing the text descriptions, inputting the summarized difference text descriptions into the large language model for summarization;

[0062] S630: Obtain annotations for the three-dimensional scene, extract text descriptions of physical features with the greatest differences, and obtain annotations for the three-dimensional scene.

[0063] Preferably, in the three-dimensional scene generation method of construction and automatic annotation in the above embodiment, the large language model uses GPT.

[0064] Preferably, in the method for constructing and automatically annotating a three-dimensional scene in the above embodiment, the visual language model uses LLaVA.

[0065] The present invention builds more diverse scenes by integrating item-level point cloud data and scene-level point cloud data, expands the scale of existing three-dimensional datasets, and provides a wider range of object categories and richer object variants; by integrating large language models and two-dimensional visual language models, an automated annotation process is constructed that can capture subtle differences between target objects and similar distractors, and provide more detailed text annotations.

[0066] The concept, specific structure and technical effects of the present invention will be further described below in conjunction with the accompanying drawings to fully understand the purpose, characteristics and effects of the present invention. BRIEF DESCRIPTION OF THE DRAWINGS

[0067] Figure 1 is a schematic diagram illustrating a structure of a three-dimensional scene generation for construction and automatic annotation according to an exemplary embodiment;

[0068] Figure 2 is a flow chart illustrating a method for constructing and automatically annotating a three-dimensional scene generation according to an exemplary embodiment;

[0069] Figure 3 is a point cloud diagram of a three-dimensional scene generated by a method for generating a three-dimensional scene by construction and automatic annotation according to an exemplary embodiment;

[0070] Figure 4 is a flow chart illustrating iterative differentiation point capture of a method for constructing and automatically annotating a three-dimensional scene generation according to an exemplary embodiment. DETAILED DESCRIPTION

[0071] The following describes several preferred embodiments of the present invention with reference to the drawings in the specification, so that the technical content is clearer and easier to understand. The present invention can be embodied in many different forms of embodiments, and the protection scope of the present invention is not limited to the embodiments mentioned in the text.

[0072] In the drawings, components with the same structure are indicated by the same numerical reference numerals, and components with similar structures or functions are indicated by similar numerical reference numerals. The size and thickness of each component shown in the drawings are arbitrarily shown, and the present invention does not limit the size and thickness of each component. In order to make the illustration clearer, the thickness of the components is schematically exaggerated in some places in the drawings.

[0073] This application proposes to build diversified scenes by integrating item-level point cloud data and scene-level point cloud data. Item-level point cloud data include ModelNet dataset, ShapeNet dataset, ScanObjectNN dataset and OmniObject dataset; scene-level point cloud data include ScanNet. Compared with scene-level point cloud data, item-level point cloud data can provide a wider range of object categories and richer object variants, especially in challenging scenes that need to include subtle object distinctions. In addition to scene construction, this application further integrates large language models and two-dimensional visual language models to build an automated and refined annotation process. Specifically, the Large Language Model (LLM) is instructed to guide the Vision Language Model (VLM) to focus on the differences between the target object and the distractor, and summarize their differences in multiple aspects such as color, shape, and position as dataset annotations.

[0074] This application designs a 3D scene generation system for construction and automatic annotation, such as Figure 1 As shown, including:

[0075] The item classification module organizes the item-level point cloud data and classifies them according to the physical features using the item classification method to obtain a candidate item pool. The physical features include the category of the item, the color of the item and the shape of the item. The item classification method includes first organizing the item-level point cloud data according to the category of the item, then using the standard RGB triplet to represent the color of the item, quantifying the color similarity through the L2 norm, and finally limiting the shape of the item to standard shapes. The standard shapes include cube, L shape and sphere. The classification of items includes common indoor items, including tables, sofas, chairs, beds, cabinets, doors and windows.

[0076] The item retrieval module searches in the candidate item pool according to the physical characteristics of the target object and the retrieval specifications to obtain similar items. The retrieval specifications include satisfying at least one physical characteristic that is the same and the physical characteristics of the combination are different.

[0077] The scene integration module adjusts the size of similar objects to the target object, displaces them according to the predefined spatial position, synthesizes the integrated object point cloud with the target object, and puts it into the background segmented by the scene-level point cloud data according to the label to obtain a three-dimensional scene. The predefined spatial positions include up, down, left, right, front, and back.

[0078] The key distinguishing point recognition module has a large language model and a visual language model. The large language model uses GPT, and the visual language model uses LLaVA. The distinguishing point recognition method is used to capture the difference between the target object and similar items, and obtain a text description of the difference between the target object and similar items. The distinguishing point recognition method includes:

[0079] Multi-view recognition, rendering the target object and similar items from multiple angles to avoid missing key differences, using the visual language model LLaVA to identify differences;

[0080] Iterative distinguishing point capture prompts the large language model GPT to continuously ask the visual language model for new potential distinguishing point dimensions, prompting the visual language model LLaVA to capture more distinctions. Iterative question and answer is performed between the visual language model LLaVA and the large language model GPT. The iterative question and answer is performed for four rounds to enhance the comprehensiveness of difference recognition.

[0081] The scene annotation integration module resides in a large language model. The large language model uses GPT to summarize the difference text descriptions between the target object and similar objects, inputs the large language model GPT for summary, extracts the key difference texts, and obtains the annotations of the 3D scene.

[0082] The item classification module, the item retrieval module, the scene integration module, the key distinguishing point recognition module and the scene annotation integration module are sequentially connected in communication;

[0083] The scene is separated into background and target object according to the segmentation labels of the scene-level point cloud. The item classification module organizes the item-level point cloud data and classifies it according to the physical features using the item classification method to obtain a candidate item pool. The item retrieval module retrieves similar items from the candidate item pool according to the physical features of the target object and the retrieval specifications. The scene integration module displaces and splices similar items according to pre-defined spatial positions, synthesizes and integrates the item point cloud with the target object, and puts it into the background segmented by the scene-level point cloud data according to the labels to obtain a three-dimensional scene. The key distinguishing point recognition module resides in the large language model GPT and the visual language model LLaVA, and uses the distinguishing point recognition method to capture the difference between the target object and similar items, and obtains the difference text description of the details of the target object and similar items. The scene annotation integration module resides in the large language model GPT, summarizes the difference text description between the target object and similar items, inputs the large language model GPT for summary, and extracts the key difference text to obtain the annotation of the three-dimensional scene.

[0084] Based on the above embodiments, the present application provides a method for constructing and automatically annotating a three-dimensional scene generation method, such as Figure 2 As shown, the following steps are included:

[0085] S100, scene separation, separating the scene into a target object and a background according to the segmentation labels of the scene-level point cloud.

[0086] S200, object classification, sorting the object-level point cloud data, and classifying them using an object classification method according to physical features to obtain a candidate object pool. The physical features include the category of the object, the color of the object, and the shape of the object; specifically, including:

[0087] S210, sorting category information, sorting according to the category of items;

[0088] S220, quantifying color similarity, using standard RGB triplet to represent color, and quantifying color similarity through L2 norm;

[0089] S230, sorting shape information, limiting the shape range of the object to standard shapes, which include cube, L-shape, and sphere;

[0090] S240: Obtain a candidate pool of similar items.

[0091] S300, item retrieval, searching in a candidate item pool according to the physical characteristics of the target object and the retrieval specification to obtain similar items; specifically including:

[0092] S310, determining the physical characteristics of the target object;

[0093] S320. Search in a candidate item pool according to the physical features of the target object and the search criteria to obtain similar items. The search criteria include satisfying at least one physical feature that is the same and a combination of physical features that are different.

[0094] S400, scene integration, adjust the size of similar objects to the target object, displace them according to the predefined spatial position, synthesize and integrate the object point cloud with the target object, and put it into the background segmented by the scene-level point cloud data according to the label to obtain a three-dimensional scene. The predefined spatial positions include up, down, left, right, front, and back; specifically include:

[0095] S410, adjusting the size of similar objects, adjusting the size of similar objects to the size of the target object;

[0096] S420, synthesizing and integrating the object point cloud, displacing the resized similar objects according to the predefined spatial position, and synthesizing and integrating the object point cloud with the target object;

[0097] S430, background integration, putting the integrated object point cloud into the background segmented by the scene-level point cloud data according to the label to obtain a three-dimensional scene.

[0098] S500, key distinguishing point recognition, using a distinguishing point recognition method to capture the difference between the target object and similar items, and obtain a text description of the difference between the target object and the similar items; specifically including:

[0099] S510, multi-view recognition, renders the target object and similar items from multiple angles to avoid missing key differences, and uses a visual language model to identify the differences. The visual language model uses LLaVA;

[0100] S520, iterative distinguishing point capture, prompting the large language model to continuously ask the visual language model LLaVA for new potential distinguishing point dimensions, the large language model uses GPT to prompt the visual language model LLaVA to capture more differences, and the visual language model LLaVA and the large language model GPT perform iterative question and answer, the iterative question and answer is performed for four rounds, and the comprehensiveness of difference recognition is enhanced;

[0101] S530: Obtain a text description of the difference between the target object and the details of similar items.

[0102] S600: Scene annotation integration, summarizing difference text descriptions, inputting into the large language model GPT for summarization, extracting key difference texts, and obtaining annotations for the 3D scene; specifically including:

[0103] S610, text description summary, summarizing the difference text descriptions;

[0104] S620, summarizing the text descriptions, inputting the summarized difference text descriptions into the large language model GPT for summarization;

[0105] S630: Obtain annotations for the three-dimensional scene, extract text descriptions of physical features with the greatest differences, and obtain annotations for the three-dimensional scene.

[0106] This application has experimentally verified the effect of the above embodiment: the target object is a chair, the similar object is a chair with a different shape from the target object, and the scene is a home scene. The three-dimensional scene generated by the above embodiment is shown in Figure 3. The three-dimensional scene in the figure is visualized as a scaling rendering result of a point cloud containing only point information at a certain viewing angle, which may be blurred or incomplete. The key distinguishing point recognition module obtains the key distinction between similar objects and target objects as different designs and heights of the chair backs. The final summary of the positioning text about the target object is: a chair surrounded by four chairs, characterized by a square back.

[0107] In the above experiment, the process of iteratively capturing distinguishing points between the target object and similar items is as follows: Figure 4As shown, first the experimenter makes an initial inquiry "Describe the difference between the two chairs in the picture", and the visual language model LLaVA outputs a description of the difference in physical features between the target object chair and the similar object chair, which are "a chair with an oval back" and "a chair with an upright back" respectively; the results of this round of questions and answers are input into the large language model GPT, and the large language model GPT continues to ask the question "Describe the difference in their heights", and the visual language model LLaVA outputs a text description of the height difference, which are "relatively taller" and "relatively shorter" respectively; then the results of this round of questions and answers are input into the large language model GPT, and a new question is asked "What is the difference in the backs of the chairs", and the visual language model LLaVA outputs descriptions of the differences in the backs of the chairs, which are: "oval" and "upright" respectively; the results of this round of questions and answers are continued to be input into the large language model GPT to ask the question "What is the difference in the legs of the chairs", and the visual language model LLaVA outputs descriptions of the differences in the legs of the chairs, which are: "slender" and "no legs" respectively.

[0108] The preferred specific embodiments of the present invention are described in detail above. It should be understood that a person skilled in the art can make many modifications and changes based on the concept of the present invention without creative work. Therefore, any technical solution that can be obtained by a person skilled in the art through logical analysis, reasoning or limited experiments based on the concept of the present invention on the basis of the prior art should be within the scope of protection determined by the claims.

Claims

1. A three-dimensional scene generation system for construction and automatic annotation, characterized in that: include: The item classification module organizes the item-level point cloud data and classifies them according to the physical characteristics using the item classification method to obtain a candidate item pool; An item retrieval module searches the candidate item pool according to the physical characteristics of the target object and the retrieval specification to obtain similar items; The scene integration module adjusts the size of the similar objects to the target object, displaces them according to the predefined spatial position, synthesizes and integrates the object point cloud with the target object, and puts it into the background segmented by the scene-level point cloud data according to the label to obtain a three-dimensional scene; A key distinguishing point recognition module, in which a large language model and a visual language model are resident, uses a distinguishing point recognition method to capture the difference between the target object and the similar items, and obtains a text description of the difference between the target object and the similar items; The distinguishing point identification method comprises: Multi-view recognition, rendering the target object and the similar items from multiple angles to avoid missing key differences, and using the visual language model to identify the differences; Iterative distinguishing point capture, prompting the large language model to continuously ask the visual language model for new potential distinguishing point dimensions, prompting the visual language model to capture more differences, and iterative question-answering between the visual language model and the large language model to enhance the comprehensiveness of difference recognition; A scene annotation integration module resides in a large language model, summarizes the difference text descriptions between the target object and the similar items, inputs the large language model for summary, extracts key difference texts, and obtains the annotation of the three-dimensional scene; The object classification module, the object retrieval module, the scene integration module, the key distinguishing point identification module and the scene annotation integration module are sequentially communicatively connected.

2. The system for constructing and automatically annotating a three-dimensional scene according to claim 1, characterized in that: The physical characteristics include the category of the object, the color of the object and the shape of the object.

3. The three-dimensional scene generation system for construction and automatic annotation as claimed in claim 2, characterized in that: The object classification method includes firstly organizing the object-level point cloud data according to the category of the object, then using a standard RGB triplet to represent the color of the object, quantifying the color similarity through the L2 norm, and finally restricting the shape of the object to a standard shape.

4. The system for constructing and automatically annotating a three-dimensional scene according to claim 1, wherein: The search criteria include satisfying that at least one of the physical features is the same and the physical features in combination are different.

5. A method for generating a three-dimensional scene by construction and automatic annotation, using the three-dimensional scene generation system for construction and automatic annotation as claimed in any one of claims 1 to 4, characterized in that: The steps include: S100, scene separation, separating the scene into a target object and a background according to the segmentation labels of the scene-level point cloud; S200, object classification, sorting the object-level point cloud data, and classifying them according to physical features using an object classification method to obtain a candidate object pool; S300, item retrieval, searching in the candidate item pool according to the physical features of the target object and the retrieval specification to obtain similar items; S400, scene integration, adjusting the size of the similar object to the target object, displacing it according to a predefined spatial position, synthesizing and integrating the object point cloud with the target object, and placing it into the background segmented by the scene-level point cloud according to the label, to obtain a three-dimensional scene; S500, identifying key distinguishing points, using a distinguishing point identification method to capture the difference between the target object and the similar items, and obtaining a text description of the difference between the target object and the similar items; S600: integrating scene annotations, summarizing the difference text descriptions, inputting a large language model for summarization, extracting key difference texts, and obtaining annotations for the three-dimensional scene.

6. The method for constructing and automatically annotating a three-dimensional scene as claimed in claim 5, characterized in that: The step S200 includes: S210, sorting category information, sorting according to the category of items; S220, quantifying color similarity, using the L2 norm to quantify the color similarity of the object represented by the standard RGB triplet; S230, sorting shape information, limiting the shape range of the object to a standard shape; S240: Obtain the similar item candidate pool.

7. The method for constructing and automatically annotating a three-dimensional scene as claimed in claim 5 or 6, characterized in that: The step S400 includes: S410, adjusting the size of the similar objects, adjusting the size of the similar objects to the target object; S420, synthesizing and integrating the object point cloud, displacing the resized similar objects according to the predefined spatial position, and synthesizing and integrating the object point cloud with the target object; S430, background integration, putting the integrated object point cloud into the background segmented by the scene-level point cloud data according to the labels to obtain a three-dimensional scene.

8. The method for constructing and automatically annotating a three-dimensional scene as claimed in claim 7, characterized in that: The step S500 includes: S510, multi-view recognition, rendering the target object and the similar items from multiple angles to avoid missing key differences, and using a visual language model to identify the differences; S520, iteratively capturing distinguishing points, prompting the large language model to continuously inquire about new potential distinguishing point dimensions from the visual language model, prompting the visual language model to capture more differences, and iteratively asking and answering questions between the visual language model and the large language model to enhance the comprehensiveness of difference recognition; S530: Obtain a text description of the difference between the target object and the similar item details.

9. The method for constructing and automatically annotating a three-dimensional scene as claimed in claim 8, characterized in that: The step S600 includes: S610, summarizing text descriptions, summarizing the difference text descriptions; S620, summarizing the text descriptions, inputting the summarized difference text descriptions into the large language model for summarization; S630: Obtain annotations for the three-dimensional scene, extract text descriptions of the physical features with the largest differences, and obtain annotations for the three-dimensional scene.

Images