Data augmentation method, device and system based on combination of three-dimensional scene and language data

Abstract

The application provides a data enhancement method, device and system based on three-dimensional scene and language data combination, the method comprising: obtaining 3D scene data and corresponding text annotation data; preprocessing the scene data and the text annotation data respectively to obtain preprocessed 3D-language combined data; sequentially performing multi-modal data enhancement and semantic quality filtering processing on the preprocessed 3D-language combined data to obtain a target 3D-language combined data set. The application integrates various data sources such as 3D point cloud data, RGB-D images, question and answer pairs and dense descriptions, uses data preprocessing, multi-modal data enhancement and semantic quality filtering to realize automatic construction of a high-quality large-scale data set, can improve the data quality of 3D scene understanding and visual question and answer tasks while enhancing the diversity and generalization ability of the data, and provides strong support for 3D visual understanding, robot task planning and other applications.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, in particular to a data enhancement method, device and system based on three-dimensional scene and language data joint. BACKGROUND

[0002] With the rapid development of artificial intelligence technology and deep learning algorithms, computer vision and natural language processing technologies have made great progress in image recognition, scene understanding and language generation. In recent years, visual tasks based on 3D scene data, such as 3D question answering, 3D dense description and robot interaction, have gradually become a research hotspot. Traditional 3D scene understanding methods mainly rely on devices such as structured light, laser scanning and RGB-D cameras to collect 3D point cloud data, and then process them in combination with manually annotated text information. Common datasets such as ScanNet, ScanQA and ScanRefer have provided certain data support for related tasks. However, these datasets still have deficiencies in data size, sample diversity and annotation accuracy, mainly in the following aspects:

[0003] 1. Limited data size: Existing datasets are usually based on manual collection and annotation, with relatively limited sample size, which is difficult to meet the demand of deep learning models for large-scale training data, limiting the generalization ability of models in complex scenes.

[0004] 2. Annotation quality varies: Due to the subjective differences and high cost of manual annotation, the semantic information in question-answer pairs and scene descriptions often has inconsistencies, repetitions, ambiguities and information missing, which may cause the model to learn incorrect semantic mappings during training.

[0005] 3. Insufficient multi-modal fusion: Current data construction methods mostly focus on single-modal data acquisition, while in the joint modeling of 3D scene and language data, how to fully integrate 3D point cloud, image and text information to achieve cross-modal information complementation and enhancement is still a technical problem to be solved.

[0006] 4. Limited data augmentation methods: Existing data expansion methods mainly rely on traditional data augmentation techniques such as rotation, translation and scaling, but these methods have little effect on text data, and cannot effectively improve the data diversity of question-answer pairs or dense descriptions, thus affecting the performance of models in practical applications. SUMMARY

[0007] The purpose of the present application is to provide a three-dimensional scene and language data combined data enhancement method, device and system, which can improve the data quality of 3D scene understanding and visual question answering task, enhance the diversity and generalization ability of data, and provide strong support for 3D visual understanding, robot task planning and other applications.

[0008] In a first aspect, the present application provides a three-dimensional scene and language data combined data enhancement method, which comprises: obtaining 3D scene data and corresponding text annotation data; the 3D scene data comprises scene point cloud data, RGB-D image and camera pose information; the text annotation data comprises question and answer pairs and dense description information related to scene object position, attribute and relationship; the 3D scene data and the text annotation data are preprocessed respectively to obtain preprocessed 3D-language joint data; the preprocessing includes normalization of scene point cloud data, consistency processing of RGB-D image, and data cleaning processing and syntax structure analysis processing of text annotation data; the preprocessed 3D-language joint data is subjected to multi-modal data enhancement respectively to obtain enhanced joint data; the multi-modal data enhancement includes rotation, translation and scale transformation of RGB-D image, and synonym replacement, logical inversion and template generation of question and answer pairs and dense description information; the enhanced joint data is subjected to semantic quality filtering processing to obtain target 3D-language joint data set.

[0009] Further, the above-mentioned step of preprocessing the 3D scene data and the text annotation data respectively comprises: normalizing the scene point cloud data in the 3D scene data, and processing the RGB-D image in resolution and color consistency; cleaning the text annotation data to eliminate repeated, grammatically incorrect and incomplete information samples; using natural language processing technology to segment, remove stop words and analyze the syntax structure of the cleaned text annotation data to obtain text annotation data with consistent and accurate semantics.

[0010] Further, the above-mentioned step of pre-processing the 3D-language joint data and the text annotation data respectively comprises: rotating, translating and scaling the RGB-D image in the pre-processed 3D scene data to realize image expansion; based on the pre-trained language model, using synonym replacement, logical inversion and template generation to expand the question and answer pairs and dense description information; combining the image expanded 3D scene data and the data expanded text annotation data to obtain enhanced joint data.

[0011] Further, the step of performing semantic quality filtering on the enhanced joint data to obtain target 3D-language joint data includes: calculating semantic similarity between generated question and answer text and original question and answer text using a pre-trained BERT model, and filtering text with a semantic similarity less than a first threshold; performing natural language inference using a RoBERTa model to calculate consistency scores between generated dense description text and original dense description text, and filtering description text with a consistency score less than a second threshold to obtain target 3D-language joint data.

[0012] Further, the method further includes: encoding text data, visual cue data, and three-dimensional point cloud data in the target 3D-language joint data set respectively to obtain respective encoding features of the three types of data; fusing the respective encoding features of the three types of data to obtain fused features; and inputting the fused features into a pre-trained large-scale language model for natural language decoding to obtain a target natural language.

[0013] Further, the step of encoding the text data, visual cue data, and three-dimensional point cloud data in the target 3D-language joint data to obtain respective encoding features of the three types of data includes: encoding the text data based on a Transformer architecture to generate high-dimensional representation features; flattening and linearly mapping the visual cue data through a multi-layer perceptron to obtain spatial prior embedding features of the visual cue data; and extracting features of the three-dimensional point cloud data using a Vote2Cap-DETR++ network to obtain deep geometric features of the three-dimensional point cloud data.

[0014] Further, the step of fusing the respective encoding features of the three types of data to obtain fused features includes: preliminarily fusing the spatial prior embedding features corresponding to the visual cue data and the deep geometric features corresponding to the three-dimensional point cloud data using a concatenation method to obtain joint features; deeply fusing the high-dimensional representation features corresponding to the text data and the joint features through a cross-modal attention mechanism to obtain interactive features; performing self-attention processing on the interactive features and the high-dimensional representation features corresponding to the text data, and stacking the features through a residual connection to form the fused features.

[0015] In a second aspect, the application further provides a data augmentation device based on three-dimensional scene and language data combination, the device comprising: a data acquisition module configured to acquire 3D scene data and corresponding text annotation data; the 3D scene data comprises scene point cloud data, RGB-D image and camera pose information; the text annotation data comprises question and answer pairs and dense description information related to scene object position, attribute and relationship; a preprocessing module configured to preprocess the 3D scene data and the text annotation data respectively to obtain preprocessed 3D-language combined data; the preprocessing comprises normalization processing of the scene point cloud data, consistency processing of the RGB-D image, and data cleaning processing and syntax structure analysis processing of the text annotation data; a data augmentation module configured to perform multi-modal data augmentation on the preprocessed 3D-language combined data respectively to obtain augmented combined data; the multi-modal data augmentation comprises rotation, translation and scale transformation of the RGB-D image, and synonym replacement, logical inversion and template generation of the question and answer pairs and the dense description information; and a filtering processing module configured to perform semantic quality filtering processing on the augmented combined data to obtain target 3D-language combined data set.

[0016] In a third aspect, the application further provides a data augmentation system based on three-dimensional scene and language data combination, comprising a processor and a memory, the memory storing computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method of the first aspect.

[0017] In a fourth aspect, the application further provides a computer readable storage medium, the computer readable storage medium storing computer executable instructions, and when the computer executable instructions are called and executed by the processor, the computer executable instructions prompt the processor to implement the method of the first aspect.

[0018] The application provides a data enhancement method, device and system based on three-dimensional scene and language data combination. First, 3D scene data and corresponding text annotation data are acquired. The 3D scene data includes scene point cloud data, RGB-D images and camera pose information. The text annotation data includes question and answer pairs and dense description information related to scene object position, attributes and relationships. Then, normalization processing of the scene point cloud data and consistency processing of the RGB-D images are performed to ensure the consistency of the coordinate range. Data cleaning processing and syntax structure analysis processing of the text annotation data are performed to remove repeated or syntax error samples. Then, rotation, translation and scale transformation of the RGB-D images are performed for image expansion operations, and synonym replacement, logical inversion and template generation of the question and answer pairs and the dense description information are performed for data expansion operations, so as to improve the diversity of the enhanced combined data. Finally, semantic quality filtering processing of the enhanced combined data is performed to filter the enhanced combined data and ensure that the data is logically consistent. Finally, a large-scale high-quality 3D-language combined data set is generated, which can improve the data quality of 3D scene understanding and visual question and answer tasks, enhance the diversity and generalization ability of the data, and provide strong support for 3D visual understanding, robot task planning and other applications. BRIEF DESCRIPTION OF DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the specific embodiments of the present application or the prior art, the drawings needed in the description of the specific embodiments or the prior art will be briefly introduced. Obviously, the drawings in the following description are some embodiments of the present application, and those skilled in the art can also obtain other drawings according to these drawings without creative labor.

[0020] Figure 1 A flowchart of a data enhancement method based on three-dimensional scene and language data combination provided by an embodiment of the present application;

[0021] Figure 2 A flowchart of a 3D-language combined data set construction provided by an embodiment of the present application;

[0022] Figure 3 A flowchart of a text data enhancement process provided by an embodiment of the present application;

[0023] Figure 4 A flowchart of a semantic quality filtering process provided by an embodiment of the present application;

[0024] Figure 5 A flowchart of a multi-modal embedding encoding process provided by an embodiment of the present application;

[0025] Figure 6A structural block diagram of a data enhancement device based on three-dimensional scene and language data combination is provided for an embodiment of the present application.

[0026] Figure 7 A structural schematic diagram of a data enhancement system based on three-dimensional scene and language data combination is provided for an embodiment of the present application. DETAILED DESCRIPTION

[0027] The technical solutions of the present application will be described in detail below with reference to the embodiments. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0028] In the prior art, the question and answer and description of the three-dimensional scene usually rely on relatively scarce data resources, and the traditional data collection method needs a large amount of annotation by professional personnel, which is time-consuming and costly. In order to alleviate the problem of data scarcity, the present embodiment proposes an overall scheme based on multi-modal data generation and enhancement, which realizes the diversification and scale expansion of data sources by combining ScanNet three-dimensional scene data, ScanQA question and answer annotation and ScanRefer dense description. At the same time, the present embodiment uses advanced semantic quality control technology to filter and optimize the generated data, ensuring the high accuracy and semantic consistency of the data.

[0029] Based on this, the present embodiment provides a data enhancement method, device and system based on three-dimensional scene and language data combination, which can improve the data quality of 3D scene understanding and visual question and answer tasks while enhancing the diversity and generalization ability of the data, providing strong support for 3D visual understanding, robot task planning and other applications.

[0030] In order to facilitate the understanding of the present embodiment, first, the construction method of a 3D-language combined data set disclosed by the present embodiment is introduced in detail.

[0031] The data enhancement method based on three-dimensional scene and language data combination provided by the present embodiment aims to construct a high-quality large-scale 3D-language combined data set through deep fusion and intelligent enhancement of multi-modal data. This method not only can expand the scale of the data set and improve the diversity and generalization ability of the data, but also can effectively eliminate low-quality samples and ensure the semantic consistency and accuracy of the data. Figure 1 A flowchart of a data enhancement method based on three-dimensional scene and language data combination provided by the present embodiment, which specifically includes the following steps:

[0032] In step S102, 3D scene data and corresponding text annotation data are obtained; the 3D scene data includes scene point cloud data, RGB-D images, camera pose information and corresponding semantic annotation information; the text annotation data includes question and answer pairs and dense description information related to the positions, attributes and relationships of scene objects;

[0033] Firstly, 3D scene data and text annotation data are obtained from multiple standard data sets (such as ScanNet, ScanQA, ScanRefer, etc.). The 3D scene data includes indoor scene point cloud, RGB-D images and camera pose information; the text annotation data includes question and answer pairs and dense description related to the positions, attributes and relationships of scene objects. Through the integration of multiple data sources, the complement and fusion between different data sources are realized, and sample basis is provided for subsequent data preprocessing and enhancement.

[0034] In specific implementation, scene point cloud data and its corresponding RGB-D images and camera pose information can be extracted from an indoor scene data set, and the scene point cloud data is stored in a standard format (such as.ply format) for subsequent data processing.

[0035] On the other hand, question and answer pairs containing scene object position, attribute and relationship information can be obtained from the question and answer data set; dense description information can be extracted from the description data set, and multiple source text annotation data can be formed by combining the question and answer data.

[0036] In step S104, the 3D scene data and the text annotation data are preprocessed respectively to obtain preprocessed 3D-language joint data; the preprocessing includes normalization processing of the scene point cloud data, consistency processing of the RGB-D images, and data cleaning processing and syntax structure analysis processing of the text annotation data.

[0037] Through normalization processing of each scene point cloud data, the range of each coordinate point is limited within a preset interval; through data cleaning processing of the text annotation data, samples with repetition, syntax error or incomplete semantics are removed, which can ensure the accuracy and integrity of the text data.

[0038] In step S106, the preprocessed 3D-language joint data is subjected to multi-modal data enhancement respectively to obtain enhanced joint data; the multi-modal data enhancement includes rotation, translation and scale transformation of the RGB-D images, and data expansion operations of the three strategies of synonym replacement, logical inversion and template generation of the question and answer pairs and dense description information.

[0039] The specific contents of the above three strategies of synonym replacement, logical inversion and template generation are as follows:

[0040] (1) Synonym replacement: The pre-trained BERT model is used to replace the keywords in the question-answer pair with synonyms. The replacement probability is set to a preset value to maintain semantic stability.

[0041] (2) Logical reversal: Negative sentences in question-answer pairs are generated through predefined rule templates, and the generated reverse questions must be verified for semantic consistency.

[0042] (3) Template generation: Based on the pre-trained T5 model, the descriptive text is converted into structured question-answer pairs and generated using a variety of pre-designed question templates.

[0043] The above three strategies can be used to augment data and improve sample diversity.

[0044] Step S108: Perform semantic quality filtering on the enhanced joint data to obtain the target 3D-language joint dataset.

[0045] In practice, deep learning models can be used to calculate semantic similarity and semantic consistency, and the enhanced joint data can be filtered to obtain high-quality 3D-language joint datasets that meet preset threshold requirements. For example, the BERT model can be used to extract text features from the original and generated question-answer pairs, and cosine similarity can be calculated to filter out question-answer pairs with similarity not lower than a preset threshold; the RoBERTa model can be used to perform natural language reasoning on the generated description and the original description, and text samples with logical inconsistencies can be filtered out based on consistency scores.

[0046] The purpose of the aforementioned multimodal data augmentation and semantic quality filtering is to improve the diversity and generalization ability of data while maintaining the original semantic information of the data, so as to support various application needs such as 3D scene question answering, semantic description, and robot task planning.

[0047] The data augmentation method based on the joint use of 3D scene and language data provided in this application integrates multiple data sources, including scene point cloud data, RGB-D images, camera pose information, and text annotations (including question-answer pairs and dense descriptions). It utilizes data preprocessing, multimodal data augmentation, and semantic quality filtering techniques to automatically construct high-quality, large-scale datasets. This method has broad application prospects in 3D scene question answering, dense description generation, robot navigation, and task planning, providing robust data support for multimodal understanding of complex scenes, reducing data annotation costs and acquisition difficulties, and simultaneously improving the model's ability and robustness in extracting semantic information from complex scenes.

[0048] This application also provides another data augmentation method based on the joint use of 3D scene and language data, which is implemented on the basis of the above embodiments; this embodiment focuses on describing the specific implementation process of preprocessing, data augmentation, and semantic quality filtering.

[0049] The above steps of respectively pre-processing the 3D scene data and the text annotation data include:

[0050] (1) performing normalization processing on scene point cloud data in the 3D scene data, and performing resolution and color consistency processing on the RGB-D image;

[0051] The normalization processing on the scene point cloud data can ensure that the coordinate ranges of the point clouds in each scene are uniform, and eliminate data distribution differences caused by factors such as collection equipment and scene size; at the same time, the RGB-D image is pre-processed to ensure the consistency of image resolution and color information.

[0052] (2) performing data cleaning on the text annotation data to eliminate duplicate, grammatical error and incomplete information samples; using natural language processing technology to perform word segmentation, stop word removal and syntax structure analysis on the cleaned text annotation data, to obtain text annotation data with consistent and accurate semantics.

[0053] In order to improve the diversity and coverage of the data set, various data augmentation techniques are introduced in the embodiments of the present application to expand the text annotation data and the 3D scene data, that is, the above steps of respectively performing multi-modal data augmentation on the pre-processed 3D-language joint data to obtain augmented joint data include:

[0054] (1) performing rotation, translation and scale transformation operations on the pre-processed RGB-D image in the 3D scene data to realize image expansion;

[0055] Performing slight rotation, translation and scale transformation operations on the 3D scene data to simulate the angle and position changes that may occur in the real environment can increase the robustness of the model to different scene perspectives.

[0056] (2) based on a pre-trained language model, using synonym replacement, logical inversion and template generation strategies to expand the question and answer pairs and the dense description information; combining the image expanded 3D scene data and the data expanded text annotation data to obtain augmented joint data.

[0057] In actual application, based on a pre-trained language model (such as BERT, T5, etc.), synonym replacement, logical inversion, template generation and other strategies can be used to expand the question and answer pairs and the dense description. For example, through synonym replacement technology, "the sofa is on the left side of the living room" is converted to "the sofa is on the west side of the living room", more variants are generated under the premise of ensuring semantic consistency; using logical inversion and template generation methods, the descriptive text is converted into question and answer pairs, enriching the expression of data.

[0058] To ensure the high quality of the augmented joint data, the embodiments of the present application use a deep learning model to detect the semantic similarity and consistency of the generated question-answer pairs and descriptions, i.e., the step of performing semantic quality filtering on the augmented joint data, to obtain the target 3D-language joint data, which includes:

[0059] (1) Calculate the semantic similarity between the generated question-answer text and the original question-answer text using a pre-trained BERT model, and filter the text with a semantic similarity less than a first threshold value;

[0060] In specific implementation, the pre-trained BERT model can be used to extract text features and calculate the cosine similarity between the generated question-answer pairs and the original question-answer pairs. A similarity threshold value is set to select samples with high semantic similarity, ensuring that the generated question-answer pairs are highly consistent with the original data in terms of semantics.

[0061] (2) Use the RoBERTa model to perform natural language inference and calculate the consistency score between the generated dense description text and the original dense description text. Filter the description text with a consistency score less than a second threshold value to obtain the target 3D-language joint data.

[0062] In specific implementation, the RoBERTa model can be used for natural language reasoning to evaluate the logical consistency of the generated description text and the original annotation, and to eliminate samples with semantic contradictions or logical errors, ensuring that the final data set has high accuracy in the semantic level.

[0063] After the above preprocessing, data augmentation and semantic filtering, the high-quality data selected is integrated to form the final 3D-language joint data set. The data set includes tens of thousands of question-answer pairs and dense descriptions, covers multiple 3D scenes, and has rich semantic information and diverse expression forms, which can effectively support 3D scene question answering, dense description generation and robot task planning applications.

[0064] Compared with traditional data sets, the data augmentation method based on the combination of three-dimensional scene and language data provided by the embodiments generates a data set with wide coverage, high data quality and good semantic consistency through intelligent data preprocessing, semantic preservation data augmentation strategy and strict quality filtering means. Not only is the data scale greatly expanded, but also the data diversity, semantic consistency and quality control are significantly improved, providing strong data support for 3D visual question answering, dense description generation and robot task planning and other complex applications, and greatly reducing the cost and cycle of data collection and annotation.

[0065] In summary, the data augmentation method based on the combination of three-dimensional scene and language data provided by the embodiments has the following advantages:

[0066] 1. Large data scale and wide coverage: Through multi-source data integration and intelligent data enhancement methods, a high-quality dataset containing tens of thousands of question-answer pairs and dense descriptions can be generated, significantly improving the scale and scenario coverage of the original dataset.

[0067] 2. High data quality and good semantic consistency: Advanced natural language processing techniques and deep learning models are used to filter the generated data semantically, effectively eliminating low-quality and inconsistent samples, ensuring the accuracy and consistency of the data in semantic expression.

[0068] 3. Full fusion of multi-modal information: The embodiments of the application simultaneously process 3D scene data and text data, through preprocessing, data enhancement and filtering strategies, achieving full fusion of multi-modal information, providing strong data support for visual question answering and dense description generation in complex scenarios.

[0069] 4. Reducing data collection and annotation costs: Using automated data enhancement and semantic filtering techniques, the workload and cost of manual collection and annotation are significantly reduced, improving the efficiency and accuracy of data construction, and having high practical value and economic benefits.

[0070] The method provided by the embodiments of the application not only realizes the construction of high-quality 3D-language joint dataset by preprocessing, enhancing, filtering and integrating the original data, but also realizes the cross-modal upgrade from sparse data to dense data in an preferred implementation, and uses advanced deep learning models to efficiently encode text, visual cues and three-dimensional point cloud data, and finally generates high-quality natural language output through large-scale pre-training language model decoding. That is, the above method can further include the following steps:

[0071] (1) Encode the text data, visual cue data and three-dimensional point cloud data in the target 3D-language joint dataset respectively to obtain the encoding features corresponding to the three kinds of data respectively;

[0072] In specific implementation, the text data can be encoded based on the Transformer architecture to generate high-dimensional representation features; the visual cue data can be flattened and linearly mapped by a multi-layer perceptron to obtain spatial prior embedding features of the visual cue data; and the three-dimensional point cloud data can be feature-extracted by a Vote2Cap-DETR++ network to obtain deep geometric features of the three-dimensional point cloud data.

[0073] (2) Fuse the encoding features corresponding to the three kinds of data respectively to obtain fusion features;

[0074] In a specific implementation, the spatial priori embedding features corresponding to the visual prompt data and the deep geometric features corresponding to the three-dimensional point cloud data can be preliminarily fused in a splicing manner to obtain joint features; the high-dimensional representation features corresponding to the text data and the joint features are deeply fused through a cross-modal attention mechanism to obtain interactive features; and the interactive features and the high-dimensional representation features corresponding to the text data are subjected to self-attention processing, and the features are superimposed through a residual connection to form fused features.

[0075] (3) inputting the fused features into a pre-trained large-scale language model for natural language decoding to obtain target natural language.

[0076] Based on the above method embodiments, the embodiments of the present application also provide a data enhancement system based on three-dimensional scene and language data combination, that is, the embodiments propose an end-to-end system based on multi-modal data generation, semantic quality filtering and large-scale pre-training language model decoding for the question and answer and dense description tasks in a three-dimensional scene. The system not only can automatically construct a large-scale high-quality 3D-language combined data set, but also realizes innovative technical breakthroughs in multi-modal information fusion, data enhancement and text generation and other key links, thereby showing excellent semantic reasoning and spatial understanding capabilities when processing complex three-dimensional scenes. Through preprocessing, enhancement, filtering and integration of the original data, the embodiments realize cross-modal upgrading from sparse data to dense data, and use advanced deep learning models to efficiently encode text, visual prompts and three-dimensional point cloud data, and finally generate high-quality natural language output through large-scale pre-training language model decoding.

[0077] As shown in Figure 2 , the overall system flow starts from raw data acquisition, including three-dimensional point cloud data from ScanNet, question and answer annotations in ScanQA and dense descriptions in ScanRefer. After preprocessing, the raw data enters the multi-modal data enhancement module. The module uses a pre-trained language model to perform synonym replacement, logical inversion and template generation on the text data, thereby expanding the diversity of the data while maintaining semantic consistency. After preliminary synthesis, the enhanced data is subjected to semantic quality filtering using a pre-trained model to ensure that the output data set has high accuracy and consistency, and finally completes the integration output of the data set. Figure 2 The figure intuitively shows the overall flow of data set construction, and describes in detail each step from raw data acquisition, preprocessing, multi-modal data enhancement, semantic quality filtering to final data set integration output, as well as the data flow and processing relationship therebetween, fully embodying the systematic and automated features of the embodiments in data construction.

[0078] In the text data enhancement link, as shown in Figure 3As shown, the embodiments of the present application utilize a pre-trained language model to expand the original text data from multiple angles and in all directions. Specifically, for the question and answer in the original question-answer pair, the system first uses synonym replacement technology for semantic expansion, and by introducing different but semantically consistent words, it generates multiple versions of the question-answer text. At the same time, for the logical relationship in the original text, the embodiments of the present application use logical inversion technology to generate question-answer pairs with reversed semantics, and then combine the pre-defined templating generation method to convert the dense description into a question-answer pair using the T5 model. The conversion process can be described by the following formula:

[0079]

[0080] where C represents the original description text, P template represents the pre-set text template, and the symbol represents the text splicing operation. Through this series of text data augmentation strategies, not only the data size is effectively expanded, but also the data diversity is greatly improved while maintaining the consistency of the text semantics. Figure 2 The specific process of the text data augmentation module is detailed, and the calling relationship and data flow between each processing module are fully embodied, providing sufficient candidate data for subsequent semantic quality filtering.

[0081] After data augmentation, the system further uses the semantic quality filtering module to strictly filter the generated text. Figure 4 The overall process of the semantic quality filtering module is shown, and the core idea is to use pre-trained BERT and RoBERTa models to calculate the semantic similarity and consistency between the generated text and the original text, and to filter the data according to the pre-set threshold. Specifically, for the generation process of the question-answer pair, the system uses the BERT model to calculate the cosine similarity between the original question and the generated question, and the calculation formula is:

[0082] SQ(Q orig ,Q gen )=cos(f BERT (Q orig ),f BERT (Q gen ));

[0083] where f BERTdenotes the semantic vector obtained based on the BERT model, and cos(·, ·) denotes a cosine similarity function. By setting a task-specific threshold (for example, the threshold is set to 0.82 for a question and answer task), the system can eliminate samples that are greatly deviated from the original question and answer in semantics. In addition, in order to further ensure the semantic consistency of the generated data, the system uses the RoBERTa model to perform natural language inference (NLI) to calculate the consistency score between the description texts, and the calculation formula is as follows:

[0084]

[0085] wherein C orig denotes the original description text, denotes the i-th generated description, and n is the number of generated samples. After the series of strict semantic filtering steps, the system finally selects high-quality and highly consistent text data, and together with other modal data, forms a large-scale 3D-language joint data set. Figure 4 The process of using the BERT and RoBERTa models to detect the semantic similarity and consistency of the enhanced text is illustrated in detail, the data information transmission and threshold screening mechanism between each link are indicated, and how to accurately eliminate low-quality samples in a large amount of data is directly reflected, so that the overall quality of the data set meets the expected requirements.

[0086] In terms of multi-modal data encoding, the embodiment adopts a unified embedding encoding architecture to map text, visual cues and three-dimensional point cloud data into the same high-dimensional feature space. Specifically, for text data, the system encodes the input text sequence based on the Transformer architecture, and the calculation process can be described as follows: After embedding lookup and position encoding, a high-dimensional representation E t is generated, and the calculation process can be described as follows:

[0087] E t =Transformer enc (Embed(T)+P t );

[0088] wherein Embed(T) denotes an embedding operation of converting text into a word vector, and P t denotes position encoding; for visual cue data, the system first obtains 3D bounding box information by using a target detection algorithm or user labeling, and then flattens and linearly maps the 3D bounding box information by using a multi-layer perceptron (MLP) to obtain the spatial prior embedding E v of the visual cue: E v =MLP(Flatten(V)) and for three-dimensional point cloud data The system adopts Vote2Cap-DETR++ network for feature extraction to generate deep geometric features E of point cloud data s :

[0089]

[0090] Figure 4 The schematic diagram of the multi-modal embedding coding structure is directly shown, and the specific process of text coding, visual cue coding and three-dimensional point cloud coding is described in detail, and how each module uses Transformer, MLP and Vote2Cap-DETR++ network to realize high-dimensional feature extraction and mapping is shown, so as to construct a unified multi-modal feature space.

[0091] In the implementation of multi-modal fusion, the system first preliminarily fuses the visual cue and the three-dimensional point cloud data, and adopts splicing to combine the coding results E v and E s to obtain joint features E vs :

[0092] E vs =Concat(E v ,E s );

[0093] Then, the system deeply fuses the text coding E t and the joint features E vs through the cross-modal attention mechanism to calculate the interaction features E f1 :

[0094]

[0095] Wherein, W q , W k and W v are projection matrices of query, key and value respectively, and d is the embedding dimension. In order to ensure the continuity of the text information and the complete expression of the context, the system further performs self-attention processing on the interaction features and the original text coding, and superimposes them through residual connection to form the final fusion representation E f :

[0096] E f =LayerNorm(TransformerLayer(E f1 ||E t )+E t );

[0097] where "||" represents the concatenation operation of features. Through the above multi-modal fusion and encoding process, the system realizes the efficient fusion of text, visual cues and three-dimensional point cloud information, thereby providing comprehensive and accurate semantic and spatial information support for subsequent natural language generation. After completing the multi-modal feature fusion, the system inputs the fused features E f into a pre-trained large-scale language model for natural language decoding. Specifically, the OPT-1.3B model is used as the decoder, and through the dynamic prefix projection technology, the fused features are mapped into the prefix information required by the language model, thereby guiding the generation process. The decoding process can be represented as:

[0098] h t =LLM(p 1:t ; LinearProj(E f ));

[0099] where LinearProj(E f ) represents the linear mapping of the fused features into prefix information suitable for language model input, p 1：t represents the generated token sequence, and the probability distribution of generating the next token is:

[0100] p(w t+1 ) = softmax(h t W vocab );

[0101] Through the above generation mechanism, the system can generate natural and fluent and semantically accurate question and answer or description text while making full use of multi-modal information, thereby significantly improving the performance of the model in three-dimensional scene semantic understanding and reasoning.

[0102] The present embodiment adopts an end-to-end joint training strategy to integrate multi-modal data generation, semantic quality filtering, multi-modal encoding and large-scale language model decoding together. During training, the system simultaneously minimizes the question and answer generation loss L QA and the description generation loss L cap , and forms a total loss function, which is expressed as:

[0103] L total = λ QA · L QA + λ cap · L cap ;

[0104] where λ QA and λ capare the weight coefficients of each task, and the optimal values are obtained through cross-validation. The question-answer generation loss is mainly based on the cross-entropy error between the generated question-answer and the true question-answer, while the description generation loss combines indicators such as ROUGE and CIDEr to comprehensively evaluate the matching degree between the generated text and the true text. To ensure stability in large-scale data training, the system uses the AdamW optimizer and combines the cosine annealing learning rate scheduling strategy to effectively avoid the model from falling into local optimum and overfitting problem. Table 1 below shows the question-answer task-contrast method and detailed information, and Table 2 shows the dense caption task-contrast method and detailed information;

[0105] Table 1

[0106]

[0107] Table 2

[0108]

[0109]

[0110] As shown in the experimental results in Table 1 and Table 2, the system proposed in this embodiment has achieved significant performance improvement over existing methods in multiple three-dimensional scene question-answer and dense description tasks. For example, on the ScanQA dataset, the CIDEr indicator is improved by about 2.15%, and on the ScanRefer dataset, the CIDEr@0.5 indicator is improved by about 1.84%. In addition, the ablation experiment verifies the positive effect of data augmentation strategy, semantic filtering mechanism and multi-modal fusion module on the overall performance of the system, further proving the technical advantages of this embodiment in large-scale data construction and end-to-end joint training.

[0111] This embodiment preprocesses and multi-modal data augments the original three-dimensional point cloud data, question-answer annotation data and dense description data, uses advanced pre-trained language models for text generation and semantic filtering, and combines cross-modal attention mechanism to realize multi-modal feature fusion, thereby generating high-quality natural language output under end-to-end joint training. Figure 2 、 Figure 3 、 Figure 4 and Figure 5The core modules and their internal working principles of the embodiment are intuitively displayed from the four aspects of the overall process, text data enhancement, semantic quality filtering, and multi-modal embedding coding, which embodies the whole process from data generation, quality control to multi-modal feature fusion and natural language decoding. The system not only realizes the dual improvement of data scale and quality in the data construction and preprocessing stage through various advanced technologies, but also fully utilizes the advantages of deep learning models in the multi-modal information fusion and language generation stage, realizes the efficient expression and accurate generation of complex three-dimensional scene semantic information, and thus provides an efficient, accurate and high practical value solution for the question and answer and dense description tasks in complex 3D scenes.

[0112] The method and system provided by the embodiment have obvious advantages in technology. On the one hand, the problem of traditional three-dimensional scene data scarcity and insufficient semantic information is effectively broken through by multi-modal data generation and enhancement. On the other hand, by using the pre-training language model and multi-modal fusion technology, the system can capture the deep correlation between text, visual cues and three-dimensional geometric information in a high-dimensional feature space, significantly improving the accuracy and naturalness of question and answer and description generation. Through the detailed design and end-to-end joint optimization of each module of the system, the embodiment not only theoretically constructs a unified 3D-language data construction and generation framework, but also realizes the automatic construction and efficient utilization of large-scale high-quality data in practice, providing solid technical support for the further development of three-dimensional vision and language tasks. Therefore, the embodiment proposes a new three-dimensional scene question and answer and dense description method by systematically integrating data generation, semantic filtering, multi-modal coding and natural language generation. The method ensures that the generated text has high semantic consistency and accuracy, and realizes efficient understanding and expression of complex spatial information through large-scale pre-training models and cross-modal attention mechanisms, thereby significantly outperforming the prior art in multiple evaluation indicators. The technology not only has high academic research value, but also has great potential in practical application, providing a new solution for the development of future intelligent three-dimensional scene analysis and human-computer interaction technology.

[0113] Based on the above method embodiment, the embodiment of the present application also provides a data enhancement device based on three-dimensional scene and language data joint, see Figure 6As shown, the device comprises: a data acquisition module 62, configured to acquire 3D scene data and corresponding text annotation data; the 3D scene data comprises: scene point cloud data, RGB-D images and camera pose information; the text annotation data comprises: question and answer pairs and dense description information related to the positions, attributes and relationships of scene objects; a preprocessing module 64, configured to preprocess the 3D scene data and the text annotation data respectively to obtain preprocessed 3D-language joint data; the preprocessing comprises: normalization processing of the scene point cloud data, consistency processing of the RGB-D images, and data cleaning processing and syntax structure analysis processing of the text annotation data; a data enhancement module 66, configured to perform multi-modal data enhancement on the preprocessed 3D-language joint data respectively to obtain enhanced joint data; the multi-modal data enhancement comprises: rotation, translation and scale transformation of the RGB-D images, and data expansion operations of the question and answer pairs and the dense description information by using three strategies of synonym replacement, logical inversion and template generation; a filtering processing module 68, configured to perform semantic quality filtering processing on the enhanced joint data to obtain a target 3D-language joint data set.

[0114] Further, the preprocessing module 64 is configured to perform normalization processing on the scene point cloud data in the 3D scene data, and perform resolution and color consistency processing on the RGB-D images; perform data cleaning on the text annotation data to eliminate repeated, syntax error and incomplete information samples; perform word segmentation, stop word removal and syntax structure analysis on the data cleaned text annotation data by using natural language processing technology to obtain text annotation data with required semantic consistency and accuracy.

[0115] Further, the data enhancement module 66 is configured to perform rotation, translation and scale transformation operations on the RGB-D images in the preprocessed 3D scene data to realize image expansion; based on a pre-trained language model, perform data expansion on the question and answer pairs and the dense description information by using three strategies of synonym replacement, logical inversion and template generation; combine the image expanded 3D scene data and the data expanded text annotation data to obtain enhanced joint data.

[0116] Further, the filtering processing module 68 is configured to calculate semantic similarity between the generated question and answer text and the original question and answer text by using a pre-trained BERT model, and filter the text with a semantic similarity less than a first threshold; perform natural language inference by using a RoBERTa model to calculate consistency score between the generated dense description text and the original dense description text, and filter the description text with a consistency score less than a second threshold to obtain target 3D-language joint data.

[0117] Further, the device further includes a language generation module configured to encode the text data, the visual cue data, and the three-dimensional point cloud data in the target 3D-language joint dataset respectively to obtain encoded features corresponding to the three types of data respectively, fuse the encoded features corresponding to the three types of data respectively to obtain fused features, and input the fused features into a pre-trained large-scale language model to perform natural language decoding to obtain a target natural language.

[0118] Further, the language generation module is configured to encode the text data based on a Transformer architecture to generate high-dimensional representation features, flatten and linearly map the visual cue data by a multilayer perceptron to obtain spatial prior embedding features of the visual cue data, and extract features of the three-dimensional point cloud data by a Vote2Cap-DETR++ network to obtain deep geometric features of the three-dimensional point cloud data.

[0119] Further, the language generation module is configured to preliminarily fuse the spatial prior embedding features corresponding to the visual cue data and the deep geometric features corresponding to the three-dimensional point cloud data in a concatenation manner to obtain joint features, deeply fuse the high-dimensional representation features corresponding to the text data and the joint features by a cross-modal attention mechanism to obtain interactive features, and perform self-attention processing on the interactive features and the high-dimensional representation features corresponding to the text data, and stack the features by a residual connection to form the fused features.

[0120] The device provided by the embodiment of the present application has the same implementation principle and technical effects as the foregoing method embodiment, and for brevity of description, the part of the embodiment of the device that is not mentioned can be referred to the corresponding content in the foregoing method embodiment.

[0121] Based on the foregoing method embodiment, the embodiment of the present application further provides a data enhancement system based on three-dimensional scene and language data joint, which is a three-dimensional scene question answering and dense description system (hereinafter referred to as "3D-MoRe system") based on multi-modal data generation, fusion and large-scale pre-training language model decoding. The system aims to automatically generate a large amount of high-quality three-dimensional language data, and effectively improve the understanding and reasoning ability of semantic information in complex three-dimensional scenes by using advanced data enhancement, semantic filtering and cross-modal interaction technology, so as to achieve an effect close to manual annotation in 3D question answering and dense description tasks.

[0122] As shown in Figure 7 the structural schematic diagram of the system, the system includes a processor 71 and a memory 70, the memory 70 stores computer executable instructions capable of being executed by the processor 71, and the processor 71 executes the computer executable instructions to implement the foregoing method.

[0123] In Figure 7In the illustrated embodiment, the system further comprises a bus 72 and a communication interface 73, wherein the processor 71, the communication interface 73 and the memory 70 are connected through the bus 72.

[0124] The memory 70 can include a high-speed random access memory (RAM), and can also include a non-volatile memory, such as at least one disk memory. The communication connection between the system network element and at least one other network element is realized through at least one communication interface 73 (which can be wired or wireless), and the Internet, a wide area network, a local area network, a metropolitan area network, etc. can be used. The bus 72 can be an ISA (Industry Standard Architecture) bus, a PCI (Peripheral Component Interconnect) bus, or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 72 can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 7 Only one bidirectional arrow is used in the figure, but it does not mean that there is only one bus or only one type of bus.

[0125] The processor 71 can be an integrated circuit chip with signal processing capability. In the implementation process, each step of the above method can be completed by the integrated logic circuit of hardware in the processor 71 or the instruction in the form of software. The processor 71 described above can be a general processor, including a central processing unit (CPU), a network processor (NP), etc.; can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component. The general processor can be a microprocessor or the processor can also be any conventional processor. The steps of the method disclosed in combination with the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a random access memory, a flash memory, a read only memory, a programmable read only memory or an electrically erasable programmable memory, a register, etc. The storage medium in the memory is read by the processor 71, and the hardware thereof is combined to complete the steps of the method of the foregoing embodiments.

[0126] The embodiment of the present application further provides a computer readable storage medium, which stores computer executable instructions. When the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to implement the above method. For details, refer to the foregoing method embodiments, which will not be described here.

[0127] The computer program product of the method, device and system provided by the embodiment of the present application includes a computer readable storage medium storing program codes. The instructions included in the program codes can be used to execute the method described in the foregoing method embodiments. For details, refer to the method embodiments, which will not be described here.

[0128] Unless otherwise specifically stated, the relative steps, numerical expressions and numerical values of the components and steps set forth in these embodiments do not limit the scope of the present application.

[0129] If the functions are implemented in the form of software function units and sold or used as independent products, they can be stored in a nonvolatile computer readable storage medium executable by a processor. Based on this understanding, the technical solutions of the present application or the part of the prior art that essentially contributes to the prior art or the part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in the embodiments of the present application. The aforementioned storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various storage medium that can store program codes.

[0130] In the description of the present application, it should be noted that the terms "center", "upper", "lower", "left", "right", "vertical", "horizontal", "inner", "outer" and the like indicate the orientation or positional relationship based on the orientation or positional relationship shown in the drawings, and are only for the purpose of facilitating the description of the present application and simplifying the description, and do not indicate or imply that the devices or elements referred to must have a particular orientation, be constructed and operated in a particular orientation, and therefore cannot be understood as limiting the present application. In addition, the terms "first", "second", "third" are only for the purpose of description, and cannot be understood as indicating or implying relative importance.

[0131] Finally, it should be noted that: the above-described embodiments are only specific embodiments of the present application, used to illustrate the technical solutions of the present application, and are not limited thereto, the protection scope of the present application is not limited thereto, although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that any person skilled in the art within the technical scope disclosed by the present application can modify or easily think of changes to the technical solutions recorded in the foregoing embodiments, or make equivalent replacement to part of the technical features; and these modifications, changes or replacements do not make the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application, and should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A method for data enhancement based on a combination of three-dimensional scene and language data, characterized in that, The method comprises: acquiring 3D scene data and corresponding text annotation data; the 3D scene data comprises scene point cloud data, RGB-D images and camera pose information; the text annotation data comprises question and answer pairs and dense description information related to scene object positions, attributes and relationships; respectively pre-processing the 3D scene data and the text annotation data to obtain pre-processed 3D-language joint data; the pre-processing comprises normalization processing of the scene point cloud data, consistency processing of the RGB-D images, and data cleaning processing and syntax structure analysis processing of the text annotation data; performing multi-modal data enhancement on the pre-processed 3D-language joint data to obtain enhanced joint data; the multi-modal data enhancement comprises rotation, translation and scale transformation multi-image expansion operations on the RGB-D images, and data expansion operations of synonym replacement, logical inversion and template generation of three strategies on the question and answer pairs and the dense description information; performing semantic quality filtering processing on the enhanced joint data to obtain a target 3D-language joint data set; The method further comprises: encoding the text data, visual cue data and three-dimensional point cloud data in the target 3D-language joint data set respectively to obtain encoding features corresponding to the three kinds of data respectively, including: encoding the text data based on a Transformer architecture to generate high-dimensional representation features; performing flattening and linear mapping on the visual cue data through a multi-layer perception machine to obtain spatial prior embedding features of the visual cue data; extracting features of the three-dimensional point cloud data using a Vote2Cap-DETR++ network to obtain deep geometric features of the three-dimensional point cloud data; fusing the encoding features corresponding to the three kinds of data respectively to obtain fused features, including: preliminarily fusing the spatial prior embedding features corresponding to the visual cue data and the deep geometric features corresponding to the three-dimensional point cloud data in a splicing manner to obtain joint features; deeply fusing the high-dimensional representation features corresponding to the text data and the joint features through a cross-modal attention mechanism to obtain interactive features; performing self-attention processing on the interactive features and the high-dimensional representation features corresponding to the text data, and performing feature superposition through residual connection to form fused features; inputting the fused features into a pre-trained large-scale language model for natural language decoding to obtain a target natural language.

2. The method of claim 1, wherein, The step of respectively pre-processing the 3D scene data and the text annotation data comprises: performing normalization processing on the scene point cloud data in the 3D scene data, and performing resolution and color consistency processing on the RGB-D images; performing data cleaning on the text annotation data to eliminate duplicate, grammatically incorrect and incomplete information samples; performing word segmentation, stop word removal and syntax structure analysis on the data cleaned text annotation data using natural language processing technology to obtain text annotation data with consistent and accurate semantics.

3. The method of claim 1, wherein, The step of performing multi-modal data enhancement on the preprocessed 3D-language joint data to obtain enhanced joint data comprises: The RGB-D image in the preprocessed 3D scene data is subjected to rotation, translation and scale transformation operations to achieve image expansion; Based on the pre-trained language model, the question-answer pair and the dense description information are expanded by using three strategies of synonym replacement, logical inversion and template generation; The image-expanded 3D scene data and the data-expanded text annotation data are combined to obtain enhanced joint data.

4. The method of claim 1, wherein, The step of performing semantic quality filtering processing on the enhanced joint data to obtain target 3D-language joint data comprises: The pre-trained BERT model is used to calculate the semantic similarity between the generated question-answer text and the original question-answer text, and the text with a semantic similarity less than a first threshold value is filtered; The RoBERTa model is used for natural language inference to calculate the consistency score between the generated dense description text and the original dense description text, and the description text with a consistency score less than a second threshold value is filtered to obtain the target 3D-language joint data.

5. A data enhancement device based on three-dimensional scene and language data combination, characterized in that, The device comprises: A data acquisition module for acquiring 3D scene data and corresponding text annotation data; the 3D scene data comprises scene point cloud data, RGB-D images and camera pose information; the text annotation data comprises question-answer pairs and dense description information related to scene object position, attributes and relationships; A preprocessing module for preprocessing the 3D scene data and the text annotation data respectively to obtain preprocessed 3D-language joint data; the preprocessing comprises normalization processing of the scene point cloud data, consistency processing of the RGB-D images, and data cleaning processing and syntax structure analysis processing of the text annotation data; A data enhancement module for performing multi-modal data enhancement on the preprocessed 3D-language joint data to obtain enhanced joint data; the multi-modal data enhancement comprises rotation, translation and scale transformation operations for the RGB-D images, and data expansion operations of the question-answer pairs and the dense description information by using three strategies of synonym replacement, logical inversion and template generation; A filtering processing module for performing semantic quality filtering processing on the enhanced joint data to obtain a target 3D-language joint data set; The natural language generation module is configured to encode the text data, the visual cue data, and the three-dimensional point cloud data in the target 3D-language joint dataset respectively to obtain encoded features corresponding to the three types of data, including: encoding the text data based on a Transformer architecture to generate high-dimensional representation features; performing flattening and linear mapping on the visual cue data by a multilayer perceptron to obtain spatial prior embedding features of the visual cue data; extracting features of the three-dimensional point cloud data by a Vote2Cap-DETR++ network to obtain deep geometric features of the three-dimensional point cloud data; and fusing the encoded features corresponding to the three types of data to obtain fused features, including: preliminarily fusing the spatial prior embedding features corresponding to the visual cue data and the deep geometric features corresponding to the three-dimensional point cloud data in a concatenation manner to obtain joint features; deeply fusing the high-dimensional representation features corresponding to the text data and the joint features by a cross-modal attention mechanism to obtain interactive features; performing self-attention processing on the interactive features and the high-dimensional representation features corresponding to the text data, and stacking the features by a residual connection to form the fused features; and inputting the fused features into a pre-trained large-scale language model for natural language decoding to obtain a target natural language. 6.A data augmentation system based on a combination of a three-dimensional scene and language data, characterized in that, The system includes a processor and a memory, the memory stores computer executable instructions capable of being executed by the processor, and the processor executes the computer executable instructions to implement the method in any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that, The computer readable storage medium stores computer executable instructions, and when the computer executable instructions are called and executed by a processor, the computer executable instructions cause the processor to implement the method in any one of claims 1 to 4.

Images