An automatic driving three-dimensional scene data preprocessing method and system based on a large language model

Abstract

The application discloses an automatic driving three-dimensional scene data preprocessing method and system based on a large language model, a text end generates a question paradigm for comparative learning based on a large language model for each category label, stimulates factual knowledge of the large language model, takes the factual knowledge as an answer space, generates a detailed category template for the category label of the automatic driving task, and caches the category template into an offline file for loading during training of a downstream model, expands the category template, and strengthens the most core category phrase; a visual end obtains key frames of an input video sequence through sparse sampling and dense sampling, uses a video random data enhancement method to perform image transformation on the obtained key frames, and enhances the robustness of a model to visual representation. The application processes information of a text mode and information of a visual mode respectively, fusion of different preprocessing methods can capture different prior knowledge, and complementary characteristics thereof are utilized to realize better performance.

Description

TECHNICAL FIELD

[0001] The present application belongs to the technical field of automatic driving, and particularly relates to an automatic driving three-dimensional scene data preprocessing method and system based on a large language model. BACKGROUND

[0002] The multi-modal model trained based on large-scale language-image sample pairs can well apply the text understanding ability of the pre-trained language model to downstream tasks and obtain good results in zero-sample or small-sample scenarios. In the automatic driving task, text information also plays an important role. Text description can provide key information about roads, traffic signs, pedestrian behavior, etc., which helps to more comprehensively understand and analyze three-dimensional scenes. Therefore, making full use of the information of the text modality is crucial to improve the performance of the automatic driving system. Some researchers have explored the use of such models in automatic driving three-dimensional scene understanding tasks, and have realized that multi-modal knowledge can effectively help three-dimensional scene understanding, thus achieving impressive performance. While exploring the network architecture of deep models, researchers have also realized the importance of data preprocessing as a means to improve model performance. However, existing data preprocessing methods mostly only focus on traditional visual modality information, and rarely process information of the text modality. SUMMARY

[0003] To solve the above problems, the present application provides an automatic driving three-dimensional scene data preprocessing method and system based on a large language model, aiming to make full use of the information of text and visual input and improve the performance and effect of the model. At the text input end, a large language model prompt template is designed according to the prompt learning method and applied to the category label. The template describes each target in each scene in detail, so that the label can carry more semantic information. At the same time, a question paradigm is designed to obtain the optimal description, so as to guide the large language model to generate more accurate and rich text information. In addition, the basic prompt template is expanded, and the category label is added according to the weight to strengthen the key information in the template, further improving the quality and accuracy of the description. A processing mode of prompting, generating, caching and expanding is formed at the text end to enhance the text data. At the visual input end, a video frame fragmentation sampling strategy and a video random data enhancement method are used to preprocess the input frames, thereby reducing the redundancy of the input frames, reducing the model parameter quantity and calculation quantity, at the same time, enhancing the robustness of the model to visual representation and expanding the diversity of the training samples. The fusion of the text end and visual end data preprocessing methods can capture different prior knowledge, enhance the expression ability of the text data and the feature robustness of the visual data, and utilize their complementary characteristics to achieve better performance.

[0004] The purpose of the present application is to solve the problems existing in the prior art, and provide an automatic driving three-dimensional scene data preprocessing method based on a large language model, and the technical scheme is as follows:

[0005] The automatic driving three-dimensional scene data preprocessing method based on the large language model comprises the following steps:

[0006] In the first aspect, the data is processed at the text input end;

[0007] Step 1: based on the large language model, a question paradigm for comparative learning is generated for each category label, and the factual knowledge in the large language model is stimulated;

[0008] Step 2: taking the factual knowledge as an answer space, a detailed category template is generated for the category label of the automatic driving task;

[0009] Step 3: the generated category template is cached in an offline file, and is loaded during training of a downstream model;

[0010] Step 4: the category template is expanded, and the most core category phrase is strengthened;

[0011] In the further technical scheme, the question paradigm in step 1 is designed as follows:

[0012] The question paradigm follows the principle of 5W1H, wherein Who represents: what identity the large language model wants to answer the question; What represents: what the large language model wants to do; Where represents: in what scenario to answer; When represents: what stage the current work has reached; Why represents: why to do this thing; and How represents: what way the large language model wants to execute;

[0013] The large language model interface is called to generate a new category template using the question paradigm.

[0014] In the further technical scheme, the caching process in step 3 comprises: a self-defined format file is used to save the category template, and the category template is loaded, and the specific process is as follows:

[0015] In the process of generating the category template, the template of each category is saved in a JSON, CSV or self-defined format offline file;

[0016] Each category template is associated with its corresponding category label, and contains detailed descriptions and related information;

[0017] In the downstream model training stage, the category template generated and cached previously needs to be loaded;

[0018] In the model training process, related information is obtained from the loaded category template according to needs, and is used to guide the model training and evaluation;

[0019] Further technical solutions, the expansion category template in step 4 includes: category expansion, manual template expansion and mark expansion mode. The specific process is:

[0020] The category expansion is to add a category label to the category template generated by the large language model in advance;

[0021] The manual template expansion is to add a manual template in front of the generated category template to optimize the category label;

[0022] The mark expansion is to modify the weight of a certain specific word through mark symbols such as "()", [], and {}.

[0023] Secondly, the data is processed at the visual input end;

[0024] Step 5: Key frames of the input video sequence are obtained through sparse sampling and dense sampling, which are used as the input of the visual end of the model;

[0025] Step 6: Using the video random data augmentation method, image transformation is performed on the key frames obtained by sampling to enhance the robustness of the model to visual representation;

[0026] Further technical solutions, the key frames of the input video sequence obtained in step 5 include: sampling video segments, performing segmentation operation, and extracting frames more critical to understanding the video; The specific process is:

[0027] Define the cumulative distribution function:

[0028] Z(x) = ∑y(x)

[0029] Where Z(x) is the probability cumulative distribution function, the horizontal axis of the distribution function is the time axis, and the vertical axis is the probability distribution function of 0-1, y(x) represents the probability density, and the motion cumulative distribution curve is constructed based on the function;

[0030] T segments are sampled from the video data, and a value is randomly selected on each segment, which can obtain T video frames;

[0031] According to the constructed motion cumulative distribution curve, the T frames sampled are subjected to segmentation operation, and M frames are sampled in each segment instead of only one frame, which can obtain T×M frame images;

[0032] Ensure the sampling efficiency and expand the channel dimension by M times at the same time;

[0033] Based on content sampling, the vehicle, pedestrian, motion, scene and other information of the input frame are analyzed;

[0034] Further technical solutions, the video random data enhancement method in step 6 includes: randomly combining image enhancement transformations, setting hyperparameters, and using data enhancement operations on the video; the specific process is:

[0035] N kinds of image enhancement are randomly combined, such as rotation, translation, scaling, cropping, color adjustment, and the like, to generate new samples after enhancement as the parameter space;

[0036] The hyperparameter Q is set as the enhancement quantity, and P is set as the enhancement intensity;

[0037] In the parameter space, Q is randomly selected from N kinds of enhancement each time, so that a total of N Q enhancement strategies are included, and different intensities P can be selected for each strategy;

[0038] The same enhancement transformation is applied to all frames of the same video, and a randomly selected enhancement transformation strategy is used for different videos.

[0039] In a third aspect, an automatic driving three-dimensional scene data preprocessing network is constructed;

[0040] Step 7: The text data preprocessed in the first aspect and the visual image preprocessed in the second aspect are sent into a text encoder, an image encoder, and a point cloud encoder together with the point cloud data to obtain text embedding, image pixel features, and point features;

[0041] Step 8: The positive samples and negative samples for contrastive learning are constructed by using the knowledge of the contrastive language-image pretraining model, to obtain point cloud-text pairs and all pixel-point-text pairs in a short time, and to learn the semantic correlation between the point cloud or pixel and the text description;

[0042] Step 9: The contrastive language pretraining model is trained to obtain the result.

[0043] Further technical solutions, the specific process in step 7 is:

[0044] Given a video segment v∈video and a text description c∈text, data preprocessing is performed on v and c.

[0045] After visual preprocessing, the following is obtained:

[0046]

[0047] Among them represents new data after the segmentation sampling and the video random data enhancement operation, v represents the visual original data, P μ represents the data processing process;

[0048] After text preprocessing, the following is obtained:

[0049]

[0050] wherein, represents the language and character description of the target category generated by the large language model, c represents the original category label, O μ represents the process of generating a new category label by the large language model;

[0051] Subsequently, the preprocessed text data is sent into a text encoder, an image encoder and a point cloud encoder together with the image and point cloud data, the text input is processed, the image data is processed, and the point cloud data is processed; the specific processing is as follows:

[0052] The category template is put into the predefined text encoder to generate a text embedding:

[0053]

[0054] wherein is the final generated text embedding representation, is the text encoder;

[0055] The enhanced image data is put into the image encoder for encoding to obtain an image embedding:

[0056]

[0057] wherein is the final generated visual representation, i.e., the image embedding, is the image encoder;

[0058] The original point cloud data p is put into the point cloud encoder to obtain point features

[0059]

[0060] wherein, point represents a point cloud data set, F P represents the point cloud encoder.

[0061] The similarity between the image embedding and the text embedding is calculated to determine the category;

[0062] The beneficial effects of the present application are:

[0063] 1. The present application faces the contrast language pre-training model, which has great potential in enhancing two-dimensional segmentation and three-dimensional classification performance, and is also beneficial to three-dimensional scene understanding (refinement). A new data preprocessing method is constructed based on the model, which integrates the data preprocessing methods of the visual end and the text end together, realizes the comprehensive utilization of different prior knowledge, and such comprehensive utilization can make the model more comprehensively consider the information of different data sources in various tasks, thereby improving the performance and generalization ability of the model.

[0064] 2、The application provides a category template based on a large language model on the text end. The factual knowledge generated by the large language model is used as an answer space, and detailed category templates are generated for the category labels of the automatic driving task. The processing mode of prompting, generating, caching, and expanding solves the problem of difficult text modal information processing. The large language model uses its deep learning model to train large-scale text data, so as to generate text content with semantic logic and rich information, providing comprehensive and accurate descriptions for the category labels of the automatic driving task. The category templates generated by the large language model are stored and managed, which not only accelerates the generation speed of the templates, but also avoids repeated calculations and improves processing efficiency. Through the expansion method, the existing category templates are constantly enriched and improved. With the development of the automatic driving field and the accumulation of knowledge, new prompt information can be provided to the large language model to generate more comprehensive and accurate category templates, further improving the usability and applicability of the text data, and greatly improving the usability of the text data.

[0065] 3、The fragmentation sampling strategy introduced on the visual end can selectively retain key frames related to the target task and filter out redundant information, so that the model can focus on processing the most important visual information without being disturbed by unnecessary details, thereby reducing the model parameter quantity and calculation amount, and significantly improving the calculation and reasoning efficiency of the model. The data enhancement method introduced can enhance the robustness of the model to visual representation by transforming the training samples, increase the diversity of the training data, help the model learn more generalized features, improve the adaptability of the model to various visual changes, make it more robust when facing complex scenes in the real world, and avoid overfitting. BRIEF DESCRIPTION OF DRAWINGS

[0066] Figure 1 A flowchart of the overall process of the automatic driving three-dimensional scene data preprocessing method based on the large language model provided by the application;

[0067] Figure 2 A schematic diagram of the overall process of the automatic driving three-dimensional scene data preprocessing method based on the large language model provided by the application;

[0068] Figure 3 A text end data preprocessing diagram of the automatic driving three-dimensional scene data preprocessing method based on the large language model provided by the application;

[0069] Figure 4 A visual end data preprocessing diagram of the automatic driving three-dimensional scene data preprocessing method based on the large language model provided by the application. DETAILED DESCRIPTION

[0070] In order to make the technical solutions and advantages of the present application clearer, the technical solutions in the embodiments of the present application will be described clearly and completely below in combination with the drawings in the embodiments of the present application. For the sake of clarity and conciseness, not all the features of the actual embodiments are described in the specification. However, it should be understood that many embodiment-specific decisions can be made in the process of developing any such actual embodiment in order to achieve the specific goals of the developers, and these decisions can vary with different embodiments. The scope of protection of the present application is not limited to the described embodiments.

[0071] As shown in Figure 1 , 2 , the present application proposes an automatic driving three-dimensional scene data preprocessing method based on a large language model, which specifically includes the following steps:

[0072] S1: According to the first aspect of the present disclosure, a method for preprocessing input data on the text side is provided, as shown in Figure 3 , specifically including:

[0073] S11: Based on the large language model, a question paradigm for comparative learning is generated for each category label to stimulate factual knowledge in the large language model.

[0074] Specifically, the designed question paradigm follows the principle of 5W1H, where Who represents: what identity the large language model wants to answer the question; What represents: what the large language model wants to do; Where represents: answer in what scenario; When represents: what stage of work is done now; Why represents: why do this; How represents: what way the large language model wants to execute;

[0075] In a specific implementation, the question paradigm can be designed as: you are a prompt word engineer, what useful visual features can be distinguished in the video {category label}, require to use no more than 50 words to describe, do not divide the discussion into several points, the answer should be concise and to the point;

[0076] The large language model can generate a reasonable answer that meets the requirements based on the relevant knowledge about the question in its training data;

[0077] S12: Take the obtained factual knowledge as the answer space, generate a category template for the category label of the automatic driving task, and describe the target in detail;

[0078] Specifically, the original category label data is taken as input;

[0079] The large language model interface is called to generate a new category template using the question paradigm;

[0080] S13: Cache the generated category template into an offline file, which is loaded during the downstream model training.

[0081] Specifically, in the process of generating the category template, the template of each category is saved to an offline file in JSON, CSV, or custom format, ensuring that the stored file has a good organizational structure and an easily readable format for subsequent loading and use;

[0082] Each category template is associated with its corresponding category label and contains detailed descriptions and related information;

[0083] During the downstream model training phase, the previously generated and cached category templates need to be loaded;

[0084] During the model training process, relevant information is obtained from the loaded category templates as needed to guide model training and evaluation;

[0085] S14: Expand the category template and strengthen the most core category phrases.

[0086] Specifically, expanding the category template includes category expansion, manual template expansion, and label expansion methods

[0087] Among them, category expansion is to add category label words to the category template generated by the large language model before;

[0088] In one specific embodiment, the original category label is "bus";

[0089] The category template is "On the road, the bus usually has a rectangular body, with multiple wheels and a series of windows. Its roof usually has special signs or advertisements, and the side profile is clear. The length is longer. The bus has different shapes and designs for the front and rear, and the front is usually equipped with headlights. There may be special signs and paint on the body. The roof may have a luggage rack, and the tail has lights, while the license plate is located in front and back of the body.";

[0090] The result of category expansion is "Bus. On the road, the bus usually has a rectangular body, with multiple wheels and a series of windows. Its roof usually has special signs or advertisements, and the side profile is clear. The length is longer. The bus has different shapes and designs for the front and rear, and the front is usually equipped with headlights. There may be special signs and paint on the body. The roof may have a luggage rack, and the tail has lights, while the license plate is located in front and back of the body.";

[0091] Among them, manual template expansion is to add a manual template to the generated category template to optimize the category label, and the manual template is manually expanded;

[0092] In a specific implementation, the result of the manual template expansion is "a rectangular bus. A bus on the road usually has a rectangular body, multiple wheels and a series of windows. Its roof usually has a specific logo or advertisement, and the side profile is clear and long. The bus head shape and tail design are different, the front part is usually equipped with front light, and the body may have specific logo and painting. The roof may be equipped with a luggage rack, the tail is equipped with light, and the license plate is located in front and back of the body.".

[0093] In the formula, the mark expansion is to modify the weight of the category label word through the mark symbol such as "()", "[]", and "{}";

[0094] In a specific implementation, the result of the manual template expansion is "a rectangular bus. A bus on the road usually has a rectangular body, multiple wheels and a series of windows. Its roof usually has a specific logo or advertisement, and the side profile is clear and long. The bus head shape and tail design are different, the front part is usually equipped with front light, and the body may have specific logo and painting. The roof may be equipped with a luggage rack, the tail is equipped with light, and the license plate is located in front and back of the body.".

[0095] S2: According to the second aspect of the embodiment of the present disclosure, a method for pre-processing input data at the visual end is provided, as shown in Figure 4 , which specifically comprises:

[0096] S21: Key frames of the input video sequence are obtained through sparse sampling and dense sampling, which are used as the input of the model visual end.

[0097] The specific process is as follows:

[0098] The cumulative distribution function is defined as:

[0099] Z(x) = ∑y(x)

[0100] Wherein Z(x) is a probability cumulative distribution function, the horizontal axis of the distribution function is set as the time axis, and the vertical axis is set as a probability distribution function of 0-1. The motion cumulative distribution curve is constructed based on the function;

[0101] T segments are sampled from the video data, and one value is randomly selected on each segment, so that T video frames can be obtained;

[0102] The T frames sampled are subjected to a segmentation operation according to the constructed motion cumulative distribution curve, and M frames are sampled in each segment instead of only one frame, so that T x M frame images can be obtained;

[0103] The sampling efficiency is ensured, and the channel dimension is expanded by M times at the same time;

[0104] The vehicle, pedestrian, motion, scene and other information of the input frame are analyzed based on content sampling;

[0105] S22: using a video random data augmentation method, performing image transformation on the key frame obtained after sampling, enhancing the robustness of the model to visual representation;

[0106] The specific process is as follows:

[0107] For each key frame obtained by sampling, image transformation technology is applied to increase the diversity of data;

[0108] As shown in Table 1, different image enhancement random methods are listed, a total of N, and these methods are randomly combined to generate new samples after enhancement as the parameter space;

[0109] Table 1: Data augmentation categories

[0110]

[0111] Set the hyperparameter Q as the enhancement number and P as the enhancement intensity;

[0112] Randomly select Q from N enhancement types in the parameter space each time, and then a total of N Q enhancement strategies are included, and different intensities P can be selected for each strategy;

[0113] The same data augmentation transformation is applied to all frames of the same video each time, ensuring that all frames within the same video are affected in the same way;

[0114] For different videos, randomly selected enhancement transformation strategies are used.

[0115] S3: Construct an automatic driving three-dimensional scene data preprocessing network; the specific content is as follows:

[0116] In a specific embodiment, given a video segment v and a text description c, data preprocessing is performed on v and c.

[0117] After visual preprocessing, we get:

[0118]

[0119] wherein represents the new data after the operations of segment sampling and video random data augmentation, v represents the original data, P μ represents the data processing process;

[0120] After text preprocessing, we get:

[0121]

[0122] wherein, represents the language and character description generated by the large language model for the target category, c represents the original category label, O μ represents the process of generating a new category label by the large language model.

[0123] S31: The preprocessed text data is sent into a text encoder, an image encoder and a point cloud encoder together with the image and point cloud data to obtain text embedding, image pixel features (i.e., image embedding) and point features.

[0124] The specific process is as follows:

[0125] The category template is put into the pre-defined text encoder to generate the text embedding:

[0126]

[0127] wherein is the final generated text embedding representation, is the text encoder;

[0128] The enhanced image data is put into the image encoder for encoding to obtain the image embedding:

[0129]

[0130] wherein is the final generated visual representation, i.e., the image embedding, is the visual encoder;

[0131] The similarity between the image embedding and the text embedding is calculated; the similarity between the image embedding and the text embedding is calculated by using the contrastive loss, the similarity between certain image and its description text is maximized, and the similarity between other image-text pairs is minimized.

[0132] S32: Construct positive samples and negative samples for contrastive learning by using the knowledge of the contrastive language pre-training model, obtain the point cloud-text pair and all image-text pairs in a short time, and learn the semantic correlation between the point cloud or image pixel and the text description.

[0133] S33: Train the contrastive language image pre-training model to obtain the result.

[0134] The model is trained using the semantic segmentation dataset SemanticKITTI to realize three-dimensional semantic segmentation and complete three-dimensional data processing.

[0135] The embodiment of the application also includes an automatic driving three-dimensional scene data preprocessing system based on a large language model, comprising:

[0136] The text input end processing data module can realize the content of step S1.

[0137] a visual input processing data module, which can realize the content of step S2;

[0138] an automatic driving three-dimensional scene data preprocessing network module, which can realize the content of S3.

[0139] The above series of detailed descriptions are only specific descriptions for the feasible implementation modes of the present application, and are not used to limit the protection scope of the present application. Any equivalent mode or change without departing from the present application technology shall be included in the protection scope of the present application.

Claims

1. A large language model-based automatic driving three-dimensional scene data preprocessing method, characterized in that, Comprise three parts: A. In the text input end processing data, according to the prompt learning method design large language model prompt template, applied to the category label, the template describes each target in each scene in detail, so that the label can carry more semantic information; At the same time, design a question paradigm to obtain the optimal description, so as to guide the large language model to generate more accurate and rich text information; In addition, the basic prompt template is expanded, and the category label is added according to the weight to strengthen the key information in the template, so as to further improve the quality and accuracy of the description, and form the processing mode of prompt, generation, cache and expansion at the text end, and enhance the text data; The specific steps are as follows: Step 1: based on large language model, generate a question paradigm for contrastive learning for each category label, stimulate the factual knowledge in the large language model; Step 2: take the factual knowledge as the answer space, and generate detailed category templates for the category labels of the automatic driving task; Step 3: cache the generated category templates to the offline file, and load it during the training of the downstream model; Step 4: expand the category template and strengthen the core category phrase; B. In the visual input end processing data, using video frame fragmentation sampling strategy and video random data enhancement method to preprocess the input frame; The specific steps are as follows: Step 5: obtain the key frames of the input video sequence by sparse sampling and dense sampling, which are used as the input of the visual end of the model; Step 6: use the video random data enhancement method to perform image transformation on the key frames obtained by sampling, and enhance the robustness of the model to visual representation; C. Construct an automatic driving three-dimensional scene data preprocessing network to realize three-dimensional data processing; The specific steps are as follows: Step 7: input the preprocessed text data, visual image data and point cloud data into the text encoder, image encoder and point cloud encoder to obtain text embedding, image embedding and point feature; Step 8: use the contrastive language image pre-training model knowledge to construct positive samples and negative samples for contrastive learning, obtain point cloud-text pairs and all image-text pairs in a short time, and learn the semantic correlation between point cloud or image pixel and text description; Calculate the similarity between image embedding and text embedding; Use contrastive loss to calculate the similarity between image embedding and text embedding, maximize the similarity between image and its corresponding description text, and minimize the similarity between other image-text pairs; Step 9: train the contrastive language pre-training model to realize three-dimensional semantic segmentation and complete the processing of three-dimensional data.

2. The automatic driving three-dimensional scene data preprocessing method based on a large language model according to claim 1, characterized in that, The question paradigm in step 1 is designed as follows: The question paradigm follows the principle of 5W1H, wherein Who represents: what identity the large language model wants to answer the question; What represents: what the large language model wants to do; Where represents: answer in what scene; When represents: what stage the current work has reached; Why represents: why do this; How represents: what way the large language model wants to execute; Call the large language model interface to generate new category templates using the question paradigm.

3. The automatic driving three-dimensional scene data preprocessing method based on a large language model according to claim 1, characterized in that, The cache implementation in step 3 includes: custom format file, save category template, load category template, as follows: In the process of generating category templates, save the template of each category into a JSON, CSV or offline file in a custom format; Each category template is associated with its corresponding category label and contains detailed information; In the downstream model training stage, the previously generated and cached category templates need to be loaded; During model training, information is obtained from the loaded category templates as needed to guide model training and evaluation.

4. The automatic driving three-dimensional scene data preprocessing method based on a large language model according to claim 1, characterized in that, The category template expansion in step 4 includes: category expansion, manual template expansion and label expansion; as follows: Category expansion is performed before adding category labels to the category templates generated by the large language model; Manual template expansion is added before generating category templates to optimize category labels; Label expansion is to modify the weight of certain specific words through markers such as "()", [], and {}.

5. The automatic driving three-dimensional scene data preprocessing method based on a large language model according to claim 1, characterized in that, The key frame acquisition of the input video sequence in step 5 includes: sampling video segments, performing segmentation operation, and extracting frames that are key to understanding the video; as follows: Define the cumulative distribution function Z(x), set the horizontal axis as the time axis and the vertical axis as the probability distribution function of 0-1, and construct the motion cumulative distribution curve based on the function; Sample T segments from the video data, and randomly select a value on each segment to obtain T video frames; Perform segmentation operation on the T frames sampled according to the constructed motion cumulative distribution curve, and sample M frames for each segment to obtain T×M frame images; To ensure sampling efficiency, the channel dimension is also expanded by M times; Based on content sampling, analyze the vehicle, pedestrian, motion, scene and other information of the input frame.

6. The automatic driving three-dimensional scene data preprocessing method based on a large language model according to claim 1, characterized in that, The video random data augmentation method in step 6 includes: randomly combining image enhancement transformations, setting hyperparameters, and using data augmentation operations on videos; as follows: Randomly combine N image enhancement operations such as rotation, translation, scaling, cropping, and color adjustment to generate new samples after enhancement as the parameter space; Set the hyperparameter Q as the number of enhancements and P as the enhancement intensity; Q are chosen randomly from N enhancements in the parameter space each time, then there are N Q total enhancement strategies, each of which can select different enhancement strength P; Apply consistent enhancement transformations to all frames of the same video, and use randomly selected enhancement transformation strategies for different videos.

7. The automatic driving three-dimensional scene data preprocessing method based on a large language model according to claim 1, characterized in that, The implementation of text embedding in step 7 is as follows: Put the category template into the predefined text encoder to generate text embedding: wherein is the final generated text embedding representation, is a text encoder, represents pre-processed text data.

8. The automatic driving three-dimensional scene data preprocessing method based on a large language model according to claim 1, characterized in that, The implementation of image embedding in step 7 is as follows: Put the enhanced image data into the image encoder for encoding to obtain image embedding: wherein is the final generated visual representation, i.e. image embedding, is the image encoder, represents the pre-processed visual image data.

9. The automatic driving three-dimensional scene data preprocessing method based on a large language model according to claim 1, characterized in that, The implementation of point features in step 7 is as follows: putting the original point cloud data p e point into a point cloud encoder to obtain point features wherein point denotes a point cloud dataset, F P denotes a point cloud encoder.

10. An automatic driving three-dimensional scene data preprocessing system based on a large language model, characterized in that, It includes: Text input data processing module, which can realize the contents of steps 1-4 of claim 1; Visual input data processing module, which can realize the contents of steps 5-6 of claim 1; Automatic driving three-dimensional scene data preprocessing network module, which can realize the contents of steps 7-9 of claim 1.

Images