A method and device for constructing traffic scenes using multi-modal data

Abstract

The present invention discloses a method and equipment for constructing traffic scenes with multi-modal data, and relates to the technical field of traffic scene simulation. The method comprises the following steps: Step 1: classifying the training set image data of the scene generation model; Step 2: batch labeling prompts for the training set image data according to the classification; Step 3: using the focused attention method, extracting the labeling prompt keywords through the self-attention mechanism, and extracting the image features through the convolutional neural network; Step 4: after completing the information extraction, inputting the respective results into the multi-head cross attention part to complete the training of the scene generation model; Step 5: collecting multi-modal data similar to the training data and converting it into text information; Step 6: inputting the text information into the scene generator to generate the traffic scene. The present invention proposes a focused attention method to extract the image features in the image data and the keywords in the labeling prompts, and deeply integrate the image and text information.

Description

Technical Field

[0001] The present invention belongs to the technical field of traffic scene simulation, and specifically relates to a method and device for constructing a traffic scene using multi-mode data. Background Art

[0002] In the field of transportation, it is often necessary to analyze traffic data, and when renovating and designing traffic facilities, it is also necessary to make reasonable planning decisions based on the number of pedestrians, roads, and vehicles. In this process, it is often necessary to change and modify the elements of traffic-related facilities to compare the effects and facilitate decision-making analysis. In actual traffic scenarios, it is difficult to change the distribution of pedestrians and vehicles at will. In order to ensure flexibility in the analysis and planning process, it is particularly important to establish virtual traffic scenes for traffic elements such as vehicles, roads, and pedestrians. In related technologies, professional large-scale traffic data analysis auxiliary software is generally used to establish scenes. The interactive method of setting parameters through software is cumbersome manual operation, and the interactivity and flexibility are poor. Existing traffic assistant software mainly uses text results, which cannot meet the needs of generating virtual traffic scenes for simulation analysis. Summary of the invention

[0003] The object of the present invention is to provide a method and device for constructing traffic scenes using multimodal data, so as to solve at least one aspect of the problems and defects raised in the above-mentioned background technology.

[0004] Specifically, a method for constructing a traffic scene using multi-modal data includes the following steps:

[0005] Step 1: Classify the training set image data of the scene generation model;

[0006] Step 2: Batch label the training set image data according to the classification;

[0007] Step 3: Use the focused attention method and the self-attention mechanism to take the annotation prompt information as the query vector Q, convert the corresponding image data into a two-dimensional matrix, and then flatten it by row to form a one-dimensional vector, which is used as the key vector K and the value vector V respectively. Then, perform Dropout on the vectors Q, K, and V respectively, and then obtain the weighted sum S through calculation to extract the keywords; then use the convolutional neural network to extract the local details and overall outline in the image data;

[0008] The formula is as follows:

[0009]

[0010] in:

[0011] Step 4: After completing the information extraction, the respective results are input into the multi-head cross attention part, and several extractions are performed according to the number of keywords. In one extraction process, a keyword is used as vector Q1, and the extracted information of the image data is used as K1 and V1 to establish the image and text information fusion matching and complete the training of the scene generation model;

[0012] Step 5: Collect multimodal data of the same classification as the training data and convert it into text information;

[0013] Step 6: Input the text information into the scene generator to generate traffic scene pictures.

[0014] Furthermore, the classification divides the data into four categories based on basic traffic scenarios: road, road+vehicle, road+person, and road+person+vehicle.

[0015] Furthermore, the self-attention mechanism includes a layer normalization part, a self-attention part and a fully connected part.

[0016] Furthermore, the layer normalization part performs normalization processing on the input data features.

[0017] Furthermore, the self-attention part obtains a weighted value according to the similarity of the data through an inner product operation.

[0018] Furthermore, the fully connected part is responsible for comprehensively adapting the extracted feature dimensions and finally completing the keyword extraction for the input prompt.

[0019] A device for constructing traffic scenes using multi-mode data, comprising: a training server, a multi-mode data acquisition converter, a scene generator, and an input device;

[0020] The training server trains the conversion text model and the scene generation model through the corresponding data sets, and deploys them to the multi-mode data acquisition converter and the scene generator respectively after the training is completed;

[0021] The multi-mode data acquisition converter realizes the conversion from voice data to text data through the input device with the help of the conversion text model, and the text data is directly used as text data after detection and correction; the scene generator receives the text data from the multi-mode data acquisition converter, and completes the generation of text data into scene pictures with the help of the scene generation model.

[0022] Furthermore, the input device includes at least one of a sound receiving device, a keyboard and a touch device.

[0023] Furthermore, the multi-mode data acquisition converter mainly collects the user's voice information into the processor, and uses the conversion text model network to convert the input information into text information. This technology is relatively mature and can be directly deployed. If the input is text information from touch or keyboard, it will be bypassed and only necessary grammar detection and correction will be performed, and it will be directly used as text information as input for the traffic scene generation part.

[0024] Furthermore, in the related art, the method of annotating pictures is to add corresponding annotation tips to each picture. This method is mostly because the original data is not analyzed and processed in an organized manner, resulting in different types of pictures being mixed together, making it difficult to process in batches.

[0025] Furthermore, the present invention proposes a batch annotation prompt method. Before training, the same category of pictures are first grouped and arranged in a path for batch processing, and prompt context information is uniformly added to them, without adding prompt context to each picture one by one. The present invention classifies the pictures in advance and uniformly annotates the prompts in batches, making the process more efficient and organized.

[0026] Compared with the prior art, the present invention has the following beneficial effects:

[0027] 1. The present invention proposes a method of focusing attention, extracting image features in image data and keywords in annotation prompts, deeply integrating image and text information, and facilitating subsequent recognition of text data to directly generate corresponding virtual traffic scenes.

[0028] 2. The present invention proposes a batch labeling prompt method, which performs batch labeling prompts on training data of the same category to facilitate training and learning and improve training efficiency.

[0029] 3. The present invention uses text and voice as input to generate traffic scene diagrams, thereby enhancing the interactivity, flexibility and intelligence in the traffic field. BRIEF DESCRIPTION OF THE DRAWINGS

[0030] In order to facilitate understanding by those skilled in the art, the present invention is further described below with reference to the accompanying drawings.

[0031] Figure 1 A schematic diagram of a method for marking images in related art;

[0032] Figure 2 is a schematic diagram of a batch labeling method of the present invention;

[0033] Figure 3 Flow chart of the method for focusing attention of the present invention. DETAILED DESCRIPTION

[0034] In order to make the purpose, technical solution and advantages of the present invention more clearly understood, the technical solution of the present invention is further specifically described below by way of examples and in conjunction with the accompanying drawings. In the specification, the same or similar reference numerals indicate the same or similar components. The following description of the embodiments of the present invention with reference to the accompanying drawings is intended to explain the overall inventive concept of the present invention and should not be construed as a limitation of the present invention.

[0035] In addition, in the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the disclosed embodiments. However, it is apparent that one or more embodiments may be implemented without these specific details. In other cases, known structures and devices are embodied in a schematic manner to simplify the accompanying drawings.

[0036] A method for constructing a traffic scene using multi-modal data, comprising the following steps:

[0037] Step 1: Classify the training set image data of the scene generation model;

[0038] Step 2: Batch label the training set image data according to the classification;

[0039] Step 3: Use the focused attention method and the self-attention mechanism to take the annotation prompt information as the query vector Q, convert the corresponding image data into a two-dimensional matrix, and then flatten it by row to form a one-dimensional vector, which is used as the key vector K and the value vector V respectively. Then, perform Dropout on the vectors Q, K, and V respectively, and then obtain the weighted sum S through calculation to extract the keywords; then use the convolutional neural network to extract the local details and overall outline in the image data;

[0040] The formula is as follows:

[0041]

[0042] in:

[0043] Step 4: After completing the information extraction, the respective results are input into the multi-head cross attention part, and several extractions are performed according to the number of keywords. In one extraction process, a keyword is used as vector Q1, and the extracted information of the image data is used as K1 and V1 to establish the image and text information fusion matching and complete the training of the scene generation model;

[0044] Step 5: Collect multimodal data of the same classification as the training data and convert it into text information;

[0045] Step 6: Input the text information into the scene generator to generate traffic scene pictures.

[0046] Furthermore, the classification divides the data into four categories based on basic traffic scenarios: road, road + car, road + person, and road + person + car.

[0047] Furthermore, the self-attention mechanism includes a layer normalization part, a self-attention part and a fully connected part.

[0048] Furthermore, the layer normalization part normalizes the input data features.

[0049] Furthermore, the self-attention part obtains a weighted value according to the similarity of the data through an inner product operation.

[0050] Furthermore, the fully connected part is responsible for comprehensively adapting the extracted feature dimensions and finally completing the keyword extraction for the input prompt.

[0051] A device for constructing a traffic scene using multi-mode data, comprising: a training server, a multi-mode data acquisition converter, a scene generator, and an input device;

[0052] The training server trains the conversion text model and the scene generation model through the corresponding data sets, and deploys them to the multi-mode data acquisition converter and the scene generator respectively after the training is completed;

[0053] The multi-mode data acquisition converter realizes the conversion from voice data to text data through the input device with the help of the conversion text model, and the text data is directly used as text data after detection and correction; the scene generator receives the text data from the multi-mode data acquisition converter, and completes the generation of text data into scene pictures with the help of the scene generation model.

[0054] Furthermore, the input device includes at least one of a sound receiving device, a keyboard and a touch device.

[0055] Furthermore, the multi-mode data acquisition converter mainly collects the user's voice information into the processor, and uses the conversion text model network to convert the input information into text information. This technology is relatively mature and can be directly deployed. If the input is text information from touch or keyboard, it will be bypassed and only necessary grammar detection and correction will be performed, and it will be directly used as text information as input for the traffic scene generation part.

[0056] Furthermore, if Figure 1 As shown, the method of annotating pictures in the related art is to add corresponding annotation tips to each picture. This method is mostly because the original data is not analyzed and processed in an organized manner, resulting in different types of pictures being mixed together and difficult to process in batches; and this method is not suitable for pictures with the same elements, that is, Figure 1 Both pictures include "a road" and "a car", but they need to be labeled separately, which is time-consuming and laborious.

[0057] Furthermore, if Figure 2 As shown, the present invention proposes a batch annotation prompt method. Before training, the same category of pictures are first clustered and arranged under a path for batch processing, that is, the pictures are first classified, and prompt context information is uniformly added to them. For example, A in the figure represents a type of picture with only one road, and the uniform annotation prompt "Generate a road"; similarly, the annotation of B-type pictures is "Generate a road and a person"; the annotation of C-type pictures is "Generate a road and a car", without adding prompt context to the pictures one by one. The present invention makes the process more efficient and organized by classifying the pictures in advance and uniformly annotating the prompts in batches.

[0058] Furthermore, in order to closely imitate the oral or written expression habits of humans, the prompt context information in the above process, based on the four major categories of traffic objects, adds the grammatical structure commonly used in daily expression, which can be expanded to "generate {prompt word}", "generate {prompt word 1} and {prompt word 2}" or "generate {prompt word 1}, {prompt word 2} and {prompt word 3}", etc., adding necessary prefixes, conjunctions, quantifiers, etc. During the training process, the number of prompt contexts and traffic objects in the diagram can start from 1, or can be designed specifically according to actual needs.

[0059] Furthermore, if Figure 3 As shown, the present invention proposes a focused attention method, which is based on the use process of the existing diffusion model. Before the annotation prompts and image data enter the multi-head cross attention, the prompt annotation keywords, the local details of the image data and the overall outline are extracted respectively. Then, the multi-head cross attention uses the extracted keywords as Q1 and the extracted image features as K1 and V1 to deeply integrate the image and text information.

[0060] Word embedding processing is a key technology in natural language processing. It refers to embedding a high-dimensional space with a dimension equal to the number of all words into a continuous vector space with a much lower dimension. Each word or phrase is mapped to a vector in the real number field. Simply put, it is the process of converting words or Chinese characters into vectors, that is, converting them into a language that computers can understand so that they can be processed by computers.

[0061] Figure 3 The M in represents that when extracting keywords, the correlation degree and weight distribution calculations are performed on the M different objects in the annotation prompts, and finally the keyword extraction is completed; Figure 3 The N in the above represents the convolution operation, which extracts features of the image data multiple times through N convolutional layers and pooling layers, and finally obtains the global feature representation, which is classified and outputted through the fully connected layer.

[0062] The essential idea of ​​the diffusion model includes forward and backward processes. The forward process is to continuously add standard Gaussian noise to the input image to gradually build a reference space. When the time step is infinite, the data eventually becomes pure noise. The backward process is the inverse of the forward process. Through cross-attention, the relationship between the annotation prompt and the current image is continuously learned and mined, and the noisy image is gradually denoised and restored until the target image is generated.

[0063] Furthermore, the self-attention processing module includes layer normalization, self-attention and full connection parts. Among them, the layer normalization processing part is to normalize the input features of each sample to prevent gradient explosion or gradient disappearance; the self-attention part performs Dropout screening on the Q, K, and V vectors to obtain D(Q), D(K), and D(V) respectively. This is to reduce the degree of fit, enhance the generalization ability of the model, and improve the randomness of the generation. Then, the inner product operation is performed on D(Q) and D(K). The size of the inner product is the size of the similarity. According to the similarity of D(Q) and D(K), the weighted value of the V vector corresponding to the K vector is obtained, and then the weighted sum S is obtained. The formula is as follows:

[0064] ；

[0065] The essence of this process is to determine the attention weight allocation by calculating the similarity between the input vectors. The feedforward fully connected part is responsible for comprehensively adapting the extracted feature dimensions and finally completing the keyword extraction for the input prompt.

[0066] Furthermore, the present invention uses a convolutional neural network to extract features from image data. Considering that the image is not easy to process when converted into a matrix with a large size, it is first divided into several small square blocks and flattened, similar to vector processing. The image features are extracted through convolution kernels of different sizes and contents.

[0067] The convolution kernels used mainly include edge detection convolution kernels, channel expansion convolution kernels, size change convolution kernels and other feature extraction convolution kernels. After each feature extraction is completed, the pooling layer is followed to realize image data compression and aggregation. Finally, the dimensional features learned by convolution are comprehensively mapped through the fully connected part to achieve output.

[0068] Furthermore, after completing the annotation prompt keyword extraction and target image feature extraction, the respective results are input into the multi-head cross attention part. Compared with ordinary cross attention, it can improve the process orientation and accuracy of image and text information fusion matching and increase the orderliness of training.

[0069] Furthermore, if it is necessary to change the traffic image information, such as changing the lane or vehicle information, add the image data set and annotation prompt information that need to be adjusted based on the original training, and perform corresponding fine-tuning according to the above steps.

[0070] The specific embodiments described above further illustrate the objectives, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for constructing traffic scenes using multi-modal data, characterized in that: The steps include: Step 1: Classify the training set image data of the scene generation model; Step 2: Batch label the training set image data according to the classification; Step 3: Convert the annotation prompt information into query vector Q, key vector K and value vector V through word embedding. Use the focused attention method and self-attention mechanism to perform Dropout filtering and self-attention processing on vectors Q, K and V respectively. Then, obtain the weighted value S through calculation and extract the keywords. The formula is as follows: ， in: ; ； ； Step 4: Extract local details and overall contours from the image data through a convolutional neural network, including: dividing the image data corresponding to the annotation prompt information into several small square blocks and flattening them to form a one-dimensional vector, extracting image features through convolution kernels of different sizes and contents, and following up with a pooling layer to compress and aggregate the image data after each feature extraction, and finally performing comprehensive mapping through the fully connected part to achieve output; Step 5: After completing the information extraction, the keywords and the corresponding image data extracted information are input into the multi-head cross attention part, and several extractions are performed according to the number of keywords. In one extraction process, a keyword is used as vector Q1, and the extracted information of the image data is used as K1 and V1 to establish the image and text information fusion matching and complete the training of the scene generation model; Step 6: Collect multimodal data of the same category as the training data and convert it into text information; Step 7: Input the text information into the scene generator to generate traffic scene images.

2. The method for constructing traffic scenes using multimodal data according to claim 1, characterized in that: The classification divides the data into four categories based on basic traffic scenarios: road, road + car, road + person, and road + person + car.

3. The method for constructing traffic scenes using multimodal data according to claim 1, characterized in that: The self-attention mechanism includes a layer normalization part, a self-attention part and a fully connected part.

4. The method for constructing traffic scenes using multi-modal data according to claim 3, characterized in that: The layer normalization part performs normalization processing on input data features.

5. The method for constructing traffic scenes using multi-modal data according to claim 3, characterized in that: The self-attention part obtains a weighted value according to the similarity of the data through an inner product operation.

6. The method for constructing traffic scenes using multi-modal data according to claim 3, characterized in that: The fully connected part is responsible for comprehensive adaptation of the extracted feature dimensions, and finally completes the keyword extraction for the input prompt.

7. A device for constructing traffic scenes using multi-modal data, characterized in that: include: Training server, multi-mode data acquisition converter, scenario generator, input device; The training server trains the conversion text model and the scene generation model through the corresponding data sets, and deploys them to the multi-mode data acquisition converter and the scene generator respectively after the training is completed; The multi-mode data acquisition converter realizes the conversion from voice data to text data through the input device with the help of the conversion text model, and the text data is directly used as text data after detection and correction; the scene generator receives the text data from the multi-mode data acquisition converter and completes the generation of text data into scene pictures with the help of the scene generation model; The training process of the scene generation model is as follows: Step 1: Classify the training set image data of the scene generation model; Step 2: Batch label the training set image data according to the classification; Step 3: Convert the annotation prompt information into query vector Q, key vector K and value vector V through word embedding. Use the focused attention method and self-attention mechanism to perform Dropout filtering and self-attention processing on vectors Q, K and V respectively. Then, obtain the weighted value S through calculation and extract the keywords. The formula is as follows: ， in: ; ； ； Step 4: Extract local details and overall contours from the image data through a convolutional neural network, including: dividing the image data corresponding to the annotation prompt information into several small square blocks and flattening them to form a one-dimensional vector, extracting image features through convolution kernels of different sizes and contents, and following up with a pooling layer to compress and aggregate the image data after each feature extraction, and finally performing comprehensive mapping through the fully connected part to achieve output; Step 5: After completing the information extraction, the keywords and the corresponding image data extracted information are input into the multi-head cross attention part, and several extractions are performed according to the number of keywords. In one extraction process, a keyword is used as vector Q1, and the extracted information of the image data is used as K1 and V1 to establish a fusion match of image and text information to complete the training of the scene generation model.

8. The device for constructing traffic scenes using multi-modal data according to claim 7, characterized in that: The input device includes at least one of a sound receiving device, a keyboard and a touch device.

Images