Automatic driving scene semantic prediction method
By performing secondary processing on sensor data and employing a combination of word embedding, local and global semantic units, the problems of missing and erroneous sensor data were solved, enabling more accurate semantic prediction for autonomous driving scenarios and improving the system's reliability and application scope.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- SOUTH CHINA UNIV OF TECH
- Filing Date
- 2023-05-15
- Publication Date
- 2026-05-12
AI Technical Summary
In existing semantic prediction for autonomous driving scenarios, sensor data may be missing, erroneous, or conflicting, leading to inaccurate semantic analysis and vulnerability to malicious tampering, which affects the judgment of driving scenarios.
By performing secondary processing on sensor data, a combination of word embedding units, local semantic convolution units, global semantic recurrent units, and semantic fusion units is used to extract and fuse semantic features, including word vector expansion, extraction and fusion of local and global semantic information, and bidirectional gated recurrent units and multi-head attention mechanisms to improve prediction accuracy.
It improves the accuracy and reliability of sensor data, enhances the precision of semantic prediction, expands the application scope of sensor data, and supports environmental perception of edge servers.
Smart Images
Figure CN116738357B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the technical field of autonomous driving scene prediction and semantic analysis, and in particular to a method for semantic prediction of autonomous driving scenes. Background Technology
[0002] Semantic prediction in autonomous driving scenarios refers to using multiple different sensors on an autonomous vehicle to capture information about the surrounding environment, processing different data types, and generating semantic perception of the vehicle's real-time driving environment.
[0003] Currently, environmental semantic analysis primarily focuses on image recognition. This involves manually annotating video and images captured by sensors to create semantically labeled data, which serves as evaluation metrics. This algorithm then evaluates other video and image data using the same annotations. However, sensor information is not always accurate. Misinterpretations can occur due to sensor errors, omissions, or data loss during transmission. Furthermore, malicious modifications and tampering of information are possible. Sensor occlusion caused by varying vehicle density can also lead to incomplete information acquisition, affecting the semantic analysis system and resulting in incorrect semantic judgments in driving scenarios. Summary of the Invention
[0004] The purpose of this invention is to overcome the shortcomings and deficiencies of the prior art and propose a semantic prediction method for autonomous driving scenarios. By performing secondary processing on sensor data in autonomous driving scenarios, it can solve the problems of missing, error and conflict between sensor data from different sources to a certain extent, and predict the semantics of the scenario more accurately.
[0005] To achieve the above objectives, the technical solution provided by this invention is: a semantic prediction method for autonomous driving scenarios, comprising the following steps:
[0006] 1) Embed the semantic input words of the vehicle sensor into the unit and expand the word vectors of the semantically rich context words;
[0007] 2) The output of the word embedding unit is used as the input of the local semantic convolution unit. The local semantic convolution unit is used to extract local semantic information to obtain the semantic feature combination between objects.
[0008] 3) The output of the word embedding unit is used as the input of the global semantic recurrent unit. The global semantic recurrent unit is used to extract global semantic information to obtain all semantic information features in both positive and negative directions.
[0009] 4) The outputs of the word embedding unit, the local semantic convolution unit, and the global semantic recurrent unit are used as the inputs of the semantic fusion unit. The semantic fusion unit is used to perform feature fusion to achieve semantic prediction in autonomous driving scenarios.
[0010] Furthermore, in step 1), the word embedding unit is responsible for expanding the semantically rich contextual vocabulary with word vectors to solve the problems of unstructured and uncomputable text data;
[0011] The word embedding unit is passed to the input layer through one-hot encoded word vectors, where C is the number of words in the context, V is the dimension of the word vector space equal to the size of the vocabulary, and N is the feature value size. A weight matrix W1 with dimension V×N and a weight matrix W2 with dimension N×V are set. The one-hot encoded word vectors are multiplied by W1 respectively, and the average of the results is calculated to obtain a hidden layer with dimension 1×N. The hidden layer is multiplied by W2 to obtain an output matrix of 1×V. The neuron output value is converted into probability through the softmax function.
[0012] Furthermore, in step 2), the local semantic convolution unit uses three different sizes of convolution kernels k∈[2,3,4] to convolve the word vectors respectively, that is, to extract the semantics between 2, 3, and 4 neighboring words, and obtain feature semantics of different lengths;
[0013] By using convolution kernels of different lengths to process different combinations of neighboring words in a word order, the scope of semantic association between neighboring words is expanded, increasing the richness of semantics; after convolution, n is obtained. k matrix X c ∈R (H-k+1)×C 'where n k R represents the number of convolution kernels, H represents the shape of the matrix, C' represents the sentence length, and k represents the number of output channels.
[0014] Furthermore, in step 3), the semantics of the sensor are analyzed globally using the local features collected by the local semantic convolution unit in step 2), and a global semantic recurrent unit is introduced. The global semantic recurrent unit is implemented based on a bidirectional gated recurrent unit neural network, and the word vector space is input into the global semantic recurrent unit according to the word order to obtain semantic information reaching different hidden layers.
[0015] The global semantic loop unit uses a bidirectional gated loop unit to perform preliminary extraction of global semantics, and then uses the output of the word embedding unit. As input, it is passed to the bidirectional gated loop unit to obtain the output. Where C E C is the number of output channels of the word embedding unit. h This indicates the size of the forward and reverse hidden layers in a bidirectional gated loop unit;
[0016] The global semantic recurrent unit incorporates the results of all hidden layers to obtain features of different semantic sequence lengths, and presents them to the attention unit for computation; the output of the bidirectional gated recurrent unit... As input to the multi-head attention mechanism, the proportion of semantic combination features in the global information is calculated through the multi-head attention mechanism. The maximum value represents the current global semantic features, and the final output is...
[0017] Further, in step 4), the semantic fusion unit fuses the semantic features of the outputs of the word embedding unit, the local semantic convolution unit, and the global semantic recurrent unit to output the classification result, including the following steps:
[0018] 4.1) Output of the word embedding unit and the output of the bidirectional gated loop unit Intermediate results were obtained by fusing data along the channel dimension. The output is obtained after passing through the ReLU activation function and max pooling.
[0019] 4.2) Combine the output of step 4.1) with the output of the local semantic convolution unit. By concatenating the vectors, we obtain the feature vectors. Where n k It is the number of convolution kernels, C f It is the number of output channels of the local semantic convolutional unit;
[0020] 4.3) The obtained feature vectors are passed through a feedforward neural network to output all features as classification results.
[0021] Compared with the prior art, the present invention has the following advantages and beneficial effects:
[0022] 1. This invention addresses the problem of inaccurate sensor information in existing systems. By performing secondary processing on sensor data in autonomous driving scenarios, it can resolve potential gaps, errors, and conflicts between sensor data from different sources, and more accurately predict scene semantics.
[0023] 2. Unlike other methods that only use image recognition for semantics, this invention can be extended to access data from other sensors to perform semantic prediction, thereby improving the accuracy and reliability of semantic prediction.
[0024] 3. This invention can be extended by combining crowdsourcing technology to transmit the semantic data collected by bicycles to edge servers, forming a range environment awareness in various areas, which has broad application prospects. Attached Figure Description
[0025] Figure 1This is a schematic diagram of the logic flow of the present invention; in the figure, ATTU represents the attention unit, BiGRU represents the bidirectional gated recurrent unit, MaxPooling represents the max pooling unit, and Convolution Layer represents the convolutional layer.
[0026] Figure 2 This is a schematic diagram of the word embedding unit used in this invention.
[0027] Figure 3 This is a schematic diagram of the semantic fusion unit (SFU) used in this invention; in the diagram, GSRU represents the global semantic loop unit, WEU represents the word embedding unit, and LSCU represents the local semantic loop unit. Detailed Implementation
[0028] The present invention will be further described in detail below with reference to the embodiments and accompanying drawings, but the embodiments of the present invention are not limited thereto.
[0029] like Figure 1 As shown, this embodiment discloses a semantic prediction method for autonomous driving scenarios, including the following steps:
[0030] 1) Embed the semantic input words of the vehicle sensor into the unit and expand the word vectors of the semantically rich context words;
[0031] The word embedding unit is mainly responsible for expanding the semantically rich context vocabulary with word vectors, thus solving the problems of unstructured and uncomputable text data.
[0032] like Figure 2 As shown, the word embedding unit passes one-hot encoded word vectors to the input layer, where C is the number of words in the context, V is the dimension of the word vector space equal to the size of the vocabulary, and N is the feature value size. A weight matrix W1 with dimension V×N and a weight matrix W2 with dimension N×V are set. The one-hot encoded word vectors are multiplied by W1 respectively, and the average of the results is calculated to obtain a hidden layer with dimension 1×N. The hidden layer is multiplied by W2 to obtain an output matrix of 1×V. The softmax function is used to convert the neuron output value into a probability.
[0033] 2) The output of the word embedding unit is used as the input of the local semantic convolution unit. The local semantic convolution unit is used to extract local semantic information to obtain the semantic feature combination between objects.
[0034] The local semantic convolutional unit uses three different sizes of convolution kernels k∈[2,3,4] to convolve the word vectors respectively, that is, to extract the semantics between 2, 3 and 4 neighboring words, and obtain feature semantics of different lengths;
[0035] By using convolution kernels of different lengths to process different combinations of neighboring words in a word order, the scope of semantic association between neighboring words is expanded, increasing the richness of semantics; after convolution, n is obtained. k Matrix X c ∈R (H-k×1)×C ', where n k R represents the number of convolution kernels, H represents the shape of the matrix, C' represents the sentence length, and k represents the number of output channels.
[0036] 3) The output of the word embedding unit is used as the input of the global semantic recurrent unit. The global semantic recurrent unit is used to extract global semantic information to obtain all semantic information features in both positive and negative directions, as follows:
[0037] Using the local features collected by the local semantic convolutional unit in step 2), the semantics of the sensor are analyzed globally, and a global semantic recurrent unit is introduced. The global semantic recurrent unit is implemented based on a bidirectional gated recurrent unit neural network. The word vector space is input into the global semantic recurrent unit according to the word order to obtain semantic information reaching different hidden layers.
[0038] The global semantic loop unit uses a bidirectional gated loop unit to perform preliminary extraction of global semantics, and then uses the output of the word embedding unit. As input, it is passed to the bidirectional gated loop unit to obtain the output. Where C E C is the number of output channels of the word embedding unit. h This indicates the size of the forward and reverse hidden layers in a bidirectional gated loop unit;
[0039] Traditional recurrent neural networks (RNNs) only consider the output of the last hidden layer in terms of temporal sequence. Although bidirectional gated recurrent units (BGUs) account for the influence of long-term memory, they cannot obtain more subtle combined features. Therefore, global semantic recurrent units (GSIs) incorporate the results of all hidden layers to obtain features of different semantic sequence lengths and present them to the attention unit for computation. The output of the bidirectional gated recurrent unit... As input to the multi-head attention mechanism, the proportion of semantic combination features in the global information is calculated through the multi-head attention mechanism. The maximum value represents the current global semantic features, and the final output is...
[0040] 4) The outputs of the word embedding unit, the local semantic convolution unit, and the global semantic recurrent unit are used as inputs to the semantic fusion unit. The semantic fusion unit performs feature fusion and outputs classification results to achieve semantic prediction in autonomous driving scenarios, such as... Figure 3 As shown, it includes the following steps:
[0041] 4.1) Output of the word embedding unit and the output of the bidirectional gated loop unit Intermediate results were obtained by fusing data along the channel dimension. The output is obtained after passing through the ReLU activation function and max pooling.
[0042] 4.2) Combine the output of step 4.1) with the output of the local semantic convolution unit. By concatenating the vectors, we obtain the feature vectors. Where n k It is the number of convolution kernels, C f It is the number of output channels of the local semantic convolutional unit;
[0043] 4.3) The obtained feature vectors are passed through a feedforward neural network to output all features as classification results.
[0044] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.
Claims
1. A semantic prediction method for autonomous driving scenarios, characterized in that, Includes the following steps: 1) Embed the semantic input words of the vehicle sensor into the unit and expand the word vectors of the semantically rich context words; The word embedding unit is responsible for expanding the semantically rich context vocabulary with word vectors, thus solving the problems of unstructured and uncomputable text data. The word embedding unit passes the one-hot encoded word vectors to the input layer, where V is the dimension of the word vector space, which is equivalent to the size of the vocabulary, and N is the size of the feature values. A weight matrix W1 with dimension V×N and a weight matrix W2 with dimension N×V are set. The one-hot encoded word vectors are multiplied by W1 respectively, and the average of the results is calculated to obtain a hidden layer with dimension 1×N. The hidden layer is multiplied by W2 to obtain an output matrix of 1×V, and the neuron output values are converted into probabilities through the softmax function. 2) The output of the word embedding unit is used as the input of the local semantic convolution unit. The local semantic convolution unit is used to extract local semantic information to obtain the semantic feature combination between objects. The local semantic convolutional unit uses three different sizes of convolution kernels k∈[2, 3, 4] to convolve the word vectors respectively, that is, to extract the semantics between 2, 3, and 4 neighboring words, and obtain feature semantics of different lengths; By using convolution kernels of different lengths to combine different neighboring words in a word order, the scope of semantic combination is expanded, and the semantic richness is increased; after convolution, n is obtained. k Matrix: R represents the shape of the matrix, H represents the sentence length, and C represents the length of the sentence. ’ This represents the number of output channels, and k represents the kernel size. 3) The output of the word embedding unit is used as the input of the global semantic recurrent unit. The global semantic recurrent unit is used to extract global semantic information to obtain all semantic information features in both positive and negative directions. Using the local features collected by the local semantic convolutional unit in step 2), the semantics of the sensor are analyzed globally, and a global semantic recurrent unit is introduced. The global semantic recurrent unit is implemented based on a bidirectional gated recurrent unit neural network. The word vector space is input into the global semantic recurrent unit according to the word order to obtain the semantic information of different hidden layers. The global semantic loop unit uses a bidirectional gated loop unit to perform preliminary extraction of global semantics, and then uses the output of the word embedding unit. As input, it is passed to the bidirectional gated loop unit to obtain the output. C E C is the number of output channels of the word embedding unit. h This indicates the size of the forward and reverse hidden layers in a bidirectional gated loop unit; The global semantic recurrent unit incorporates the results of all hidden layers to obtain features of different semantic sequence lengths, and presents them to the attention unit for computation; the output of the bidirectional gated recurrent unit... As input to the multi-head attention mechanism, the proportion of semantic combination features in the global information is calculated through the multi-head attention mechanism. The maximum value represents the current global semantic features, and the final output is... ; 4) The outputs of the word embedding unit, the local semantic convolution unit, and the global semantic recurrent unit are used as the inputs of the semantic fusion unit. The semantic fusion unit is used to perform feature fusion to achieve semantic prediction in autonomous driving scenarios.
2. The semantic prediction method for autonomous driving scenarios according to claim 1, characterized in that: In step 4), the semantic fusion unit fuses the semantic features of the word embedding unit, the local semantic convolution unit, and the global semantic recurrent unit to output the classification result, including the following steps: 4.1) Output of the word embedding unit and the output of the global semantic loop unit Intermediate results were obtained by fusing data along the channel dimension. The output is then obtained after passing through the ReLU activation function and max pooling. ; 4.2) Combine the output of step 4.1) with the output of the local semantic convolution unit. By concatenating the vectors, we obtain the feature vectors. , where n k It is the number of convolution kernels, C f It is the number of output channels of the local semantic convolutional unit; 4.3) Input the obtained feature vector into a feedforward neural network and output all features as classification results.