Automatic driving scene classification and recognition system and method

Abstract

The invention discloses an automatic driving scene classification and recognition system and method, and relates to the field of automatic driving, and the system comprises a data acquisition module which collects an original image and point cloud data of an automatic driving scene based on a multi-source device; the feature extraction module projects the point cloud data to a two-dimensional plane by using a point cloud projection method to generate a two-dimensional image, and performs feature extraction on the two-dimensional image and the original image to obtain feature vectors; the scene classification module inputs the feature vectors into a multi-layer perceptron for automatic driving scene classification to obtain a preliminary classification probability of the automatic driving scene; and the fusion module is used for fusing the preliminary classification probability of the automatic driving scene with map information and global positioning information to obtain the classification probability of the automatic driving scene, and determining the category of the automatic driving scene according to the classification probability of the automatic driving scene. According to the invention, the accuracy of scene classification is improved.

Description

Technical Field

[0001] The present application relates to the field of autonomous driving technology, and in particular to an autonomous driving scene classification and recognition system and method. Background Art

[0002] Autonomous driving technology refers to the use of various sensors, computer vision, artificial intelligence and other advanced technologies to enable vehicles to complete driving tasks without human intervention. The wire-controlled steering and positioning assisted driving system, as one of its components, mainly provides automated steering assistance functions through the perception and analysis of vehicle driving trajectory and environmental information. The system usually uses a variety of sensors (such as cameras, lidar, millimeter-wave radar, etc.) to perceive the surrounding environment, and uses high-precision map data for positioning and navigation. Through the processing and analysis of sensor data, combined with intelligent algorithms, the system can identify road signs, traffic lights, vehicles, pedestrians and other targets, and make steering decisions based on this information to assist the driver in completing steering operations. However, the existing driving system has poor environmental perception performance during driving, resulting in the inability to accurately identify autonomous driving scenarios during driving. Summary of the invention

[0003] The purpose of this application is to provide an autonomous driving scene classification and recognition system and method, which can enhance environmental perception capabilities and improve the accuracy of scene classification by acquiring comprehensive environmental information.

[0004] To achieve the above objectives, this application provides the following solutions:

[0005] In a first aspect, the present application provides an autonomous driving scene classification and recognition system, comprising:

[0006] A data acquisition module, which is used to collect raw images and point cloud data of autonomous driving scenes based on multi-source devices;

[0007] A feature extraction module is connected to the data acquisition module, projects the point cloud data onto a two-dimensional plane using a point cloud projection method to generate a two-dimensional image, and performs feature extraction on the two-dimensional image and the original image to obtain a feature vector;

[0008] A scene classification module, connected to the feature extraction module, for inputting the feature vector into a multi-layer perceptron to classify the autonomous driving scene, and obtaining a preliminary classification probability of the autonomous driving scene;

[0009] A fusion module is connected to the scene classification module and is used to fuse the preliminary classification probability of the autonomous driving scene with the map information and the global positioning information to obtain the classification probability of the autonomous driving scene, and determine the autonomous driving scene category according to the classification probability of the autonomous driving scene.

[0010] In a second aspect, the present application provides a method for classifying and identifying autonomous driving scenes, comprising:

[0011] Collect raw images and point cloud data of autonomous driving scenes based on multi-source devices;

[0012] Projecting the point cloud data onto a two-dimensional plane using a point cloud projection method to generate a two-dimensional image, and performing feature extraction on the two-dimensional image and the original image to obtain a feature vector;

[0013] Inputting the feature vector into a multi-layer perceptron to classify the autonomous driving scene, and obtaining a preliminary classification probability of the autonomous driving scene;

[0014] The preliminary classification probability of the autonomous driving scenario is integrated with the map information and the global positioning information to obtain the classification probability of the autonomous driving scenario, and the autonomous driving scenario category is determined according to the classification probability of the autonomous driving scenario.

[0015] According to the specific embodiments provided in this application, this application has the following technical effects:

[0016] The present application provides an autonomous driving scene classification and recognition system and method, which obtains more comprehensive environmental information through multi-source devices, further performs feature extraction, and combines a multi-layer perceptron to accurately classify autonomous driving scenes. Furthermore, the classification results are integrated with map information and global positioning information to enhance environmental perception capabilities, enabling it to accurately identify scenes over a wider range, thereby improving the accuracy of scene classification. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 This is a schematic diagram of the structure of an autonomous driving scene classification and recognition system in this application.

[0019] Figure 2 This is a flowchart of an autonomous driving scene classification and recognition method in this application.

[0020] Reference numerals:

[0021] Data acquisition module-1, feature extraction module-2, scene classification module-3, fusion module-4, update module-5. DETAILED DESCRIPTION

[0022] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0023] In order to make the above-mentioned objects, features and advantages of the present application more obvious and easy to understand, the present application is further described in detail below in conjunction with the accompanying drawings and specific implementation methods.

[0024] In an exemplary embodiment, Figure 1 As shown, a system for classifying and identifying an autonomous driving scene is provided, and the system for classifying and identifying an autonomous driving scene includes:

[0025] The data acquisition module 1 is used to collect original images and point cloud data of the autonomous driving scene based on multi-source devices, including cameras, laser radars, millimeter wave radars, etc.

[0026] The feature extraction module 2 is connected to the data acquisition module 1 and is used to project the point cloud data onto a two-dimensional plane using a point cloud projection method to generate a two-dimensional image, and perform feature extraction on the two-dimensional image and the original image to obtain a feature vector. The feature extraction module 2 performs feature extraction on the two-dimensional image and the original image using a convolutional neural network, an edge detection method, or a corner detection method to obtain a feature vector.

[0027] The scene classification module 3 is connected to the feature extraction module 2 and is used to input the feature vector into a multi-layer perceptron to classify the autonomous driving scene and obtain a preliminary classification probability of the autonomous driving scene.

[0028] The fusion module 4 is connected to the scene classification module 3 and is used to fuse the preliminary classification probability of the autonomous driving scene with the map information and the global positioning information to obtain the classification probability of the autonomous driving scene, and determine the autonomous driving scene category according to the classification probability of the autonomous driving scene. The fusion module uses a nonlinear activation function to fuse the preliminary classification probability of the autonomous driving scene with the map information and the global positioning information to obtain the classification probability of the autonomous driving scene.

[0029] In another exemplary embodiment of the present application, the autonomous driving scene classification and recognition system further includes: a preprocessing module connected to the data acquisition module 1 and the feature extraction module 2, for performing standardization processing on the original image and the point cloud data to obtain the processed original image and the processed point cloud data, and send them to the feature extraction module 2. In the data preprocessing, the collected original image and point cloud data are denoised and standardized, and the processed original image and point cloud data are processed using the standardization processing module. To adjust the data, we get the denoised and standardized image data and point cloud data, where p is the original image and point cloud data, q is the denoised and standardized image data and point cloud data, μ is the mean, and σ is the standard deviation. The mean of the denoised and standardized image data and point cloud data becomes 0, and the standard deviation becomes 1.

[0030] In another exemplary embodiment of the present application, the feature extraction module 2 specifically includes: a two-dimensional image generation submodule, connected to the data acquisition module 1, and used to project the point cloud data onto a two-dimensional plane using a point cloud projection method to generate a two-dimensional image.

[0031] The feature map generation submodule is connected to the two-dimensional image generation submodule and is used to extract features from the two-dimensional image and the original image using a convolution operation to generate a feature map.

[0032] The feature vector acquisition submodule is connected to the feature map generation submodule and is used to flatten the feature map to obtain the feature vector.

[0033] In another exemplary embodiment of the present application, the convolution operation is specifically:

[0034]

[0035] Among them, m and n are the indices of the convolution kernel, i and j are the row and column indices of the two-dimensional image and the original image, and c i,j is the pixel value of the i-th row and j-th column in the feature map, B i,j is the pixel value of the i-th row and j-th column in the two-dimensional image and the original image, θ is the parameter of feature extraction module 2, B i+m,j+n is the pixel value of the i+mth row and j+nth column in the two-dimensional image and the original image, K m,n is the weight value of the mth row and nth column in the convolution kernel K, and b is the bias term.

[0036] Specifically, assume that the size of the two-dimensional image and the original image B is H×W, where H is the height of the two-dimensional image and the original image B, and W is the width of the two-dimensional image and the original image B. The convolution kernel K is an odd matrix, for example: the convolution kernel K is a 3*3 or 5*5 matrix, which is used to scan the two-dimensional image and the original image B and extract features. Assume that the size of the convolution kernel is kh ×k w , where k h is the height of the convolution kernel, k w is the width of the convolution kernel. The specific convolution operation is: for each position (i, j) in the two-dimensional image and the original image B, where i is the i-th row in the two-dimensional image and the original image B, and j is the j-th column in the two-dimensional image and the original image B, use the convolution kernel K to perform convolution operation at this position and its neighborhood:

[0037]

[0038] The range of m and n ensures the sliding of the convolution kernel on the two-dimensional image and the original image B, and b is the bias term. After the convolution operation, a nonlinear activation function is applied: c' i,j =σ(c i,j ), where c' i,j is the feature map. The feature map is obtained after the two-dimensional image and the original image B undergo convolution operation and activation function. The role of the activation function is to introduce nonlinearity so that the model can fit more complex functions. The size of the feature map depends on the size of the two-dimensional image and the original image B, the size of the convolution kernel, and the step size and padding.

[0039] In another exemplary embodiment of the present application, the scene classification module 3 uses a machine learning algorithm to train and optimize the feature vector, specifically through the algorithm D=g(C;φ) to train, and obtain the preliminary classification probability D of the autonomous driving scene, where g is the learning algorithm, φ is the algorithm parameter, and C is the feature vector. For example: the autonomous driving scene can be an urban road scene, a highway scene, and a rural road scene. D=[0.1, 0.8, 0.1], indicating that the probability of urban roads is 0.1, the probability of highways is 0.8, and the probability of rural roads is 0.1. The largest probability value in the vector result is selected to determine the scene, so as to make decisions in subsequent operations, such as executing an operating speed limit of 30km / h corresponding to the rainy day scene.

[0040] Specifically, a multilayer perceptron is used as the learning algorithm g. During forward propagation, the feature vector C is gradually transformed into the preliminary classification probability D of the autonomous driving scene through the linear transformation and nonlinear activation function of each layer. During back propagation, the gradient is calculated according to the loss function, and the algorithm parameters are adjusted through the gradient descent optimization algorithm; the algorithm parameters φ include but are not limited to the weight matrix W, the bias vector b, the activation function f, the loss function L and the optimizer parameters. For the weight matrix W, the weight matrix W of each layer is used to linearly transform the input features. For the weight matrix of the lth layer, it is expressed as W (l) For the bias vector b, the bias vector b of each layer is used to adjust the offset of the activation function. For the lth layer, the bias vector is represented by b (l). For the activation function f, the activation function f of each layer is used to introduce nonlinear characteristics. For the loss function L, it is used to evaluate the difference between the model prediction output and the true label. For the optimizer parameters, it is used to update and optimize the algorithm parameters, including the learning rate η and the momentum μ.

[0041] The specific steps are as follows: First, initialize the parameters. Initialize the weight matrix W and bias vector b of each layer, using random initialization or predefined initialization methods, such as Xavier / Glorot initialization and He initialization. Second, perform forward propagation. For the input feature vector C, forward propagate through each layer of the multilayer perceptron. Calculation formula: h (l+1) =f(W (l) h (l) +b (l) ), where l is the lth layer of the neural network. When l = 0, h (l) is the output of the lth layer of the neural network, h (0) is the input feature vector. Then calculate the loss. Calculate the loss L between the preliminary classification probability D of the autonomous driving scene and the true label. For classification tasks, the loss function is the cross entropy loss. Finally, perform back propagation. Calculate the gradient of the loss function with respect to the parameters of each layer; use the chain rule to calculate the gradient of each layer, and update the weight matrix W and the bias vector b. The scene classification module 3 is also used for regression tasks. At this time, the scene classification module 3 outputs a scalar representing the predicted value.

[0042] In another exemplary embodiment of the present application, the fusion module 4 uses a fusion function to fuse the preliminary classification probability of the autonomous driving scene with the map information and the global positioning information to obtain the classification probability of the autonomous driving scene.

[0043] The fusion function is F = h(D, E; ψ). Where D is the preliminary classification probability of the autonomous driving scene, E is the map information and global positioning information, and ψ is the parameter of the fusion function, including weight parameters ψ1, ψ2 and bias term b. For example: E = [37.7749, -122.4194, 120, cos(0°), sin(0°), 0, 1, 0, 0] represents the latitude and longitude coordinates of the current location, the driving speed is 120 km / h, the direction is due north, and the surrounding environment features are no construction area and there are traffic signs.

[0044] The fusion function is specifically a nonlinear activation function, and the expression of the nonlinear activation function is F=σ(ψ 1 D+ψ 2E+b). Among them, F is the classification probability of the autonomous driving scenario, σ is a nonlinear activation function used to introduce nonlinear characteristics, ψ1 and ψ2 are weight parameters, indicating the relative importance of D and E in the final decision, and b is a bias term used to adjust the offset of the activation function.

[0045] At the beginning, the weight parameters ψ1 and ψ2 and the bias term b are set by random initialization or predefined methods. The weight parameters are initialized using Xavier / Glorot initialization or He initialization methods. The bias term b can be initialized to zero or a small constant. During the training process, labeled training data needs to be prepared. The training data contains the preliminary classification probability D of the autonomous driving scene, map information and global positioning information E, and the corresponding true value label. A loss function L is defined to evaluate the difference between the classification probability F of the autonomous driving scene and the true value label. Use an optimization algorithm (gradient descent) to minimize the loss function, calculate the gradient through back propagation, and update the weight parameters ψ1, ψ2 and the bias term b.

[0046] The specific steps are as follows: First, initialize ψ1 and ψ2 to small random values ​​or use Xavier / Glorot initialization, and initialize b to zero or a small constant. Second, perform forward propagation. For each training sample, calculate the classification probability F of the autonomous driving scene. Furthermore, use the loss function L to calculate the difference between the classification probability F of the autonomous driving scene and the true value label. Finally, calculate the gradient of the loss function with respect to ψ1, ψ2, and b: Among them, η is the learning rate, which controls the step size of parameter update. Repeat the above steps for iterative optimization until the loss function converges or reaches the predetermined number of training rounds.

[0047] For example, assume that D and E are both vectors of length 10, ψ1 = 0.1, ψ2 = 0.1, and b = 0. During the training process, by continuously adjusting ψ1, ψ2, and b, the classification probability F of the autonomous driving scene is made closer to the true value label, thereby improving the performance of the algorithm.

[0048] In another exemplary embodiment of the present application, the autonomous driving scene classification and recognition system further includes: an updating module 5, which is connected to the feature extraction module 2 and the fusion module 4, and is used to update the parameters of the feature extraction module 2 and the fusion module 4 based on the original image and point cloud data collected in real time by using a stochastic gradient descent method.

[0049] The specific calculation formula of the stochastic gradient descent method is:

[0050]

[0051] Among them, θ' is the updated parameter of feature extraction module 2, φ' is the updated parameter of fusion module 4, θ is the parameter of feature extraction module 2, φ is the parameter of fusion module 4, G is the original image and point cloud data collected in real time, L(θ,φ;G) is the loss function, is the parameter gradient of the loss function with respect to feature extraction module 2, is the parameter gradient of the loss function with respect to fusion module 4, λ θ is the parameter learning rate of feature extraction module 2, λ φ is the parameter learning rate of fusion module 4.

[0052] This application collects data by integrating multiple source devices such as cameras, lidars and millimeter-wave radars to obtain more comprehensive environmental information, thereby improving the accuracy and reliability of scene classification. At the same time, the integration of map information and global positioning information further enhances the system's environmental perception capabilities, enabling it to accurately identify scenes in a wider range. Convolutional Neural Networks (CNN) are used for feature extraction and combined with machine learning algorithms for optimization. It not only efficiently processes large amounts of data, but also ensures accurate identification of complex and changeable autonomous driving scenarios. The raw data is processed by denoising and standardizing to improve data quality, providing high-quality input for deep learning models. At the same time, additional data and parameter adjustments in the data fusion and enhancement stages further improve the accuracy of autonomous driving scene classification.

[0053] Based on the same inventive concept, the present application embodiment also provides a method for classifying and identifying autonomous driving scenes. Figure 2 As shown, the autonomous driving scene classification and recognition method includes the following steps 101 to 104.

[0054] Step 101: Collect original images and point cloud data of the autonomous driving scene based on multi-source devices.

[0055] Step 102: Project the point cloud data onto a two-dimensional plane using a point cloud projection method to generate a two-dimensional image, and perform feature extraction on the two-dimensional image and the original image to obtain a feature vector.

[0056] Step 103: Input the feature vector into a multi-layer perceptron to classify the autonomous driving scene, and obtain a preliminary classification probability of the autonomous driving scene.

[0057] Step 104: The preliminary classification probability of the autonomous driving scenario is integrated with the map information and the global positioning information to obtain the classification probability of the autonomous driving scenario, and the autonomous driving scenario category is determined according to the classification probability of the autonomous driving scenario.

[0058] As an optional implementation, the autonomous driving scene classification and recognition method further includes:

[0059] The original image and the point cloud data are standardized to obtain a processed original image and processed point cloud data.

[0060] The technical features of the above embodiments may be combined arbitrarily. To make the description concise, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0061] This article uses specific examples to illustrate the principles and implementation methods of this application. The description of the above embodiments is only used to help understand the method and core ideas of this application. At the same time, for those skilled in the art, according to the ideas of this application, there will be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting this application.

Claims

1. An autonomous driving scene classification and recognition system, characterized in that: The autonomous driving scene classification and recognition system comprises: A data acquisition module, which is used to collect raw images and point cloud data of autonomous driving scenes based on multi-source devices; A feature extraction module, connected to the data acquisition module, is used to project the point cloud data onto a two-dimensional plane using a point cloud projection method to generate a two-dimensional image, and perform feature extraction on the two-dimensional image and the original image to obtain a feature vector; A scene classification module, connected to the feature extraction module, for inputting the feature vector into a multi-layer perceptron to classify the autonomous driving scene, and obtaining a preliminary classification probability of the autonomous driving scene; A fusion module is connected to the scene classification module and is used to fuse the preliminary classification probability of the autonomous driving scene with the map information and the global positioning information to obtain the classification probability of the autonomous driving scene, and determine the autonomous driving scene category according to the classification probability of the autonomous driving scene.

2. The automatic driving scene classification and recognition system according to claim 1, characterized in that: The autonomous driving scene classification and recognition system further includes: The preprocessing module is connected to the data acquisition module and the feature extraction module, and is used to perform standardization processing on the original image and the point cloud data to obtain the processed original image and the processed point cloud data, and send them to the feature extraction module.

3. The automatic driving scene classification and recognition system according to claim 1, characterized in that: The feature extraction module extracts features from the two-dimensional image and the original image through a convolutional neural network, an edge detection method or a corner detection method to obtain a feature vector.

4. The automatic driving scene classification and recognition system according to claim 1, characterized in that: The fusion module uses a nonlinear activation function to fuse the preliminary classification probability of the autonomous driving scene with the map information and the global positioning information to obtain the classification probability of the autonomous driving scene.

5. The automatic driving scene classification and recognition system according to claim 1, characterized in that: The feature extraction module comprises: A two-dimensional image generation submodule, connected to the data acquisition module, for projecting the point cloud data onto a two-dimensional plane using a point cloud projection method to generate a two-dimensional image; A feature map generation submodule, connected to the two-dimensional image generation submodule, for extracting features from the two-dimensional image and the original image using a convolution operation to generate a feature map; The feature vector acquisition submodule is connected to the feature map generation submodule and is used to flatten the feature map to obtain the feature vector.

6. The automatic driving scene classification and recognition system according to claim 5, characterized in that: The convolution operation is specifically as follows: Among them, m and n are the indices of the convolution kernel, i and j are the row and column indices of the two-dimensional image and the original image, and c i,j is the pixel value of the i-th row and j-th column in the feature map, B i,j is the pixel value of the i-th row and j-th column in the two-dimensional image and the original image, θ is the parameter of the feature extraction module, B i+m,j+n is the pixel value of the i+mth row and j+nth column in the two-dimensional image and the original image, K m,n is the weight value of the mth row and nth column in the convolution kernel K, and b is the bias term.

7. The automatic driving scene classification and recognition system according to claim 1, characterized in that: The autonomous driving scene classification and recognition system further includes: The updating module is connected to the feature extraction module and the fusion module, and is used to update the parameters of the feature extraction module and the fusion module by using a stochastic gradient descent method based on the original image and point cloud data collected in real time.

8. The automatic driving scene classification and recognition system according to claim 7, characterized in that: The specific calculation formula of the stochastic gradient descent method is: Among them, θ' is the parameter of the updated feature extraction module, φ' is the parameter of the updated fusion module, θ is the parameter of the feature extraction module, φ is the parameter of the fusion module, G is the original image and point cloud data collected in real time, L(θ,φ;G) is the loss function, is the parameter gradient of the loss function with respect to the feature extraction module, is the parameter gradient of the loss function with respect to the fusion module, λ θ is the parameter learning rate of the feature extraction module, λ φ is the parameter learning rate of the fusion module.

9. An autonomous driving scene classification and recognition method, applied to the autonomous driving scene classification and recognition system according to any one of claims 1 to 8, characterized in that: The automatic driving scene classification and recognition method comprises: Collect raw images and point cloud data of autonomous driving scenes based on multi-source devices; Projecting the point cloud data onto a two-dimensional plane using a point cloud projection method to generate a two-dimensional image, and performing feature extraction on the two-dimensional image and the original image to obtain a feature vector; Inputting the feature vector into a multi-layer perceptron to classify the autonomous driving scene, and obtaining a preliminary classification probability of the autonomous driving scene; The preliminary classification probability of the autonomous driving scenario is integrated with the map information and the global positioning information to obtain the classification probability of the autonomous driving scenario, and the autonomous driving scenario category is determined according to the classification probability of the autonomous driving scenario.

10. The method for classifying and identifying autonomous driving scenes according to claim 9, characterized in that: The automatic driving scene classification and recognition method further includes: The original image and the point cloud data are standardized to obtain a processed original image and processed point cloud data.

Images