Automatic driving obstacle avoidance system with hierarchical accurate identification

Abstract

The invention relates to an automatic driving obstacle avoidance system with hierarchical accurate recognition, and belongs to the technical field of automatic driving. The system comprises a data acquisition module which comprises a sensor and a preprocessing module and is used for acquiring driving environment data of an automatic driving vehicle and data of the vehicle by using the sensor and preprocessing the data; the image processing module is used for performing hierarchical semantic segmentation on each frame of streetscape image to obtain a semantic segmentation result of a key target in each frame of streetscape image; the data fusion module is used for receiving the laser radar point cloud data and the image with the semantic segmentation result obtained by the image processing module, performing data fusion to obtain comprehensive perception data, and perceiving the environment around the autonomous vehicle; the decision module is used for receiving the comprehensive sensing data, making an obstacle avoidance strategy and generating a decision instruction; and the control module is used for receiving the decision instruction to control the autonomous vehicle so as to realize obstacle avoidance. And rapid and accurate automatic driving obstacle avoidance is realized.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to an autonomous driving obstacle avoidance system with hierarchical precise identification. Background Art

[0002] As the market demand for active safety and intelligence of automobiles continues to increase, the social and economic value of autonomous driving has become increasingly prominent. Autonomous driving technology has become an important topic in the development of cutting-edge science and technology today, and has had a profound impact on many fields such as scientific and technological development and social and economic development.

[0003] The autonomous driving obstacle avoidance system is a system that can help autonomous vehicles perceive and avoid obstacles. It uses various sensors, algorithms and decision-making systems to monitor and analyze the environment around the vehicle, and automatically controls the vehicle's steering, acceleration and braking to avoid potential obstacles. The goal of the system is to ensure that the vehicle can safely avoid collisions with other vehicles, pedestrians, buildings, animals or other obstacles during driving. Existing autonomous driving obstacle avoidance systems usually independently predict the category of each pixel during the semantic recognition process, ignoring the correlation between pixels, resulting in similar but semantically different objects being misclassified. As a result, the vehicle makes wrong decisions during the obstacle avoidance process, posing a serious threat to the safety and reliability of autonomous vehicles. Summary of the invention

[0004] In view of the above analysis, an embodiment of the present invention aims to provide an autonomous driving obstacle avoidance system with hierarchical and precise identification, so as to achieve fast and accurate autonomous driving obstacle avoidance.

[0005] The purpose of the present invention is mainly achieved through the following technical solutions:

[0006] The present invention provides an automatic driving obstacle avoidance system with hierarchical and precise identification, the automatic driving obstacle avoidance system comprising: a data acquisition module, an image processing module, a data fusion module, a decision module and a control module;

[0007] The data acquisition module includes a sensor and a preprocessing module, which is used to use the sensor to obtain the driving environment data and the vehicle's own data of the autonomous driving vehicle and perform preprocessing; wherein the driving environment data includes: laser radar point cloud data and continuous frame street view images taken during driving; the vehicle's own data includes the vehicle's current position and speed;

[0008] The image processing module is used to perform hierarchical semantic segmentation on each frame of street view image to obtain semantic segmentation results of key targets in each frame of street view image;

[0009] The data fusion module is used to receive the laser radar point cloud data and the image with semantic segmentation results obtained by the image processing module, perform data fusion to obtain comprehensive perception data, and perceive the environment around the autonomous driving vehicle;

[0010] The decision module is used to receive the comprehensive perception data, make obstacle avoidance strategies, and generate decision instructions;

[0011] The control module is used to receive the decision instruction to control the autonomous driving vehicle to achieve obstacle avoidance.

[0012] Furthermore, the data fusion module includes a camera radar calibration module, a feature extraction module and a feature fusion module;

[0013] The camera radar calibration module uses a calibration plate to obtain the camera's intrinsic parameters and the extrinsic parameters relative to the laser radar, and aligns the coordinates of the camera and the laser radar;

[0014] The feature extraction module is used to extract the features of the laser radar point cloud and the features of the image with semantic segmentation results respectively;

[0015] The feature fusion module is used to fuse the extracted laser radar point cloud features with the features of the image to obtain comprehensive perception data with distance information and semantic segmentation results.

[0016] Furthermore, the decision module includes: a state module, an action space module, a reward function module and a reinforcement learning module;

[0017] The state module is used to convert the comprehensive perception data into a state representation required for driving decision-making, including vehicle position, speed, and surrounding environment information;

[0018] The action space module is used to define feasible driving actions based on the driving decision model, including acceleration, deceleration, and steering, and construct an action space;

[0019] The reward function module is used to design a reward function to guide the optimization of driving decisions, including punishing collisions and encouraging smooth driving;

[0020] The reinforcement learning module is used to train the reinforcement learning algorithm and optimize the driving decision model so that the driving decision model can select the optimal driving action under different conditions.

[0021] Furthermore, the image processing module uses a trained semantic segmentation network based on hierarchical perception to perform semantic segmentation;

[0022] The semantic segmentation network based on hierarchical perception includes: a feature extraction layer and a segmentation layer;

[0023] The feature extraction layer is used to extract feature information of each frame of street view image;

[0024] The segmentation layer is used to perform pixel-level classification on each frame image based on the feature information to obtain a semantic segmentation result of a key target in each frame image.

[0025] Furthermore, the semantic segmentation network based on hierarchical perception is trained using the following method:

[0026] Using a street view semantic segmentation dataset as a training sample; the street view semantic segmentation dataset includes a street view image set and a corresponding label set;

[0027] Constructing a label tree of street view semantics based on the hierarchical relationship of each label in the label set;

[0028] Based on the street view semantic segmentation dataset and the label tree of the street view semantics, a semantic segmentation network and a semantic mapping network are trained simultaneously; wherein the semantic mapping network includes a feature extraction layer and a mapping layer shared with the semantic segmentation network, and is used to reorganize the features of the input image to obtain a new feature vector; during training, based on the similarity information of the labels in the label tree of the street view semantics, new feature vectors of similar categories are close to each other in the feature space, and new feature vectors of dissimilar categories are far away from each other.

[0029] Furthermore, the segmentation layer includes a first convolution layer, an activation function, a second convolution layer and a sigmoid function, which is used to perform semantic segmentation on the input image based on the number of nodes in the label tree of the street view semantics; wherein,

[0030] The size of the first convolutional layer is 1×1×|V|, and is used to perform a convolution operation on the feature map to obtain transformed feature information; wherein |V| represents the number of nodes in the label tree of the street view semantics;

[0031] The activation function is a ReLU activation function, which is used to perform a nonlinear transformation on each feature to obtain feature information after the nonlinear transformation;

[0032] The size of the second convolutional layer is 1×1×|V|, and is used to perform a convolution operation on the feature information after the nonlinear transformation again;

[0033] The sigmoid function is used to perform a pixel-by-pixel classification probability estimation operation on the output features of the second convolutional layer to obtain the semantic probability distribution of each pixel.

[0034] Furthermore, the step of constructing a label tree of street view semantics based on the hierarchical relationship of each label includes:

[0035] For each tag, find its parent class in the WordNet dataset, and search up layer by layer until the root node of the WordNet dataset is reached to obtain the hierarchical relationship of each tag;

[0036] Based on the hierarchical relationship of each label, a label tree T = {V, E} of street view semantics is constructed; where V represents the set of nodes in the label tree, which is the label class and its parent class in the WordNet dataset; E represents the node relationship of the label tree.

[0037] Furthermore, the simultaneously training of a semantic segmentation network and a semantic mapping network based on the street view semantic segmentation dataset and the street view semantic label tree includes:

[0038] The street view semantic segmentation dataset and the label tree of the street view semantics are loaded, the semantic segmentation network and the semantic mapping network are trained simultaneously using the tree maximum and minimum loss function and the tree triplet loss function, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are updated using gradient back propagation, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are saved after the training is completed, and the semantic segmentation network is obtained based on the trained feature extraction layer and the segmentation layer.

[0039] Furthermore, the tree maximum and minimum loss function is expressed as:

[0040]

[0041] Wherein, B represents the number of images in the street view semantic segmentation dataset; M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree of the street view semantics; Indicates the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; Represents the category probability value of the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image after the tree maximum and minimum transformation.

[0042] Furthermore, the tree triplet loss function is expressed as:

[0043]

[0044] Among them, D TT Represents a valid label triple vector set; N b Represents the number of triplets in the valid label triple vector set; Represents the first eigenvector in the sth valid label triple vector set; Represents the second eigenvector in the sth valid label triple vector set; represents the third eigenvector in the sth valid label triple vector set; d() represents the vector distance function; ms Representing the optimal label interval of the sth valid label triple vector set; optionally selecting three new feature vectors output by the mapping layer and the corresponding labels to form a triple vector set; traversing the triple vector set, and obtaining a valid label triple vector set based on the label tree.

[0045] Compared with the prior art, the present invention can achieve at least one of the following beneficial effects:

[0046] 1. The autonomous driving obstacle avoidance system of the present invention integrates the LiDAR point cloud data and the semantic segmentation results of the street view image through a data fusion module. The LiDAR provides accurate distance and shape information, while the image semantic segmentation provides object category information. The combination of the two can improve the perception accuracy of the surrounding environment and provide a more comprehensive environmental perception capability.

[0047] 2. The decision module of the automatic driving obstacle avoidance system of the present invention utilizes a reinforcement learning algorithm to optimize the driving decision model, and is able to select the optimal driving action under different conditions, thereby improving the efficiency and effectiveness of decision-making.

[0048] 3. The image processing module of the autonomous driving obstacle avoidance system of the present invention can achieve accurate and rapid semantic recognition and segmentation, and the system can respond quickly, thereby improving the safety of the autonomous driving vehicle during the obstacle avoidance process. Through the hierarchical perception semantic segmentation network, the system can process complex street scene images and adapt to different driving environments and conditions.

[0049] 4. The semantic segmentation network used in the image processing module of the autonomous driving obstacle avoidance system of the present invention can effectively integrate the hierarchical structure of labels, including hierarchical dependency and hierarchical similarity, based on the constructed label tree during the training process by introducing the tree maximum and minimum loss function and the tree triplet loss function. This fusion helps to further improve the accuracy of segmentation, so that the network can more accurately identify and segment different objects and scenes in the image.

[0050] In the present invention, the above-mentioned technical solutions can also be combined with each other to achieve more preferred combination solutions. Other features and advantages of the present invention will be described in the subsequent description, and some advantages can become obvious from the description, or can be understood by practicing the present invention. The purpose and other advantages of the present invention can be realized and obtained through the contents particularly pointed out in the description and the drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0051] The drawings are only for the purpose of illustrating specific embodiments and are not to be considered limiting of the present invention. Like reference symbols denote like components throughout the drawings.

[0052] Figure 1It is a structural schematic diagram of an automatic driving obstacle avoidance system with hierarchical precise identification in an embodiment of the present invention;

[0053] Figure 2 Schematic diagram of the structure of the semantic segmentation network in an embodiment of the present invention. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present invention are described in detail below in conjunction with the accompanying drawings, wherein the accompanying drawings constitute a part of this application and are used together with the embodiments of the present invention to illustrate the principles of the present invention, but are not used to limit the scope of the present invention.

[0055] A specific embodiment of the present invention discloses a hierarchical and precise identification automatic driving obstacle avoidance system, such as Figure 1 As shown, the automatic driving obstacle avoidance system includes: a data acquisition module, an image processing module, a data fusion module, a decision module and a control module;

[0056] The data acquisition module includes a sensor and a preprocessing module, which is used to use the sensor to obtain the driving environment data and the vehicle's own data of the autonomous driving vehicle and perform preprocessing; wherein the driving environment data includes: lidar point cloud data and continuous frame street view images taken during driving; the vehicle's own data includes the vehicle's current position and speed.

[0057] Specifically, the sensor includes a laser radar and a camera installed on the autonomous driving vehicle, which are used to obtain the environmental conditions around the autonomous driving vehicle during driving.

[0058] The laser radar can measure the distance to surrounding objects by emitting laser pulses and measuring the time for reflection, providing accurate distance information of surrounding objects. At the same time, the laser radar can also generate three-dimensional laser radar point cloud data of the surrounding environment for understanding the spatial structure around the vehicle.

[0059] The camera can capture high-resolution continuous-frame street view semantic images of the autonomous driving vehicle during driving, and is used to identify key targets in the autonomous driving process, such as road signs, traffic signals, pedestrians, and vehicles. By performing semantic segmentation on the street view semantic images, each pixel in the image is identified and classified as to which object it belongs to.

[0060] The sensors also include a global positioning system and a speed sensor installed on the autonomous driving vehicle, which are used to obtain the position and speed of the autonomous driving vehicle during driving.

[0061] The global positioning system can provide accurate position and time information of the vehicle around the world.

[0062] The speed sensor is installed on the wheel or drive shaft of the autonomous driving vehicle and is used to measure the real-time speed of the vehicle.

[0063] Specifically, the laser radar point cloud is preprocessed, including using a filtering algorithm to remove noise points in the point cloud. The noise in the point cloud will reduce the accuracy and reliability of the data. Removing noise points by using a filtering algorithm can improve the quality of the radar point cloud data. The filtering algorithm can use median filtering or Gaussian filtering, which is not specified in this embodiment.

[0064] For the continuous frames of street view images taken during the driving process, after each frame of the street view image in the video image is collected using OpenCV, a filter is used to remove noise in the image, improve image quality and enhance features. The filter can use Gaussian filtering, median filtering or bilateral filtering, which is not specified in this embodiment.

[0065] Furthermore, the image processing module is used to perform hierarchical semantic segmentation on each frame of street view image to obtain semantic segmentation results of key targets in each frame of street view image.

[0066] Specifically, the street view image is semantically segmented to identify various targets in the street view image, such as vehicles, pedestrians, traffic signs, road boundaries, etc. The system can understand which category each pixel in the image belongs to, thereby understanding the surrounding environment at a semantic level and providing the necessary environmental information for the obstacle avoidance system. Accurate semantic segmentation can improve the autonomous driving vehicle's perception of the surrounding environment, thereby improving the safety of autonomous driving. Therefore, accurate semantic segmentation is crucial to the autonomous driving obstacle avoidance system.

[0067] Furthermore, the image processing module uses a trained semantic segmentation network based on hierarchical perception to perform semantic segmentation; the semantic segmentation network based on hierarchical perception, such as Figure 2 As shown, it includes: feature extraction layer and segmentation layer.

[0068] The feature extraction layer is used to extract feature information of each frame of street view image.

[0069] Specifically, the feature extraction layer of the semantic segmentation network uses an existing segmentation network, removing its last classification layer, to extract features of the input image. Exemplarily, the existing segmentation network can use DeepLab, PSPNet, UNet, etc., which is not limited in this embodiment.

[0070] The segmentation layer is used to perform pixel-level classification on each frame image based on the feature information to obtain a semantic segmentation result of a key target in each frame image.

[0071] Furthermore, the segmentation layer includes a first convolution layer, an activation function, a second convolution layer and a sigmoid function, which is used to perform semantic segmentation on the input image based on the number of nodes in the label tree of the street view semantics; wherein,

[0072] The size of the first convolutional layer is 1×1×|V|, which is used to perform a convolution operation on the feature map to obtain transformed feature information; wherein |V| represents the number of nodes in the label tree of the street view semantics.

[0073] Specifically, the label tree of street view semantics is that when the semantic segmentation network is trained, a street view semantic segmentation dataset is used as a training sample; the street view semantic segmentation dataset includes a street view image set and a corresponding label set; and the label tree of street view semantics is constructed based on the hierarchical relationship of each label in the label set.

[0074] Furthermore, the step of constructing a label tree of street view semantics based on the hierarchical relationship of each label includes:

[0075] For each tag, find its parent class in the WordNet dataset, and search up layer by layer until the root node of the WordNet dataset is reached to obtain the hierarchical relationship of each tag;

[0076] Based on the hierarchical relationship of each tag, a tag tree T = {V, E} is constructed; where V represents the node set of the tag tree, which is the tag class and its parent class in the WordNet dataset; E represents the node relationship of the tag tree.

[0077] Specifically, the label tree is a tree-like hierarchical structure used to represent the hierarchical relationship between categories. Each node represents a category, and the edge represents the parent-child relationship between categories. The WordNet dataset is a large-scale English vocabulary database, in which words are organized in the form of synonym sets, which are connected by semantic relationships (such as parent class, child class, etc.).

[0078] In this embodiment, for several labels corresponding to each image in the image set, each label represents a category, and its parent category and the parent category of the parent category are searched in the WordNet dataset until the root node of the WordNet dataset is reached. The category of the label and all parent nodes up to the root node are used as nodes of the label tree. For example, in a city street view picture, "vehicle" can be a parent node, and "car", "truck" and "motorcycle" can be child nodes.

[0079] It should be noted that the 1×1 convolution layer of the first convolution layer is used to fuse the feature information of different layers and compress the feature dimension at the same time. The high-dimensional feature map obtained from the feature extraction layer is compressed to the same dimension as the number of label tree nodes, and the multi-channel feature map can be converted into a feature map of |V| channels, each channel corresponding to a label tree node.

[0080] Furthermore, the activation function of the segmentation layer is a ReLU activation function, which is used to perform a nonlinear transformation on each feature to obtain feature information after the nonlinear transformation.

[0081] Specifically, the ReLU activation function introduces nonlinear transformation by setting all negative values ​​to 0 and keeping positive values ​​unchanged, enabling the network to learn more complex feature representations and patterns.

[0082] Furthermore, the size of the second convolutional layer of the segmentation layer is 1×1×|V|, which is used to perform a convolution operation on the feature information after the nonlinear transformation again.

[0083] Specifically, the convolutional layer that passes through again can adjust the feature weights, strengthen the features that are useful for the segmentation task, and suppress irrelevant features.

[0084] Furthermore, the sigmoid function of the segmentation layer is used to perform a pixel-by-pixel classification probability estimation operation on the output features of the second convolutional layer to obtain the semantic probability distribution of each pixel.

[0085] Specifically, the sigmoid function is finally used to obtain the probability distribution of each pixel for |V| categories.

[0086] It should be noted that the segmentation layer can be expressed as f seg (I)∈R M×N×|V| , where M represents the height of the image; N represents the width of the image; and |V| represents the number of nodes in the label tree. The segmentation layer receives the output of the feature extraction layer, and classifies the output of the feature extraction layer at the pixel level according to the category nodes defined in the label tree to predict the category of each pixel in the image.

[0087] Furthermore, the semantic segmentation network based on hierarchical perception is trained using the following method:

[0088] A street view semantic segmentation dataset is used as a training sample; the street view semantic segmentation dataset includes a street view image set and a corresponding label set.

[0089] Specifically, the training sample image set described in this embodiment is a street scene semantic segmentation dataset Cityscapes. This dataset is specifically for urban street scenes, consisting of street scenes of 50 different cities, including 5,000 finely annotated images and 20,000 roughly annotated images, and uses 19 commonly used categories to evaluate segmentation accuracy, such as roads, sidewalks, buildings, vehicles, etc.

[0090] A label tree of street view semantics is constructed based on the hierarchical relationship of each label in the label set.

[0091] Based on the street view semantic segmentation dataset and the label tree of the street view semantics, a semantic segmentation network and a semantic mapping network are trained simultaneously; wherein the semantic mapping network includes a feature extraction layer and a mapping layer shared with the semantic segmentation network, and is used to reorganize the features of the input image to obtain a new feature vector; during training, based on the similarity information of the labels in the label tree of the street view semantics, new feature vectors of similar categories are close to each other in the feature space, and new feature vectors of dissimilar categories are far away from each other.

[0092] Specifically, in the semantic mapping network, through the feature extraction layer shared by the semantic mapping network and the semantic segmentation network, the semantic mapping network and the semantic segmentation network can share features extracted from the input image to improve the generalization ability of the model, and at the same time help to improve the expressive ability of the features during the training stage.

[0093] The mapping layer of the semantic mapping network includes a first mapping convolution layer, a mapping activation function, a second mapping convolution layer and a sigmoid function, which are used to map the feature information in the image to a new space; wherein,

[0094] The size of the first mapping convolution layer is 1×1×32, and is used to perform a convolution operation on the feature map to obtain transformed feature information.

[0095] The mapping activation function is a ReLU activation function, which is used to perform a nonlinear transformation on each feature to obtain feature information after the nonlinear transformation.

[0096] The size of the second mapping convolution layer is also 1×1×32, which is used to adjust the feature weights.

[0097] The sigmoid function is used to map the output features of the second mapping convolution layer to a new space to obtain a new feature vector.

[0098] It should be noted that the mapping layer is represented by f proj (I)∈R M×N×32, the features extracted by the feature extraction layer are reorganized into a new space of 32 channels to obtain a new feature vector. During the training process, a loss function is designed based on the distance of the new feature vector in the feature space and the distance of the annotated label in the label tree. The designed loss function fuses the similarity information of the labels by penalizing the categories with close but dissimilar distances of the new feature vectors in the feature space, as well as the categories with far but similar distances. By fusing the similarity information of the labels, the feature extraction layer can extract more discriminative features, providing a better basis for the segmentation layer to perform segmentation.

[0099] Furthermore, the simultaneously training of a semantic segmentation network and a semantic mapping network based on the street view semantic segmentation dataset and the street view semantic label tree includes:

[0100] The street view semantic segmentation dataset and the label tree of the street view semantics are loaded, the semantic segmentation network and the semantic mapping network are trained simultaneously using the tree maximum and minimum loss function and the tree triplet loss function, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are updated using gradient back propagation, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are saved after the training is completed, and the semantic segmentation network is obtained based on the trained feature extraction layer and the segmentation layer.

[0101] Specifically, the semantic segmentation network and the semantic mapping network are based on the total loss function: L total =L TMM +L TT Perform simultaneous training; where L TMM is the maximum and minimum loss function of the tree; L TT is the tree triplet loss function. The two networks are optimized as a whole at the same time, which influences each other and learns collaboratively during the training process, while reducing the risk of overfitting and increasing the robustness of the model.

[0102] Furthermore, the tree maximum and minimum loss function is:

[0103]

[0104] Wherein, B represents the number of images in the training sample image set; M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree; Indicates the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; Represents the category probability value of the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image after the tree maximum and minimum transformation.

[0105] Specifically, the semantic segmentation network obtains the difference between the predicted category probability value and its true label through the tree maximum and minimum loss function. In the process of minimizing the loss function, the segmentation result of the semantic segmentation network is made more consistent with the hierarchical structure defined in the label tree.

[0106] Furthermore, the following formula is used to perform the tree maximum and minimum transformation to obtain the category probability value of each pixel after transformation:

[0107]

[0108] in, A represents the transformed category probability value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; k represents the parent node set of category k in the label tree; D k Represents the set of child nodes of category k in the label tree; represents the (i,j)th pixel in the bth image. The category probability value corresponding to each class; Indicates category The label mask of .

[0109] Specifically, when performing semantic segmentation, the segmentation layer will output a probability distribution for each pixel, indicating the possibility that the pixel belongs to each category. Through the hierarchical structure between the category labels on the label tree, when a pixel in the segmentation layer is predicted to be a subcategory, the prediction probability of its parent category should also be high. For example, if the model predicts that a pixel belongs to the subcategory of "car", then it should also predict that this pixel is likely to belong to its parent category "vehicle". The tree maximum and minimum transformation is used to ensure that the hierarchical structure of the output of the network segmentation layer is consistent with the hierarchical structure defined in the label tree, that is, the output value of the subcategory will not exceed the output value of its parent category. This transformation ensures that when performing pixel-level classification, the output value of the segmentation layer conforms to the hierarchical relationship between labels, which is conducive to improving the logical consistency and accuracy of segmentation.

[0110] Furthermore, the tree triplet loss function

[0111]

[0112] Among them, D TT Represents a valid label triple vector set; N b Represents the number of triplets in the valid label triple vector set; Represents the first eigenvector in the sth valid label triple vector set; Represents the second eigenvector in the sth valid label triple vector set; represents the third eigenvector in the sth valid label triple vector set; d() represents the vector distance function; m s Representing the optimal label interval of the sth valid label triple vector set; optionally selecting three new feature vectors output by the mapping layer and the corresponding labels to form a triple vector set;

[0113] The triple vector set is traversed, and a valid label triple vector set is obtained based on the label tree.

[0114] Specifically, the tree triplet loss function measures the hierarchical similarity between labels by calculating the distance between feature vectors. By minimizing the tree triplet loss function, the feature vectors of similar categories in the features extracted by the feature extraction layer are closer, and the feature vectors of dissimilar categories are farther apart, which is beneficial for the segmentation layer to perform semantic segmentation more accurately.

[0115] The triplet vector set is all feature vectors mapped to the new space for the output of the mapping layer Take out three different feature vectors and their corresponding labels at random to form a triple vector set, recorded as Among them, N t Denotes the triple vector set D T The number of triple vectors in ; represents the first eigenvector in the rth triple vector set; represents the second eigenvector in the rth triple vector set; represents the third eigenvector in the rth triplet vector set; Represents the label corresponding to the first eigenvector in the rth triple vector set; Represents the label corresponding to the second eigenvector in the rth triple vector set; Represents the label corresponding to the third eigenvector in the rth triple vector set.

[0116] Further, the triple vector set is traversed to obtain a valid label triple vector set based on the label tree, including: the shortest path length between the label of the first feature vector and the label of the second feature vector in the triple vector set in the label tree is less than the shortest path length between the label of the first feature vector and the label of the third feature vector in the label tree.

[0117] Specifically, the valid label triple vector is a label of three feature vectors in the triple vector based on their positions in the label tree, satisfying the following conditions:

[0118]

[0119] Wherein, Ψ() represents the shortest path length between two label nodes on the label tree.

[0120] In the label tree, the shortest path length between two nodes is considered as a measure of similarity in the hierarchy. The shortest path length indicates that the two categories are more similar. During the training process, constructing a valid label triple vector set can enable the entire model to more accurately distinguish between visually similar but semantically different categories, which can help the model capture the differences and similarities between categories and improve the segmentation accuracy of the semantic segmentation network.

[0121] The set of triple vectors that meet the above conditions constitutes a valid label triple vector set: Among them, N b Represents the number of triplets in the set of valid label triple vectors.

[0122] Specifically, for the first eigenvector, the second eigenvector, and the third eigenvector in the valid label triple vector set, the distances between the vectors are calculated using the following formulas:

[0123]

[0124] The formula is a variant of the cosine similarity formula, and the result is that the distance value ranges from 0 to 1, where 0 means that the vectors are exactly the same and 1 means that the vectors are completely opposite.

[0125] Furthermore, the optimal label interval of the valid label triple vector set is calculated using the following formula:

[0126]

[0127] in, Represents the label of the first eigenvector in the sth valid label triplet vector set; Represents the label of the third eigenvector in the sth valid label triplet vector set; represents the label of the second eigenvector in the sth valid label triple vector set; Ψ() represents the shortest path length between two labels in the label tree.

[0128] Specifically, the optimal label interval is to quantify the hierarchical similarity between labels and use the quantified similarity to optimize the performance of the model when training the model. s When is a positive value, it means the label and tags The distance is greater than the label and tags The distance between With label On the contrary, when m sWhen it is a negative value, it means the label With label More similar.

[0129] During the joint training process of the semantic segmentation network and the semantic mapping network, the mapping layer maps the features extracted by the feature extraction layer to a new space. Through feature transformation, the feature extraction layer can extract feature information that is more suitable for segmentation from the original features, thereby enhancing the feature expression capability. At the same time, during the training process, the mapping layer integrates the hierarchical structure of labels into the information representation, so that the feature extraction layer can not only identify local features, but also combine the hierarchical relationship between categories. By improving the feature representation, the segmentation layer is provided with features that are more suitable for pixel-level classification, thereby directly improving the classification accuracy of the segmentation layer at each pixel in the image.

[0130] Furthermore, the data fusion module is used to receive the lidar point cloud data and the image with semantic segmentation results obtained by the image processing module, perform data fusion to obtain comprehensive perception data, and perceive the environment around the autonomous driving vehicle.

[0131] Specifically, since different sensors have different sensitivities to environmental changes, the system can reduce reliance on the failure of a single sensor by combining data from lidar and cameras, and can more accurately identify and locate surrounding objects, thereby enhancing adaptability and robustness to complex environments.

[0132] Furthermore, the data fusion module includes a camera radar calibration module, a feature extraction module and a feature fusion module.

[0133] The camera radar calibration module uses a calibration plate to obtain the camera's internal parameters and external parameters relative to the laser radar, and aligns the coordinates of the camera and the laser radar.

[0134] Specifically, use a camera and a lidar to respectively capture images of a calibration plate, and use a camera calibration algorithm to calculate the camera's intrinsic parameters, including focal length, principal point position, and lens distortion parameters; extract feature points of the image of the calibration plate captured by the camera, and match them with corresponding points in the lidar point cloud; use the matched feature points to use a generalized ICP algorithm to solve the rotation matrix and translation vector of the camera relative to the lidar, that is, the extrinsic parameters of the camera relative to the lidar.

[0135] The feature extraction module is used to extract the features of the laser radar point cloud and the features of the image with semantic segmentation results respectively.

[0136] Specifically, an edge detection algorithm is used to extract features of the laser radar point cloud; the edge detection algorithm identifies areas in the point cloud with drastic spatial changes, corresponding to the edges of objects.

[0137] The features of the image are extracted using an ORB feature extraction algorithm; the ORB feature extraction algorithm is an efficient feature extraction method that is invariant to rotation.

[0138] The feature fusion module is used to fuse the extracted laser radar point cloud features with the features of the image to obtain comprehensive perception data with distance information and semantic segmentation results.

[0139] Specifically, the lidar point cloud features and the image features are added to obtain fused features, which serve as comprehensive perception data and can combine the lidar's precise distance measurement with the rich semantic information of the image to obtain comprehensive and accurate environmental data.

[0140] Furthermore, the decision module is used to receive the comprehensive perception data, make obstacle avoidance strategies, and generate decision instructions.

[0141] Specifically, the decision-making module is responsible for evaluating the current state of the autonomous driving vehicle based on the comprehensive perception data, identifying possible collision risks and other dangerous situations, formulating obstacle avoidance strategies, making driving decisions, and generating corresponding control instructions.

[0142] Furthermore, the decision module includes: a state module, an action space module, a reward function module and a reinforcement learning module.

[0143] The state module is used to convert the comprehensive perception data into the state representation required for driving decision-making, including vehicle position, speed, and surrounding environment information.

[0144] Specifically, a state representation vector including the comprehensive perception data and the position and speed of the autonomous driving vehicle is constructed to provide accurate and real-time dynamic information of the vehicle and the surrounding environment as a basis for decision-making, path planning, obstacle avoidance, and vehicle control functions.

[0145] The action space module is used to define feasible driving actions based on the driving decision model, including acceleration, deceleration, and steering, and to construct an action space.

[0146] Specifically, the action space module defines the basic actions that the vehicle can perform, and selects one or more feasible actions for each state for further processing by the decision module.

[0147] The reward function module is used to design a reward function to guide the optimization of driving decisions, including punishing collisions and encouraging smooth driving.

[0148] Specifically, the reward function is used to quantify the pros and cons of each action. Exemplarily, positive rewards are assigned to safe driving behaviors. Exemplarily, safe driving behaviors may be complying with traffic regulations, keeping a safe distance, etc.; negative rewards are assigned to unsafe behaviors. Exemplarily, unsafe behaviors may be near collisions; positive rewards are assigned to efficient driving behaviors. Exemplarily, efficient driving behaviors may be reaching a destination quickly; negative rewards are assigned to inefficient driving behaviors. Exemplarily, inefficient driving behaviors may be unnecessary stops or decelerations; positive rewards are assigned to smooth driving behaviors. Exemplarily, smooth driving behaviors may be smooth acceleration and / or steering; negative rewards are assigned to aggressive driving behaviors. Exemplarily, aggressive driving behaviors may be sudden acceleration or sudden braking; rewards are calculated at each time step to guide the training and decision-making of the reinforcement learning algorithm.

[0149] The reinforcement learning module is used to train the reinforcement learning algorithm and optimize the driving decision model so that the driving decision model can select the optimal driving action under different conditions.

[0150] Specifically, this embodiment selects Q-learning as the reinforcement learning algorithm, which is applicable to discrete action spaces and can achieve optimal strategy learning in unknown environments.

[0151] Construct a training dataset, including images of various driving scenarios, lidar point cloud data, vehicle position and speed, and corresponding driving action labels; train the reinforcement learning algorithm in an iterative manner. In each iteration, the driving decision model can select the optimal driving action based on the current state and reward function, and execute the action to obtain rewards and new states; the reinforcement learning algorithm can update the model parameters based on the rewards and new states.

[0152] It should be noted that through the state module, action space module, reward function module and reinforcement learning module, the data fusion and processing results are converted into the state representation required for driving decision-making, and the space of driving actions is defined. By designing a suitable reward function and using the reinforcement learning algorithm for training, the driving decision model can be optimized to select the optimal driving action under different states. The optimization of this decision algorithm can improve the decision-making performance of the autonomous driving obstacle avoidance system, enabling it to make more appropriate and safer driving decisions.

[0153] Furthermore, the control module is used to receive the decision instruction to control the autonomous driving vehicle to achieve obstacle avoidance.

[0154] Specifically, the control module receives decision instructions from the decision module. Based on the decision instructions, the control module sends control signals to the vehicle's actuators (such as the engine, braking system, steering system, etc.). The control signals can accurately regulate the vehicle's acceleration, braking and steering behaviors, so that the vehicle can drive safely and stably according to the planning of the decision module.

[0155] In summary, the hierarchical and precise identification automatic driving obstacle avoidance system of the embodiment of the present invention has the following beneficial effects:

[0156] 1. The autonomous driving obstacle avoidance system of the present invention integrates the LiDAR point cloud data and the semantic segmentation results of the street view image through a data fusion module. The LiDAR provides accurate distance and shape information, while the image semantic segmentation provides object category information. The combination of the two can improve the perception accuracy of the surrounding environment and provide a more comprehensive environmental perception capability.

[0157] 2. The decision module of the automatic driving obstacle avoidance system of the present invention utilizes a reinforcement learning algorithm to optimize the driving decision model, and is able to select the optimal driving action under different conditions, thereby improving the efficiency and effectiveness of decision-making.

[0158] 3. The image processing module of the autonomous driving obstacle avoidance system of the present invention can achieve accurate and rapid semantic recognition and segmentation, and the system can respond quickly, thereby improving the safety of the autonomous driving vehicle during the obstacle avoidance process. Through the hierarchical perception semantic segmentation network, the system can process complex street scene images and adapt to different driving environments and conditions.

[0159] 4. The semantic segmentation network used in the image processing module of the autonomous driving obstacle avoidance system of the present invention can effectively integrate the hierarchical structure of labels, including hierarchical dependency and hierarchical similarity, based on the constructed label tree during the training process by introducing the tree maximum and minimum loss function and the tree triplet loss function. This fusion helps to further improve the accuracy of segmentation, so that the network can more accurately identify and segment different objects and scenes in the image.

[0160] The above description is only a preferred specific implementation manner of the present invention, but the protection scope of the present invention is not limited thereto. Any changes or substitutions that can be easily conceived by any technician familiar with the technical field within the technical scope disclosed by the present invention should be covered within the protection scope of the present invention.

Claims

1. A hierarchical and precise identification automatic driving obstacle avoidance system, characterized in that: The automatic driving obstacle avoidance system comprises: a data acquisition module, an image processing module, a data fusion module, a decision module and a control module; The data acquisition module includes a sensor and a preprocessing module, which is used to use the sensor to obtain the driving environment data and the vehicle's own data of the autonomous driving vehicle and perform preprocessing; wherein the driving environment data includes: laser radar point cloud data and continuous frame street view images taken during driving; the vehicle's own data includes the vehicle's current position and speed; The image processing module is used to perform hierarchical semantic segmentation on each frame of street view image to obtain semantic segmentation results of key targets in each frame of street view image; The data fusion module is used to receive the laser radar point cloud data and the image with semantic segmentation results obtained by the image processing module, perform data fusion to obtain comprehensive perception data, and perceive the environment around the autonomous driving vehicle; The decision module is used to receive the comprehensive perception data, make obstacle avoidance strategies, and generate decision instructions; The control module is used to receive the decision instruction to control the autonomous driving vehicle to achieve obstacle avoidance.

2. The system according to claim 1, characterized in that: The data fusion module includes a camera radar calibration module, a feature extraction module and a feature fusion module; The camera radar calibration module uses a calibration plate to obtain the camera's intrinsic parameters and the extrinsic parameters relative to the laser radar, and aligns the coordinates of the camera and the laser radar; The feature extraction module is used to extract the features of the laser radar point cloud and the features of the image with semantic segmentation results respectively; The feature fusion module is used to fuse the extracted laser radar point cloud features with the features of the image to obtain comprehensive perception data with distance information and semantic segmentation results.

3. The system according to claim 2, characterized in that: The decision module includes: a state module, an action space module, a reward function module and a reinforcement learning module; The state module is used to convert the comprehensive perception data into a state representation required for driving decision-making, including vehicle position, speed, and surrounding environment information; The action space module is used to define feasible driving actions based on the driving decision model, including acceleration, deceleration, and steering, and construct an action space; The reward function module is used to design a reward function to guide the optimization of driving decisions, including punishing collisions and encouraging smooth driving; The reinforcement learning module is used to train the reinforcement learning algorithm and optimize the driving decision model so that the driving decision model can select the optimal driving action under different conditions.

4. The system according to claim 1, characterized in that: The image processing module uses a trained semantic segmentation network based on hierarchical perception to perform semantic segmentation; The semantic segmentation network based on hierarchical perception includes: a feature extraction layer and a segmentation layer; The feature extraction layer is used to extract feature information of each frame of street view image; The segmentation layer is used to perform pixel-level classification on each frame image based on the feature information to obtain a semantic segmentation result of a key target in each frame image.

5. The system according to claim 4, characterized in that: The semantic segmentation network based on hierarchical perception is trained using the following method: Using a street view semantic segmentation dataset as a training sample; the street view semantic segmentation dataset includes a street view image set and a corresponding label set; Constructing a label tree of street view semantics based on the hierarchical relationship of each label in the label set; Based on the street view semantic segmentation dataset and the label tree of the street view semantics, a semantic segmentation network and a semantic mapping network are trained simultaneously; wherein the semantic mapping network includes a feature extraction layer and a mapping layer shared with the semantic segmentation network, and is used to reorganize the features of the input image to obtain a new feature vector; during training, based on the similarity information of the labels in the label tree of the street view semantics, new feature vectors of similar categories are close to each other in the feature space, and new feature vectors of dissimilar categories are far away from each other.

6. The system according to claim 5, characterized in that: The segmentation layer includes a first convolution layer, an activation function, a second convolution layer and a sigmoid function, and is used to perform semantic segmentation on the input image based on the number of nodes in the label tree of the street view semantics; wherein, The size of the first convolutional layer is 1×1×|V|, and is used to perform a convolution operation on the feature map to obtain transformed feature information; wherein |V| represents the number of nodes in the label tree of the street view semantics; The activation function is a ReLU activation function, which is used to perform a nonlinear transformation on each feature to obtain feature information after the nonlinear transformation; The size of the second convolutional layer is 1×1×|V|, and is used to perform a convolution operation on the feature information after the nonlinear transformation again; The sigmoid function is used to perform a pixel-by-pixel classification probability estimation operation on the output features of the second convolutional layer to obtain the semantic probability distribution of each pixel.

7. The system according to any one of claims 5 or 6, characterized in that: The step of constructing a label tree of street view semantics based on the hierarchical relationship of each label includes: For each tag, find its parent class in the WordNet dataset, and search up layer by layer until the root node of the WordNet dataset is reached to obtain the hierarchical relationship of each tag; Based on the hierarchical relationship of each label, a label tree T = {V, E} of street view semantics is constructed; where V represents the node set of the label tree, which is the label class and its parent class in the WordNet dataset; E represents the node relationship of the label tree.

8. The system according to claim 7, characterized in that: The method of simultaneously training a semantic segmentation network and a semantic mapping network based on the street view semantic segmentation dataset and the street view semantic label tree includes: The street view semantic segmentation dataset and the label tree of the street view semantics are loaded, the semantic segmentation network and the semantic mapping network are trained simultaneously using the tree maximum and minimum loss function and the tree triplet loss function, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are updated using gradient back propagation, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are saved after the training is completed, and the semantic segmentation network is obtained based on the trained feature extraction layer and the segmentation layer.

9. The system according to claim 8, characterized in that: The tree maximum and minimum loss function is expressed as: Wherein, B represents the number of images in the street view semantic segmentation dataset; M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree of the street view semantics; Indicates the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image; Represents the category probability value of the label value corresponding to the k-th class of the (i, j)-th pixel in the b-th image after the tree maximum and minimum transformation.

10. The system according to claim 8, characterized in that: The tree triplet loss function is expressed as: Among them, D TT Represents a valid label triple vector set; N b Represents the number of triplets in the valid label triple vector set; Represents the first eigenvector in the sth valid label triple vector set; Represents the second eigenvector in the sth valid label triple vector set; represents the third eigenvector in the sth valid label triple vector set; d() represents the vector distance function; m s represents the optimal label interval of the sth valid label triple vector set; the three outputs of the mapping layer are selected The new feature vector and the corresponding label constitute a triple vector set; traverse the triple vector set, A valid label triple vector set is obtained based on the label tree.

Images