An automatic driving obstacle avoidance system with hierarchical accurate identification

Abstract

The application relates to an automatic driving obstacle avoidance system based on hierarchical accurate identification and belongs to the technical field of automatic driving. The system comprises a data acquisition module, an image processing module, a data fusion module and a decision module. The data acquisition module comprises a sensor and a preprocessing module, is used for acquiring automatic driving vehicle driving environment data and vehicle self data by means of the sensor and performing preprocessing, the image processing module is used for performing hierarchical semantic segmentation on each frame of street view image to obtain semantic segmentation results of key targets in each frame of street view image, the data fusion module is used for receiving laser radar point cloud data and the image obtained by the image processing module and having the semantic segmentation results, performing data fusion to obtain comprehensive perception data, and perceiving the environment around the automatic driving vehicle, the decision module is used for receiving the comprehensive perception data, making an obstacle avoidance strategy, and generating a decision instruction, and the control module is used for receiving the decision instruction to control the automatic driving vehicle and realizing obstacle avoidance. The automatic driving obstacle avoidance is fast and accurate.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of automatic driving, in particular to an automatic driving obstacle avoidance system with hierarchical accurate identification. BACKGROUND

[0002] With the increasing demand for active safety and intelligence of automobiles in the market, the value of automatic driving in society and economy is increasingly highlighted, and automatic driving technology has become an important topic in the development of today's frontier science and technology, which has a profound impact on the development of science and technology, social economy and many other fields.

[0003] The automatic driving obstacle avoidance system is a system that can help autonomous vehicles perceive and avoid obstacles. Various sensors, algorithms and decision systems are used to monitor and analyze the environment around the vehicle, and through automatic control of the steering, acceleration and braking of the vehicle, potential obstacles are avoided. The goal of this system is to ensure that the vehicle can safely avoid collisions with other vehicles, pedestrians, buildings, animals or other obstacles during driving. The existing automatic driving obstacle avoidance system, due to the fact that in the semantic recognition process, each pixel is usually independently classified, ignoring the correlation between pixels, similar but semantically different objects are misclassified, so that the vehicle makes wrong decisions in the obstacle avoidance process, which poses a serious threat to the safety and reliability of autonomous vehicles. SUMMARY

[0004] In view of the above analysis, the embodiments of the present application aim to provide an automatic driving obstacle avoidance system with hierarchical accurate identification, in order to realize fast and accurate automatic driving obstacle avoidance.

[0005] The purpose of the present application is mainly realized through the following technical solutions:

[0006] The present application provides an automatic driving obstacle avoidance system with hierarchical accurate identification, which comprises a data acquisition module, an image processing module, a data fusion module, a decision module and a control module.

[0007] The data acquisition module comprises a sensor and a preprocessing module, which is used to acquire the driving environment data of the autonomous vehicle and the vehicle itself data by using the sensor and to preprocess them. The driving environment data includes laser radar point cloud data and continuous frame street view images taken during driving. The vehicle itself data includes the current position and speed of the vehicle.

[0008] The image processing module is used to perform hierarchical semantic segmentation on each frame of street view image to obtain the semantic segmentation result of the key target in each frame of street view image.

[0009] The data fusion module is configured to receive the laser radar point cloud data and the image with the semantic segmentation result obtained by the image processing module, perform data fusion to obtain comprehensive perception data, and perceive the environment around the autonomous vehicle.

[0010] The decision module is configured to receive the comprehensive perception data, make an obstacle avoidance strategy, and generate a decision instruction.

[0011] The control module is configured to receive the decision instruction to control the autonomous vehicle and realize obstacle avoidance.

[0012] Further, the data fusion module comprises a camera radar calibration module, a feature extraction module and a feature fusion module.

[0013] The camera radar calibration module uses a calibration board to obtain the intrinsic parameters of the camera and the extrinsic parameters relative to the laser radar, and aligns the coordinates of the camera and the laser radar.

[0014] The feature extraction module is configured to extract the laser radar point cloud features and the features of the image with the semantic segmentation result, respectively.

[0015] The feature fusion module is configured to fuse the extracted laser radar point cloud features and the features of the image to obtain comprehensive perception data with distance information and semantic segmentation results.

[0016] Further, the decision module comprises a state module, an action space module, a reward function module and a reinforcement learning module.

[0017] The state module is configured to convert the comprehensive perception data into a state representation required for driving decision, including vehicle position, speed and surrounding environment information.

[0018] The action space module is configured to define feasible driving actions based on a driving decision model, including acceleration, deceleration and steering, and construct an action space.

[0019] The reward function module is configured to design a reward function to guide the optimization of driving decision, including punishing collision and encouraging smooth driving.

[0020] The reinforcement learning module is configured to train a reinforcement learning algorithm, optimize the driving decision model, and enable the driving decision model to select the optimal driving action under different states.

[0021] Further, the image processing module uses a trained hierarchical perception-based semantic segmentation network for semantic segmentation.

[0022] The hierarchical perception-based semantic segmentation network comprises a feature extraction layer and a segmentation layer.

[0023] The feature extraction layer is configured to extract feature information of each frame of the street view image.

[0024] The segmentation layer is configured to perform pixel-level classification on each frame of the image based on the feature information to obtain a semantic segmentation result of a key target in each frame of the image.

[0025] Further, the hierarchical perception-based semantic segmentation network is trained using the following method:

[0026] A street view semantic segmentation dataset is used as a training sample; the street view semantic segmentation dataset includes a street view image set and a corresponding label set;

[0027] A label tree of street view semantics is constructed based on the hierarchical relationship of each label in the label set;

[0028] The semantic segmentation network and the semantic mapping network are simultaneously trained based on the street view semantic segmentation dataset and the label tree of street view semantics; wherein the semantic mapping network includes a feature extraction layer shared with the semantic segmentation network and a mapping layer, which is configured to recombine the features of the input image to obtain a new feature vector; during training, based on the similarity information of the labels in the label tree of street view semantics, the new feature vectors of similar categories are close to each other in the feature space, and the new feature vectors of dissimilar categories are far away from each other.

[0029] Further, the segmentation layer includes a first convolutional layer, an activation function, a second convolutional layer, and a sigmoid function, which are configured to perform semantic segmentation on the input image based on the number of nodes of the label tree of street view semantics; wherein,

[0030] The size of the first convolutional layer is 1x1x|V|, which is configured to perform convolution operation on the feature map to obtain transformed feature information; wherein |V| represents the number of nodes of the label tree of street view semantics;

[0031] The activation function is a ReLU activation function, which is configured to perform nonlinear transformation on each feature to obtain nonlinearly transformed feature information;

[0032] The size of the second convolutional layer is 1x1x|V|, which is configured to perform convolution operation on the nonlinearly transformed feature information again;

[0033] The sigmoid function is configured to perform pixel-by-pixel classification probability estimation operation on the output feature of the second convolutional layer to obtain the probability distribution of the semantics of each pixel point.

[0034] Further, the label tree of street view semantics is constructed based on the hierarchical relationship of each label, including:

[0035] For each label, find its parent class in the WordNet dataset, and find the hierarchical relationship of each label by layer by layer until reaching the root node of the WordNet dataset;

[0036] Based on the hierarchical relationship of each label, a label tree T = {V, E} of the street view semantics is constructed; wherein V represents a node set of the label tree, which is a label class and its parent class in the WordNet dataset; E represents a node relationship of the label tree.

[0037] Further, the training of the semantic segmentation network and the semantic mapping network based on the street view semantic segmentation dataset and the label tree of the street view semantics comprises:

[0038] Loading the street view semantic segmentation dataset and the label tree of the street view semantics, simultaneously training the semantic segmentation network and the semantic mapping network using a tree maximum minimum loss function and a tree triplet loss function, updating feature extraction layer parameters, segmentation layer parameters and mapping layer parameters by gradient back propagation, saving the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters after training, and obtaining a semantic segmentation network based on the trained feature extraction layer and the segmentation layer.

[0039] Further, the tree maximum minimum loss function is represented as:

[0040]

[0041] Wherein, B represents the number of images in the street view semantic segmentation dataset; M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree of the street view semantics; represents the label value corresponding to the kth class of the (i, j) pixel in the bth image; represents the class probability value of the label value corresponding to the kth class of the (i, j) pixel in the bth image after tree maximum minimum transformation.

[0042] Further, the tree triplet loss function is represented as:

[0043]

[0044] Wherein, D TT represents a set of effective label triplet vectors; N b represents the number of triplets in the set of effective label triplet vectors; represents the first feature vector in the s th effective label triplet vector set; represents the second feature vector in the s th effective label triplet vector set; represents the third feature vector in the s th effective label triplet vector set; d() represents a vector distance function; ms The label optimal interval representing the s-th effective label triple vector set; optionally, three new feature vectors output by the mapping layer and the corresponding labels constitute a triple vector set; traversing the triple vector set, obtaining an effective label triple vector set based on the label tree.

[0045] Compared with the prior art, the present application can at least achieve one of the following beneficial effects:

[0046] 1、The automatic driving obstacle avoidance system of the present application integrates the laser radar point cloud data and the semantic segmentation result of the street view image through the data fusion module, the laser radar provides accurate distance and shape information, and the image semantic segmentation provides object category information, the combination of the two can improve the perception accuracy of the surrounding environment and provide more comprehensive environmental perception capability.

[0047] 2、The decision module of the automatic driving obstacle avoidance system of the present application optimizes the driving decision model by using a reinforcement learning algorithm, which can select the optimal driving action in different states and improve the efficiency and effect of decision-making.

[0048] 3、The image processing module of the automatic driving obstacle avoidance system of the present application can quickly react through precise and rapid semantic recognition and segmentation, thereby improving the safety of the automatic driving vehicle in the obstacle avoidance process, and through the hierarchical perception semantic segmentation network, the system can process complex street view images and adapt to different driving environments and conditions.

[0049] 4、The semantic segmentation network used by the image processing module of the automatic driving obstacle avoidance system of the present application can effectively fuse the hierarchical structure of the label, including the hierarchical dependency relationship and the hierarchical similarity relationship, in the training process based on the constructed label tree by introducing the tree maximum and minimum loss function and the tree triple loss function. This fusion helps to further improve the segmentation accuracy, so that the network can more accurately identify and segment different objects and scenes in the image.

[0050] In the present application, the above technical solutions can also be combined with each other to realize more preferred combination schemes. Other features and advantages of the present application will be described in the subsequent specification, and some advantages will become apparent from the specification, or will be understood by implementing the present application. The purpose and other advantages of the present application can be realized and obtained through the contents specifically pointed out in the specification and the drawings. BRIEF DESCRIPTION OF DRAWINGS

[0051] The accompanying drawings are included to provide a further understanding of the application and are incorporated herein and constitute a part of the detailed description. The drawings illustrate embodiments of the application and, together with the description, serve to explain the principles of the application. In the drawings:

[0052] Figure 1A structural schematic diagram of an automatic driving obstacle avoidance system with hierarchical accurate identification in an embodiment of the present application;

[0053] Figure 2 A structural schematic diagram of a semantic segmentation network in an embodiment of the present application. DETAILED DESCRIPTION

[0054] The preferred embodiments of the present application will be described in detail below with reference to the drawings, wherein the drawings constitute a part of this application and are used to explain the principles of the embodiments of the present application, but are not used to limit the scope of the present application.

[0055] One specific embodiment of the present application discloses an automatic driving obstacle avoidance system with hierarchical accurate identification, as shown in the figure, the automatic driving obstacle avoidance system comprises a data acquisition module, an image processing module, a data fusion module, a decision module and a control module. Figure 1

[0056] The data acquisition module comprises a sensor and a preprocessing module, which is used to acquire the driving environment data of the automatic driving vehicle and the vehicle itself data and perform preprocessing by using the sensor; wherein the driving environment data comprises laser radar point cloud data and continuous frame street view images taken in the driving process; the vehicle itself data comprises the current position and speed of the vehicle.

[0057] Specifically, the sensor comprises a laser radar and a camera installed on the automatic driving vehicle, which is used to acquire the surrounding environment of the automatic driving vehicle in the driving process.

[0058] The laser radar can measure the distance to the surrounding objects by emitting laser pulses and measuring the time of reflection, and provide accurate distance information of the surrounding objects, at the same time, the laser radar can also generate three-dimensional laser radar point cloud data of the surrounding environment, which is used to understand the spatial structure around the vehicle.

[0059] The camera can capture high-resolution continuous frame street view semantic images of the automatic driving vehicle in the driving process, which is used to identify key targets such as road signs, traffic signals, pedestrians and vehicles in the automatic driving process, and through semantic segmentation of the street view semantic images, identify and classify each pixel in the image to belong to which object.

[0060] The sensor further comprises a global positioning system and a speed sensor installed on the automatic driving vehicle, which is used to acquire the position and speed of the automatic driving vehicle in the driving process.

[0061] The global positioning system can provide accurate position and time information of the vehicle in the global range.

[0062] ​The speed sensor is installed on a wheel or a transmission shaft of the autonomous vehicle, and is used to measure the real-time speed of the vehicle.

[0063] Specifically, the laser radar point cloud is preprocessed, including using a filtering algorithm to remove noise points in the point cloud, and the noise in the point cloud will reduce the accuracy and reliability of the data, and the quality of the radar point cloud data can be improved by removing the noise points through the filtering algorithm, and the filtering algorithm can use median filtering or Gaussian filtering, and the embodiment is not limited.

[0064] For the continuous frame of the street view image shot in the driving process, after using OpenCV to collect each frame of the street view image in the video image, a filter is used to remove noise in the image, improve image quality and enhance features, and the filter can use Gaussian filtering, median filtering or bilateral filtering, and the embodiment is not limited.

[0065] Further, the image processing module is used to perform hierarchical semantic segmentation on each frame of the street view image to obtain the semantic segmentation result of the key target in each frame of the street view image.

[0066] Specifically, the street view image is subjected to semantic segmentation to identify various targets in the street view image, such as vehicles, pedestrians, traffic signs, road boundaries, etc., and the system can understand which category each pixel in the image belongs to, thereby understanding the surrounding environment at a semantic level and providing necessary environmental information for the obstacle avoidance system. Precise semantic segmentation can improve the perception ability of the autonomous vehicle to the surrounding environment, thereby improving the safety of automatic driving, and therefore, accurate semantic segmentation is crucial for the autonomous driving obstacle avoidance system.

[0067] Further, the image processing module uses a trained hierarchical perception-based semantic segmentation network for semantic segmentation; the hierarchical perception-based semantic segmentation network, as shown in Figure 2 , includes a feature extraction layer and a segmentation layer.

[0068] The feature extraction layer is used to extract feature information of each frame of the street view image.

[0069] Specifically, the feature extraction layer of the semantic segmentation network uses an existing segmentation network, removes the last classification layer thereof, and is used to extract features of an input image. Exemplarily, the existing segmentation network can use DeepLab, PSPNet and UNet, and the embodiment is not limited.

[0070] The segmentation layer is used to perform pixel-level classification on each frame of the image based on the feature information to obtain the semantic segmentation result of the key target in each frame of the image.

[0071] Further, the segmentation layer comprises a first convolutional layer, an activation function, a second convolutional layer and a sigmoid function, and is configured to perform semantic segmentation on the input image based on the number of nodes of the label tree of the street view semantics; wherein,

[0072] The first convolutional layer has a size of 1*1*|V|, and is configured to perform convolutional operation on the feature map to obtain transformed feature information; wherein |V| represents the number of nodes of the label tree of the street view semantics.

[0073] Specifically, the label tree of the street view semantics is obtained by using a street view semantic segmentation dataset as a training sample when the semantic segmentation network is trained; the street view semantic segmentation dataset comprises a street view image set and a corresponding label set; and the label tree of the street view semantics is constructed based on the hierarchical relationship of each label in the label set.

[0074] Further, the label tree of the street view semantics is constructed based on the hierarchical relationship of each label, and comprises:

[0075] For each label, the parent class is found in the WordNet dataset, and the hierarchical relationship of each label is found layer by layer upwards until the root node of the WordNet dataset is reached;

[0076] Based on the hierarchical relationship of each label, a label tree T={V, E} is constructed; wherein V represents a set of nodes of the label tree, which is a label class and its parent class in the WordNet dataset; and E represents the node relationship of the label tree.

[0077] Specifically, the label tree is a tree-like hierarchical structure, which is used to represent the hierarchical relationship between classes. Each node of the label tree represents a class, and the edge represents the parent-child relationship between classes. The WordNet dataset is a large-scale English vocabulary database, in which the vocabulary is organized in the form of synonym sets, and these synonym sets are connected by semantic relationships (such as parent class, child class, etc.).

[0078] In this embodiment, for a plurality of labels corresponding to each image in the image set, each label represents a class, and the parent class and the parent class of the parent class are found in the WordNet dataset until the root node of the WordNet dataset is reached. The class of the label and all parent nodes up to the root node are nodes of the label tree. For example, in a city street view picture, “vehicle” can be a parent node, and “car”, “truck” and “motorcycle” can be child nodes.

[0079] It should be noted that the 1*1 convolution layer of the first convolution layer is used to fuse the feature information of different layers, and compress the feature dimension at the same time, compress the high-dimensional feature map obtained from the feature extraction layer to the same dimension as the number of label tree nodes, and convert the multi-channel feature map into a feature map with |V| channels, each channel corresponding to a label tree node.

[0080] Further, the activation function of the segmentation layer is a ReLU activation function, which is used for nonlinear transformation of each feature to obtain nonlinear transformed feature information.

[0081] Specifically, the ReLU activation function introduces nonlinear transformation by setting all negative values to 0 and keeping positive values unchanged, so that the network can learn more complex feature representation and patterns.

[0082] Further, the size of the second convolution layer of the segmentation layer is 1*1*|V|, which is used for convolution operation on the nonlinear transformed feature information again.

[0083] Specifically, the convolution layer can adjust the feature weight again, strengthen the features useful for segmentation task, and suppress irrelevant features.

[0084] Further, the sigmoid function of the segmentation layer is used for pixel-wise classification probability estimation operation on the output features of the second convolution layer to obtain the probability distribution of each pixel point semantic.

[0085] Specifically, the sigmoid function is used to obtain the probability distribution of each pixel point for |V| categories.

[0086] It should be noted that the segmentation layer can be represented as f seg (I)∈R M×N×|V| Where M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree. The segmentation layer receives the output of the feature extraction layer, and performs pixel-level classification on the output of the feature extraction layer according to the category nodes defined in the label tree, and predicts the category of each pixel in the image.

[0087] Further, the hierarchical perception-based semantic segmentation network is trained using the following method:

[0088] The Street View semantic segmentation dataset is used as a training sample; the Street View semantic segmentation dataset includes a set of Street View images and a corresponding label set.

[0089] Specifically, the training sample image set in the embodiment is a street view semantic segmentation dataset Cityscapes. The dataset is specifically for urban street scenes, composed of street views of 50 different cities, containing 5,000 finely labeled pictures and 20,000 roughly labeled pictures, and using 19 commonly used categories to evaluate segmentation accuracy, such as roads, sidewalks, buildings, vehicles, etc.

[0090] A label tree of street view semantics is constructed based on the hierarchical relationship of each label in the label set.

[0091] The semantic segmentation network and the semantic mapping network are simultaneously trained based on the street view semantic segmentation dataset and the label tree of street view semantics; wherein the semantic mapping network includes a feature extraction layer shared with the semantic segmentation network and a mapping layer, which is used to reorganize the features of the input image to obtain a new feature vector; in the training, based on the similarity information of the labels in the label tree of street view semantics, the new feature vectors of similar categories are close to each other in the feature space, and the new feature vectors of dissimilar categories are far away from each other.

[0092] Specifically, in the semantic mapping network, the semantic mapping network and the semantic segmentation network can share the features extracted from the input image through the feature extraction layer shared by the semantic mapping network and the semantic segmentation network, so as to improve the generalization ability of the model, and at the same time, help to improve the expression ability of the features in the training phase.

[0093] The mapping layer of the semantic mapping network includes a first mapping convolution layer, a mapping activation function, a second mapping convolution layer and a sigmoid function, which are used to map the feature information in the image to a new space; wherein,

[0094] The size of the first mapping convolution layer is 1x1x32, which is used for convolution operation on the feature map to obtain transformed feature information.

[0095] The mapping activation function is a ReLU activation function, which is used for nonlinear transformation of each feature to obtain nonlinear transformed feature information.

[0096] The size of the second mapping convolution layer is also 1x1x32, which is used to adjust the feature weight.

[0097] The sigmoid function is used to map the output feature of the second mapping convolution layer to a new space to obtain a new feature vector.

[0098] It should be noted that the mapping layer is represented as f proj (I)∈R M×N×32The features extracted by the feature extraction layer are reorganized into a new space of 32 channels to obtain a new feature vector. In the training process, a loss function is designed based on the distance of the new feature vector in the feature space and the distance of the labeled label in the label tree. The designed loss function fuses the similarity information of the label by punishing the similar classes with similar distances and the similar classes with distant distances in the feature space, so that the feature extraction layer can extract more distinctive features to provide a better basis for the segmentation layer.

[0099] Further, the semantic segmentation network and the semantic mapping network are simultaneously trained based on the street view semantic segmentation dataset and the label tree of the street view semantic, which comprises:

[0100] The street view semantic segmentation dataset and the label tree of the street view semantic are loaded, and the semantic segmentation network and the semantic mapping network are simultaneously trained using a tree maximum minimum loss function and a tree triplet loss function. The feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are updated by gradient backpropagation. After the training is completed, the feature extraction layer parameters, the segmentation layer parameters and the mapping layer parameters are saved. The semantic segmentation network is obtained based on the trained feature extraction layer and the segmentation layer.

[0101] Specifically, the semantic segmentation network and the semantic mapping network are simultaneously trained based on a total loss function L total = L TMM + L TT ; wherein L TMM is a tree maximum minimum loss function; and L TT is a tree triplet loss function. The two networks are simultaneously optimized as a whole, interact with each other during the training process, and learn cooperatively. At the same time, the risk of overfitting can be reduced, and the robustness of the model can be increased.

[0102] Further, the tree maximum minimum loss function is:

[0103]

[0104] wherein B represents the number of images in the training sample image set; M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree; represents the label value corresponding to the kth class of the (i,j) pixel in the bth image; represents the class probability value of the label value corresponding to the kth class of the (i,j) pixel in the bth image after tree maximum minimum transformation.

[0105] Specifically, the semantic segmentation network obtains the difference between the predicted class probability value and the real label through the tree max-min loss function, and makes the segmentation result of the semantic segmentation network more consistent with the hierarchical structure defined in the label tree in the process of minimizing the loss function.

[0106] Further, the tree max-min transformation is performed using the following formula to obtain the transformed class probability value of each pixel:

[0107]

[0108] wherein, represents the transformed class probability value corresponding to the kth class of the (i, j) pixel in the bth image; A k represents the parent node set of class k in the label tree; D k represents the child node set of class k in the label tree; represents the class probability value corresponding to the kth class of the (i, j) pixel in the bth image; A represents the class probability value corresponding to the kth class of the (i, j) pixel in the bth image; A represents the label mask of class k.

[0109] Specifically, when performing semantic segmentation, the segmentation layer outputs a probability distribution for each pixel, indicating the likelihood of the pixel belonging to each class. Through the hierarchical structure between the class labels on the label tree, when the pixel in the segmentation layer is predicted to be a certain sub-class, the prediction probability of its parent class should also be high. For example, if the model predicts that a pixel belongs to the sub-class "car", it should also predict that the pixel has a high probability of belonging to its parent class "vehicle". The tree max-min transformation is used to ensure that the hierarchical structure output by the network segmentation layer is consistent with the hierarchical structure defined in the label tree, i.e., the output value of the sub-class should not exceed that of its parent class. This transformation ensures that the output value of the segmentation layer conforms to the hierarchical relationship between the labels when performing pixel-level classification, which is beneficial to improving the logical consistency and accuracy of the segmentation.

[0110] Further, the tree triplet loss function

[0111]

[0112] wherein, D TT represents the set of valid label triplet vectors; N b represents the number of triplets in the set of valid label triplet vectors; represents the first feature vector in the s th valid label triplet vector set; represents the second feature vector in the s th valid label triplet vector set; ​denotes the third feature vector in the s-th effective label triplet vector set; d() denotes a vector distance function; m s denotes the label optimal interval of the s-th effective label triplet vector set; optionally, three new feature vectors output by the mapping layer and corresponding labels constitute a triplet vector set;

[0113] The effective label triplet vector set is obtained based on the label tree by traversing the triplet vector set.

[0114] Specifically, the tree triplet loss function measures the hierarchical similarity between labels by calculating the distance between feature vectors, and by minimizing the tree triplet loss function, the feature extraction layer extracts feature vectors of similar categories closer and feature vectors of dissimilar categories farther, which is beneficial to the semantic segmentation performed by the segmentation layer more accurately.

[0115] The triplet vector set is all feature vectors output by the mapping layer and mapped to a new space Any three different feature vectors and their corresponding labels are taken out to form a triplet vector set, denoted as where N t denotes the number of triplet vectors in the triplet vector set D T . denotes the first feature vector in the r-th triplet vector set; denotes the second feature vector in the r-th triplet vector set; denotes the third feature vector in the r-th triplet vector set; denotes the label corresponding to the first feature vector in the r-th triplet vector set; denotes the label corresponding to the second feature vector in the r-th triplet vector set; denotes the label corresponding to the third feature vector in the r-th triplet vector set.

[0116] Further, the effective label triplet vector set is obtained based on the label tree by traversing the triplet vector set, including that the shortest path length of the label of the first feature vector and the label of the second feature vector in the label tree is less than the shortest path length of the label of the first feature vector and the label of the third feature vector in the label tree.

[0117] Specifically, the effective label triplet vector is the label of the three feature vectors in the triplet vector, which satisfies the following conditions based on its position in the label tree:

[0118]

[0119] Ψ() represents the shortest path length of two label nodes on the label tree.

[0120] In the label tree, the shortest path length between two nodes is considered as the similarity measure in the hierarchy, the shorter the path length, the more similar the two classes are. In the training process, the effective label triple vector set is constructed, which can make the whole model more accurately distinguish the classes that are visually similar but semantically different, help the model capture the differences and similarities between classes, and improve the segmentation accuracy of the semantic segmentation network.

[0121] The triple vector set that meets the above conditions constitutes an effective label triple vector set: N b represents the number of triples in the effective label triple vector set.

[0122] Specifically, for the first feature vector, the second feature vector and the third feature vector in the effective label triple vector set, the distance between the vectors is calculated using the following formula respectively:

[0123]

[0124] The formula is a variant formula of cosine similarity, and the result is to represent the distance value in the range of 0 to 1, 0 means that the vectors are completely the same, and 1 means that the vectors are completely opposite.

[0125] Further, the label optimal interval of the effective label triple vector set is calculated using the following formula:

[0126]

[0127] wherein, represents the label of the first feature vector in the s-th effective label triple vector set; represents the label of the third feature vector in the s-th effective label triple vector set; represents the label of the second feature vector in the s-th effective label triple vector set; Ψ() represents the shortest path length of two labels in the label tree.

[0128] Specifically, the label optimal interval is to quantify the hierarchical similarity between labels, and use the quantified similarity to optimize the performance of the model when training the model. When m s is a positive value, it means that the distance between the label and the label is greater than the distance between the label and the label , that is, the label is more similar to the label . Conversely, when m sis a negative value, indicating that the label is more similar to the label .

[0129] In the joint training process of the semantic segmentation network and the semantic mapping network, the mapping layer maps the features extracted by the feature extraction layer to a new space. Through the transformation of the features, the feature extraction layer can extract more suitable feature information for segmentation from the original features, enhancing the feature expression capability. At the same time, the mapping layer integrates the hierarchical structure of the label into the information representation during the training process, so that the feature extraction layer can not only recognize local features but also combine the hierarchical relationship between classes. By improving the feature representation, the segmentation layer is provided with more suitable features for pixel-level classification, thereby directly improving the classification accuracy of each pixel point in the image.

[0130] Further, the data fusion module is configured to receive the laser radar point cloud data and the image with the semantic segmentation result obtained by the image processing module, and perform data fusion to obtain comprehensive perception data for perceiving the environment around the autonomous vehicle.

[0131] Specifically, due to the different sensitivities of different sensors to environmental changes, the system can reduce the dependence on a single sensor failure by combining the data of the laser radar and the camera, and can more accurately identify and locate the surrounding objects, thereby enhancing the adaptability and robustness to complex environments.

[0132] Further, the data fusion module comprises a camera-laser radar calibration module, a feature extraction module, and a feature fusion module.

[0133] The camera-laser radar calibration module uses a calibration board to obtain the intrinsic parameters of the camera and the extrinsic parameters relative to the laser radar, and aligns the coordinates of the camera and the laser radar.

[0134] Specifically, the camera and the laser radar are used to respectively capture images of the calibration board, the intrinsic parameters of the camera are calculated by a camera calibration algorithm, including the focal length, the principal point position, and the lens distortion parameters; the feature points of the image of the calibration board captured by the camera are extracted and matched with the corresponding points in the laser radar point cloud; and the matched feature points are used to solve the rotation matrix and the translation vector of the camera relative to the laser radar, i.e., the extrinsic parameters of the camera relative to the laser radar, by a generalized ICP algorithm.

[0135] The feature extraction module is configured to extract the features of the laser radar point cloud and the image with the semantic segmentation result, respectively.

[0136] Specifically, an edge detection algorithm is used to extract the features of the laser radar point cloud; the edge detection algorithm identifies the regions with sharp spatial changes in the point cloud, which correspond to the edges of the objects.

[0137] extracting features of the image using an ORB feature extraction algorithm; the ORB feature extraction algorithm is a high-efficiency and rotation-invariant feature extraction method.

[0138] The feature fusion module is configured to fuse the extracted lidar point cloud features and the features of the image to obtain comprehensive perception data with distance information and semantic segmentation results.

[0139] Specifically, the lidar point cloud features and the features of the image are added to obtain fused features as comprehensive perception data, which can combine the accurate distance measurement of the lidar and the rich semantic information of the image to obtain comprehensive and accurate environmental data.

[0140] Further, the decision module is configured to receive the comprehensive perception data, make an obstacle avoidance strategy, and generate a decision instruction.

[0141] Specifically, the decision module is responsible for evaluating the current state of the autonomous vehicle, identifying possible collision risks and other dangerous situations, formulating an obstacle avoidance strategy, making a driving decision, and generating a corresponding control instruction based on the comprehensive perception data.

[0142] Further, the decision module includes a state module, an action space module, a reward function module, and a reinforcement learning module.

[0143] The state module is configured to convert the comprehensive perception data into a state representation required for driving decisions, including vehicle position, speed, and surrounding environment information.

[0144] Specifically, a state representation vector including the comprehensive perception data and the position and speed of the autonomous vehicle is constructed to provide accurate and real-time dynamic information of the vehicle and the surrounding environment as the basis for decision making, path planning, obstacle avoidance, and vehicle control functions.

[0145] The action space module is configured to define feasible driving actions based on a driving decision model, including acceleration, deceleration, and steering, and construct an action space.

[0146] Specifically, the action space module defines the basic actions that the vehicle can perform, and selects one or more feasible actions for each state for further processing by the decision module.

[0147] The reward function module is configured to design a reward function to guide the optimization of driving decisions, including penalizing collisions and encouraging smooth driving.

[0148] Specifically, the reward function is used to quantify the pros and cons of each action. For example, a positive reward is assigned to safe driving behavior, such as obeying traffic rules, maintaining a safe distance, etc. A negative reward is assigned to unsafe behavior, such as approaching a collision. A positive reward is assigned to efficient driving behavior, such as quickly reaching the destination. A negative reward is assigned to inefficient driving behavior, such as unnecessary parking or deceleration. A positive reward is assigned to smooth driving behavior, such as smooth acceleration and / or steering. A negative reward is assigned to aggressive driving behavior, such as sudden acceleration or sudden braking. The reward is calculated at each time step to guide the training and decision-making of the reinforcement learning algorithm.

[0149] The reinforcement learning module is configured to train the reinforcement learning algorithm, optimize the driving decision-making model, and enable the driving decision-making model to select the optimal driving action in different states.

[0150] Specifically, the embodiment selects Q-learning as the reinforcement learning algorithm, which is suitable for discrete action space and can achieve optimal policy learning in unknown environments.

[0151] The training data set is constructed, including images, laser radar point cloud data, vehicle position and speed, and corresponding driving action labels in various driving scenarios. The reinforcement learning algorithm is trained iteratively. In each iteration, the driving decision-making model can select the optimal driving action based on the current state and the reward function, and execute the action to obtain the reward and the new state. The reinforcement learning algorithm can update the model parameters based on the reward and the new state.

[0152] It should be noted that through the state module, the action space module, the reward function module and the reinforcement learning module, the data fusion and processing results are converted into the state representation required for driving decision-making, and the space of driving actions is defined. By designing a suitable reward function and using a reinforcement learning algorithm for training, the driving decision-making model can be optimized to select the optimal driving action in different states. The optimization of this decision-making algorithm can improve the decision-making performance of the automatic driving obstacle avoidance system, enabling it to make more appropriate and safer driving decisions.

[0153] Further, the control module is configured to receive the decision-making instructions to control the autonomous vehicle and achieve obstacle avoidance.

[0154] Specifically, the control module receives decision instructions from the decision module, and based on the decision instructions, the control module sends control signals to the actuators of the vehicle (such as the engine, braking system, steering system, etc.), which can accurately control the acceleration, braking and steering behavior of the vehicle, so that the vehicle can safely and stably travel according to the planning of the decision module.

[0155] In summary, the automatic driving obstacle avoidance system of the embodiment of the application has the following beneficial effects:

[0156] 1. The automatic driving obstacle avoidance system of the application integrates the semantic segmentation results of the laser radar point cloud data and the street image through the data fusion module, the laser radar provides accurate distance and shape information, and the image semantic segmentation provides object category information, the combination of the two can improve the perception accuracy of the surrounding environment and provide more comprehensive environmental perception ability.

[0157] 2. The decision module of the automatic driving obstacle avoidance system of the application optimizes the driving decision model using the reinforcement learning algorithm, which can select the optimal driving action in different states and improve the efficiency and effect of decision making.

[0158] 3. The image processing module of the automatic driving obstacle avoidance system of the application can quickly react through precise and rapid semantic recognition and segmentation, thereby improving the safety of the automatic driving vehicle in the obstacle avoidance process, and through the hierarchical perception semantic segmentation network, the system can process complex street images and adapt to different driving environments and conditions.

[0159] 4. The semantic segmentation network used by the image processing module of the automatic driving obstacle avoidance system of the application can effectively fuse the hierarchical structure of the label, including the hierarchical dependency relationship and the hierarchical similarity relationship, by introducing the tree maximum and minimum loss function and the tree triplet loss function based on the constructed label tree during the training process. This fusion helps to further improve the accuracy of segmentation, so that the network can more accurately identify and segment different objects and scenes in the image.

[0160] The above is only the preferred specific embodiment of the application, but the protection scope of the application is not limited thereto, any person skilled in the art can easily think of changes or replacements within the technical range disclosed by the application, which should be covered within the protection scope of the application.

Claims

1. A hierarchical and precise obstacle avoidance system for autonomous driving, characterized in that, The autonomous driving obstacle avoidance system includes: a data acquisition module, an image processing module, a data fusion module, a decision-making module, and a control module; The data acquisition module includes sensors and a preprocessing module, used to acquire and preprocess driving environment data and vehicle data of the autonomous vehicle using sensors; wherein, the driving environment data includes: lidar point cloud data and continuous frame street view images captured during driving; the vehicle data includes the vehicle's current position and speed. The image processing module is used to perform hierarchical semantic segmentation on each frame of street view image to obtain the semantic segmentation results of key targets in each frame of street view image; The data fusion module is used to receive the lidar point cloud data and the image with semantic segmentation results obtained by the image processing module, perform data fusion to obtain comprehensive perception data, and perceive the environment around the autonomous vehicle. The decision-making module is used to receive the integrated perception data, formulate obstacle avoidance strategies, and generate decision instructions. The control module is used to receive the decision instructions to control the autonomous vehicle and achieve obstacle avoidance; In the image processing module, a pre-trained hierarchical perceptual semantic segmentation network is used for semantic segmentation; wherein, during the training of the semantic segmentation network, The street view semantic segmentation dataset is used as training samples; the street view semantic segmentation dataset includes a street view image set and a corresponding label set. A tag tree for street view semantics is constructed based on the hierarchical relationship of each tag in the tag set; Load the street view semantic segmentation dataset and the label tree of street view semantics, and train the semantic segmentation network and the semantic mapping network simultaneously using the tree minimax loss function and the tree triplet loss function. The hierarchical perception-based semantic segmentation network includes a feature extraction layer and a segmentation layer; the semantic mapping network includes a feature extraction layer and a mapping layer shared with the semantic segmentation network. The parameters of the feature extraction layer, segmentation layer, and mapping layer are updated using gradient backpropagation. After training, the parameters of the feature extraction layer, segmentation layer, and mapping layer are saved. Based on the trained feature extraction layer and segmentation layer, a semantic segmentation network is obtained.

2. The system according to claim 1, characterized in that, The data fusion module includes a camera radar calibration module, a feature extraction module, and a feature fusion module; The camera-radar calibration module uses a calibration board to obtain the camera's intrinsic parameters and extrinsic parameters relative to the lidar, and aligns the coordinates of the camera and lidar. The feature extraction module is used to extract the features of the lidar point cloud and the features of the image with semantic segmentation results, respectively. The feature fusion module is used to fuse the extracted lidar point cloud features and the image features to obtain comprehensive perception data with distance information and semantic segmentation results.

3. The system according to claim 2, characterized in that, The decision-making module includes: a state module, an action space module, a reward function module, and a reinforcement learning module; The state module is used to convert the comprehensive perception data into a state representation required for driving decisions, including vehicle position, speed, and surrounding environment information. The action space module is used to define feasible driving actions based on the driving decision model, including acceleration, deceleration, and steering, and to construct the action space. The reward function module is used to design reward functions to guide the optimization of driving decisions, including penalizing collisions and encouraging smooth driving. The reinforcement learning module is used to train the reinforcement learning algorithm and optimize the driving decision model, so that the driving decision model can select the optimal driving action under different conditions.

4. The system according to claim 1, characterized in that, The feature extraction layer based on the hierarchical perceptual semantic segmentation network is used to extract feature information of each frame of street view image; The segmentation layer of the hierarchical perceptual semantic segmentation network is used to perform pixel-level classification of each frame of images based on the feature information, so as to obtain the semantic segmentation results of key targets in each frame of images.

5. The system according to claim 4, characterized in that, The hierarchical perception-based semantic segmentation network was trained using the following method: Based on the street view semantic segmentation dataset and the street view semantic label tree, a semantic segmentation network and a semantic mapping network are trained simultaneously. The semantic mapping network is used to reorganize the features of the input image to obtain new feature vectors. During training, based on the similarity information of the labels in the street view semantic label tree, new feature vectors of similar categories are made closer to each other in the feature space, while new feature vectors of dissimilar categories are made farther apart.

6. The system according to claim 5, characterized in that, The segmentation layer includes a first convolutional layer, an activation function, a second convolutional layer, and a sigmoid function, used to perform semantic segmentation of the input image based on the number of nodes in the label tree of the street view semantics; wherein, The first convolutional layer has a size of 1×1×|V| and is used to perform convolution operations on the feature map to obtain transformed feature information; where |V| represents the number of nodes in the label tree of the street scene semantics. The activation function is the ReLU activation function, which is used to perform a nonlinear transformation on each feature to obtain the feature information after nonlinear transformation; The second convolutional layer has a size of 1×1×|V| and is used to perform convolution operation on the feature information after the nonlinear transformation. The sigmoid function is used to perform pixel-by-pixel classification probability estimation on the output features of the second convolutional layer to obtain the semantic probability distribution of each pixel.

7. The system according to any one of claims 5 or 6, characterized in that, The construction of a street view semantic tag tree based on the hierarchical relationship of each tag in the tag set includes: For each label, its parent class is found in the WordNet dataset, and the search is performed upwards layer by layer until the root node of the WordNet dataset is reached to obtain the hierarchical relationship of each label. Based on the hierarchical relationship of each label, a label tree T = {V, E} for street view semantics is constructed; where V represents the set of nodes in the label tree, which is the label class and its parent class in the WordNet dataset; and E represents the node relationship in the label tree.

8. The system according to claim 1, characterized in that, The maximum and minimum loss functions of the tree are expressed as follows: Where B represents the number of images in the street view semantic segmentation dataset; M represents the height of the image; N represents the width of the image; |V| represents the number of nodes in the label tree of the street view semantics; This represents the label value corresponding to the k-th class of the (i,j)-th pixel in the b-th image; Let represent the class probability value of the label value corresponding to the k-th class of the (i,j)-th pixel in the b-th image after undergoing tree minimax transformation.

9. The system according to claim 1, characterized in that, The tree triplet loss function is expressed as follows: Among them, D TT N represents the set of valid label triplet vectors; b This represents the number of triplets in the set of valid label triplet vectors; This represents the first feature vector in the set of vectors containing the s-th valid label triplet; This represents the second feature vector in the set of valid label triplet vectors; The vector represents the third feature vector in the set of valid label triplet vectors; d() represents the vector distance function; m s The optimal label margin of the s-th effective label triplet vector set is represented; three new feature vectors output by the mapping layer and their corresponding labels are randomly selected to form a triplet vector set; the triplet vector set is traversed, and the effective label triplet vector set is obtained based on the label tree.

Images