Unstructured road passable area segmentation method and device, vehicle sensing system, computer equipment and computer readable storage medium
By using the first constraint network of the encoder-decoder structure and multi-sensor fusion technology in the unstructured road scenario, the problem of low detection accuracy of passable areas in the unstructured road scenario in the prior art is solved, and higher detection accuracy and stronger robustness are achieved.
Patent Information
- Application Number
- CN202311011332.5
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-08-11
- Publication Date
- 2025-02-25
AI Technical Summary
The prior art is difficult to detect passable areas with high accuracy in unstructured road scenarios, and traditional image detection algorithms are difficult to obtain sufficient accuracy in such complex scenarios.
The first constraint network with an encoder-decoder structure is adopted, and combined with RGB images and depth images, the passable area of the unstructured road scene is initially segmented. Then, in the pre-trained segmentation model, the detection accuracy is further improved through feature extraction and multi-sensor fusion.
It effectively improves the passable area detection accuracy of the vehicle perception system in unstructured road scenarios, improves the robustness and environmental adaptability of the perception system, and provides accurate and reliable road perception information for the intelligent driving system.
Smart Images
Figure CN119516180A_ABST
Abstract
Description
Technical Field
[0001] The present invention belongs to the field of intelligent driving technology, and specifically relates to a method and device for segmenting a drivable area of an unstructured road, a vehicle perception system, a computer device, and a computer-readable storage medium. Background Art
[0002] In the current research on autonomous driving technology, compared with urban structured road scenes, relatively little research has been invested in the detection of drivable areas in unstructured road scenes such as rural roads and off-road roads. Most of the current autonomous driving data sets are regular structured road data collected in urban scenes and under good perception conditions. Compared with regular urban structured roads, the road boundaries in unstructured road scenes are irregular, the road type attributes are variable, and the sensor working environment is often more severe. It is very difficult to collect a sufficiently rich and comprehensive unstructured road data set. For the drivable area detection task in such complex scenes, traditional image detection algorithms often have difficulty in obtaining sufficient accuracy and are difficult to apply to different scenes. Summary of the invention
[0003] The technical problem to be solved by the embodiments of the present invention is to provide a method, device, vehicle perception system, computer equipment and computer-readable storage medium for segmenting unstructured road passable areas to improve the detection accuracy of unstructured road passable areas.
[0004] In order to solve the above technical problems, the present invention provides a method for segmenting a drivable area of an unstructured road, comprising:
[0005] Acquire RGB images and depth images of unstructured road scenes;
[0006] Inputting the RGB image and the depth image into a pre-trained first constraint network to perform preliminary segmentation on the drivable area of the unstructured road scene;
[0007] In the pre-trained segmentation model, feature extraction is performed on the drivable area of the initially segmented unstructured road scene;
[0008] The segmentation result of the unstructured road passable area processed by the segmentation model is output.
[0009] Preferably, the first constraint network adopts an encoder-decoder structure, the encoder network performs feature extraction and downsampling operations on the input RGB image and depth image, and performs feature fusion; the decoder network restores the features extracted by the encoder network through the upsampling layer, and extracts features from the fused features output by the encoder network through the feature extraction module. The parameters of the first constraint network remain fixed after training is completed.
[0010] Preferably, the pre-trained segmentation model includes a segmentation network and a second constraint network, the segmentation network and the first constraint network adopt the same model structure, and the second constraint network adopts the downsampling network structure and parameters of the first constraint network.
[0011] Preferably, the joint loss function of the segmentation model is:
[0012] L total (I rgb ,I d )=L dist (I rgb ,I d )+L seg (I rgb ,I d )
[0013] Among them, L seg is the loss function of the segmentation network for the annotation label, L dist is a loss function that characterizes the feature offset distance between the segmentation network and the second constraint network; (I rgb , I d ) represents the input image sample, I rgb is the input RGB image, I d is the input depth image.
[0014] Preferably, the loss function characterizing the feature offset distance between the segmentation network and the second constraint network is as follows:
[0015]
[0016] Where G and M represent the second constraint network and the segmentation network feature extractor network channels respectively. For the same set of input image samples (I rgb , I d ), G(I rgb ,I d )、M(I rgb ,I d ) represent the feature maps extracted by the second constraint network and the segmentation network respectively; h represents the feature map height, w represents the feature map width, c represents the number of feature map channels; C represents the number of pixel categories.
[0017] Preferably, the loss function L of the segmentation network for the annotation label is seg The cross entropy loss function is used to represent the input image sample (I rgb , I d )’s predicted segmentation result and the difference between the label value:
[0018]
[0019] Where M and N are the image width and height respectively; I ij is the indicator function; p ij Output prediction results for the network Sigmoid function.
[0020] The present invention also provides a computer device, comprising:
[0021] one or more processors;
[0022] Memory;
[0023] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method for segmenting the unstructured road drivable area.
[0024] The present invention also provides a computer-readable storage medium, which includes a stored computer program; wherein, when the computer program is running, it controls the device where the computer-readable storage medium is located to execute the method for segmenting the unstructured road passable area.
[0025] The present invention also provides a device for segmenting a traversable area of an unstructured road, comprising:
[0026] An acquisition module, used to acquire RGB images and depth images of unstructured road scenes;
[0027] A preliminary segmentation module, used for inputting the RGB image and the depth image into a pre-trained first constraint network to perform preliminary segmentation on the drivable area of the unstructured road scene;
[0028] A feature extraction module, used for extracting features of the drivable area of the initially segmented unstructured road scene in the pre-trained segmentation model;
[0029] The output module is used to output the segmentation result of the unstructured road passable area after being processed by the segmentation model.
[0030] The present invention also provides a vehicle perception system, characterized in that it comprises the device for segmenting the traversable area of the unstructured road.
[0031] The implementation of the present invention has the following beneficial effects: the present invention can effectively improve the detection accuracy of the vehicle perception system for the passable area of unstructured road scenes, realize accurate perception of the road area, and improve the robustness and environmental adaptability of the perception system, provide accurate and reliable road perception information input for the intelligent driving system, and improve the reliability and safety of intelligent driving in unstructured road scenes. The transfer learning method is used to improve the problems of low model accuracy and poor scene adaptability caused by insufficient data sample size and low data scene coverage in unstructured road scenes. At the same time, the depth image data collected by the lidar is introduced on the basis of the perception scheme based on RGB image data, and the multi-dimensional information input is provided to the model through a multi-sensor fusion scheme, thereby improving the model performance and robustness in unstructured road scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0032] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0033] Figure 1 It is a flowchart of a method for segmenting a drivable area of an unstructured road according to an embodiment of the present invention.
[0034] Figure 2 Schematic diagram of the model structure of the first constraint network in an embodiment of the present invention.
[0035] Figure 3 Schematic diagram of the structure of the traversable area segmentation model in an embodiment of the present invention. DETAILED DESCRIPTION
[0036] The following descriptions of the embodiments refer to the accompanying drawings to illustrate specific embodiments in which the present invention may be implemented.
[0037] Please refer to Figure 1 As shown, the first embodiment of the present invention provides a method for segmenting a drivable area of an unstructured road, comprising:
[0038] Acquire RGB images and depth images of unstructured road scenes;
[0039] Inputting the RGB image and the depth image into a pre-trained first constraint network to perform preliminary segmentation on the drivable area of the unstructured road scene;
[0040] In the pre-trained segmentation model, feature extraction is performed on the drivable area of the initially segmented unstructured road scene;
[0041] The segmentation result of the unstructured road passable area processed by the segmentation model is output.
[0042] Specifically, the RGB image of the unstructured road scene is collected by the vehicle camera, and the depth image is collected by the laser radar. In the embodiment of the present invention, the first constraint network is pre-built and trained. Figure 2 As shown in the figure, the first constraint network adopts an encoder-decoder structure. In addition to the RGB image, its input also includes a depth image obtained from the laser radar to represent the distance information, so as to improve the model performance and the algorithm environment adaptability. Among them, the encoder network performs feature extraction and downsampling operations on the input, and performs feature fusion of the RGB image and the depth image. The encoder network part includes two network channels, the RGB image encoder network and the depth image encoder network. The RGB image encoder and the depth image encoder are both composed of an initial layer and four residual network downsampling modules. The initial layer is composed of a convolutional layer, a batch normalization layer, and a ReLU activation layer, and the number of output channels of the convolutional layer is set to 64, so that the image features and the depth features have the same dimension to facilitate subsequent feature extraction and feature fusion operations. After each residual feature extraction layer extracts the image and depth features, the RGB image features and the depth features are fused by the MSE feature fusion method that considers the correlation between feature channels. The fused features are sent to the next layer of residual network module as the next input of the image feature extraction network.
[0043] In the decoder network, the features extracted by the encoder network are restored through the upsampling layer, which is the same as the decoder network of the general semantic segmentation model. Each upsampling module includes a bilinear upsampling layer and a convolution layer with a convolution kernel size of 3×3. The upsampling layer increases the resolution of the feature map layer by layer and reduces its number of channels, and finally restores it to the resolution size of the original input image. The output classification result of each pixel is obtained by the sigmoid function. In addition, an additional feature extraction module is added to the decoder network to extract features from the fused features of the encoder, and is connected to the upsampling layer through dense skip connections. The feature extraction module uses a convolution layer with the same convolution kernel size of 3×3 as the upsampling module, and ensures that its output channel dimension is the same as the upsampling channel dimension of the corresponding layer.
[0044] In order to train the first constraint network, a source domain urban structured road dataset is used for pre-training. This dataset contains sufficiently large-scale urban road scene data. Using this dataset for training can obtain an optimal constraint network model that can extract and learn the characteristics and constraint rules of urban road scenes. During the training process, by adjusting the weights and biases of the network, the first constraint network is gradually optimized and obtains a good understanding of the road's passable area. After the training is completed, the obtained constraint network parameters will be saved and kept fixed in subsequent model training without being updated or adjusted. The purpose of this architecture and training method is to extract and fuse the features of RGB images and depth images using the encoder-decoder structure, and to capture road constraint rules through the pre-trained constraint network. Such a combination can effectively improve the segmentation accuracy of the passable area of unstructured road scenes and ensure that the segmentation results meet the physical constraints in the actual scene.
[0045] Transfer learning is a machine learning technique that can apply knowledge learned in one domain (source domain) to another related domain (target domain). Domain adaptation refers to adjusting the model to adapt to the characteristics of the target task so that the knowledge learned in the source domain can be well used in the target domain. In this embodiment, domain adaptation will be used for the traversable area segmentation task.
[0046] like Figure 3 As shown in the figure, the segmentation model consists of two parts: the segmentation network and the second constraint network. The segmentation network is responsible for performing the actual road segmentation task, while the second constraint network is mainly used to provide additional constraint information during the training process of the segmentation network to help the segmentation network perform more accurate segmentation tasks. The segmentation network and the pre-trained first constraint network use the same model structure, and the second constraint network retains the downsampling network structure and parameters of the first constraint network. In other words, the second constraint network shares the downsampling network structure and parameters with the first constraint network, and the knowledge learned by the first constraint network in the encoder stage can be shared with the second constraint network, that is, transfer learning is performed.
[0047] During the training process of the segmentation model, both the second constraint network and the segmentation network receive the same road data input (i.e. the target scene collected above - unstructured road scene road data). The segmentation network should minimize the distance between its feature distribution and the feature distribution of the second constraint network, so that the segmentation network has more similarities with the already trained constraint network, and improve the model's generalization ability for new scenes. The optimization goal of the training process includes not only reducing the prediction error of the sample label, but also considering reducing the difference between the source domain and the target domain features. In this way, the parameters of the segmentation network can be optimized and updated while considering the label error and domain shift error.
[0048] In order to achieve the goal of shortening the feature distribution distance of different task fields and making the segmentation network learn the common knowledge about road features in the second constraint network, the embodiment of the present invention adds the loss function L of the segmentation network for the annotation label. seg On the basis of , we add a loss function L that represents the offset distance between the segmentation network and the second constraint network features. dist .
[0049] In the original segmentation network model, its loss function L seg The cross entropy loss function is used to represent the input image sample (I rgb , I d )’s predicted segmentation result and the difference between the label value:
[0050]
[0051] Where M and N are the image width and height respectively; C represents the number of pixel categories; I rgb is an RGB image, I d is the depth image; I ij is the indicator function; p ij Output prediction results for the network Sigmoid function.
[0052] On this basis, a second constraint network is added to constrain the distribution offset of network features. G and M represent the network channels of the second constraint network and the segmentation network feature extractor, respectively. For the same set of input image samples (I rgb , I d ), the second constraint network and the segmentation network extract the feature map G(I rgb ,I d )、M(I rgb ,I d ). Use the L1 distance between two feature maps to characterize their domain shift loss:
[0053]
[0054] In the formula, h represents the height of the feature map (Height), w represents the width of the feature map (Width), and c represents the number of channels of the feature map (Channel);
[0055] Therefore, the joint loss function for constructing the migration model training is:
[0056] L total (I rgb ,I d )=L dist (I rgb ,I d )+L seg (I rgb ,Id )
[0057] The weight update of the model in training based on the domain adaptation transfer learning method can reduce the error between the model output result value and the sample label value, while reducing the difference between the encoder network output features and the constraint network output features, thereby achieving the purpose of migrating the common feature extraction knowledge of road surface data in the constraint network to the encoder feature extraction network.
[0058] After completing the constraint network training, the constraint network is fixed and the segmentation network is trained through the loss function constructed above to obtain the final traversable area segmentation model.
[0059] The final drivable area segmentation model is deployed to a real vehicle and used as a perception module for autonomous driving vehicles in unstructured road scenarios.
[0060] Corresponding to the method for segmenting the unstructured road drivable area described in the first embodiment of the present invention, the second embodiment of the present invention provides a computer device, including:
[0061] one or more processors;
[0062] Memory;
[0063] One or more applications, wherein the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more applications are configured to execute the method for segmenting the unstructured road passable area as described in Example 1 of the present invention.
[0064] Corresponding to the method for segmenting the unstructured road passable area described in the aforementioned embodiment 1 of the present invention, embodiment 3 of the present invention also provides a computer-readable storage medium, wherein the computer-readable storage medium includes a stored computer program; wherein, when the computer program is running, the device where the computer-readable storage medium is located controls the method for segmenting the unstructured road passable area as described in embodiment 1 of the present invention.
[0065] Preferably, the computer program can be divided into one or more modules / units (such as computer program 1, computer program 2, ...), and the one or more modules / units are stored in the memory and executed by the processor to complete the present invention. The one or more modules / units can be a series of computer program instruction segments that can complete specific functions, and the instruction segments are used to describe the execution process of the computer program in the device.
[0066] The processor may be a central processing unit (CPU), or other general-purpose processors, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor may be a microprocessor, or the processor may be any conventional processor. The processor is the control center of the device, and various parts of the device are connected using various interfaces and lines.
[0067] The memory mainly includes a program storage area and a data storage area, wherein the program storage area can store an operating system, an application required for at least one function, etc., and the data storage area can store related data, etc. In addition, the memory can be a high-speed random access memory, or a non-volatile memory, such as a plug-in hard disk, a smart memory card (SmartMedia Card, SMC), a secure digital (Secure Digital, SD) card, and a flash card (Flash Card), etc., or the memory can also be other volatile solid-state storage devices.
[0068] It should be noted that the above-mentioned device may include but is not limited to a processor and a memory, which can be understood by those skilled in the art.
[0069] Corresponding to the method for segmenting an unstructured road traversable area described in the first embodiment of the present invention, the fourth embodiment of the present invention further provides a device for segmenting an unstructured road traversable area, including:
[0070] An acquisition module, used to acquire RGB images and depth images of unstructured road scenes;
[0071] A preliminary segmentation module, used for inputting the RGB image and the depth image into a pre-trained first constraint network to perform preliminary segmentation on the drivable area of the unstructured road scene;
[0072] A feature extraction module, used for extracting features of the drivable area of the initially segmented unstructured road scene in the pre-trained segmentation model;
[0073] The output module is used to output the segmentation result of the unstructured road passable area after being processed by the segmentation model.
[0074] Corresponding to the device for segmenting the unstructured road passable area described in the fourth embodiment of the present invention, the fifth embodiment of the present invention further provides a vehicle perception system, characterized in that it includes the device for segmenting the unstructured road passable area described in the fourth embodiment of the present invention.
[0075] For the working principle and process of the above embodiment, please refer to the description of the above embodiment of the present invention, which will not be repeated here.
[0076] It can be seen from the above description that compared with the prior art, the beneficial effects of the present invention are: the present invention can effectively improve the detection accuracy of the vehicle perception system for the passable area of the unstructured road scene, realize the accurate perception of the road area, and improve the robustness and environmental adaptability of the perception system, provide accurate and reliable road perception information input for the intelligent driving system, and improve the reliability and safety of intelligent driving in unstructured road scenes. The transfer learning method is used to improve the problems of low model accuracy and poor scene adaptability caused by insufficient data sample size and low data scene coverage in unstructured road scenes. At the same time, on the basis of the perception scheme based on RGB image data, the depth image data collected by the lidar is introduced, and the multi-dimensional information input is provided to the model through a multi-sensor fusion scheme, which improves the model performance and robustness in unstructured road scenes.
[0077] The above disclosure is only the preferred embodiment of the present invention, which certainly cannot be used to limit the scope of the present invention. Therefore, equivalent changes made according to the claims of the present invention are still within the scope of the present invention.
Claims
1. A method for segmenting a traversable area of an unstructured road, characterized in that: include: Acquire RGB images and depth images of unstructured road scenes; Inputting the RGB image and the depth image into a pre-trained first constraint network to perform preliminary segmentation on the drivable area of the unstructured road scene; In the pre-trained segmentation model, feature extraction is performed on the drivable area of the initially segmented unstructured road scene; The segmentation result of the unstructured road passable area processed by the segmentation model is output.
2. The method according to claim 1, characterized in that: The first constraint network adopts an encoder-decoder structure. The encoder network performs feature extraction and downsampling operations on the input RGB image and depth image, and performs feature fusion; the decoder network restores the features extracted by the encoder network through the upsampling layer, and extracts features from the fused features output by the encoder network through the feature extraction module. The parameters of the first constraint network remain fixed after training is completed.
3. The method according to claim 2, characterized in that The pre-trained segmentation model includes a segmentation network and a second constraint network. The segmentation network and the first constraint network use the same model structure, and the second constraint network uses the downsampling network structure and parameters of the first constraint network.
4. The method according to claim 3, characterized in that The joint loss function of the segmentation model is: L total (I rgb ,I d )=L dist (I rgb ,I d )+L seg (I rgb ,I d ) Among them, L seg is the loss function of the segmentation network for the annotation label, L dist is a loss function that characterizes the feature offset distance between the segmentation network and the second constraint network; (I rgb , I d ) represents the input image sample, I rgb is the input RGB image, I d is the input depth image.
5. The method according to claim 4, characterized in that The loss function that characterizes the feature offset distance between the segmentation network and the second constraint network is as follows: Where G and M represent the second constraint network and the segmentation network feature extractor network channels respectively. For the same set of input image samples (I rgb , I d ), G(I rgb ,I d )、M(I rgb ,I d ) represent the feature maps extracted by the second constraint network and the segmentation network respectively; h represents the feature map height, w represents the feature map width, c represents the number of feature map channels; C represents the number of pixel categories.
6. The method according to claim 5, characterized in that The loss function L of the segmentation network for the annotation label seg The cross entropy loss function is used to represent the input image sample (I rgb , I d )’s predicted segmentation result and the difference between the label value: Where M and N are the image width and height respectively; I ij is the indicator function; p ij Output prediction results for the network Sigmoid function.
7. A computer device, characterized in that: include: one or more processors; Memory; One or more applications, wherein the one or more applications are stored in the memory and are configured to be executed by the one or more processors, and the one or more applications are configured to execute the method for segmenting the unstructured road passable area as described in any one of claims 1 to 6.
8. A computer-readable storage medium, characterized in that: The computer-readable storage medium includes a stored computer program; wherein, when the computer program is run, it controls the device where the computer-readable storage medium is located to execute the method for segmenting the unstructured road passable area according to any one of claims 1 to 6.
9. A device for segmenting a traversable area of an unstructured road, characterized in that: include: An acquisition module, used to acquire RGB images and depth images of unstructured road scenes; A preliminary segmentation module, used for inputting the RGB image and the depth image into a pre-trained first constraint network to perform preliminary segmentation on the drivable area of the unstructured road scene; A feature extraction module, used for extracting features of the drivable area of the initially segmented unstructured road scene in the pre-trained segmentation model; The output module is used to output the segmentation result of the unstructured road passable area after being processed by the segmentation model.
10. A vehicle perception system, characterized in that: The device comprises the device for dividing the unstructured road traversable area as claimed in claim 9.