A road grid map construction method and system with real and simulated data enhancement

Abstract

The present invention aims to obtain semantic classification of visible categories within the current range through semantic understanding of images, and on this basis, predict the blocked road area to obtain the complete road area. The problem that it is difficult to obtain accurate complete road area detection effect by training the model with real training data alone is solved by joint training of simulation data and real data. On this basis, a road grid map is constructed to represent the complete road conditions in front of the vehicle.

Description

Technical Field

[0001] The present invention relates to the field of autonomous driving technology, and in particular to a training method for an unobstructed road detection model based on real and simulated data enhancement, a road grid map construction system, and a road recognition method. Background Art

[0002] Road detection is one of the important technologies in the field of intelligent vehicle environment perception. Vision-based methods mainly use the color and texture features of RGB images to achieve pixel-level classification and obtain visible road areas. Most of the current leading road segmentation algorithms are based on deep learning methods.

[0003] Pixel-level classification based on the color and texture features of RGB images can only obtain visible road areas. In practical applications, a complete road area representation can better express the road distribution around the vehicle and better serve the planning and control module. However, since roads are often blocked by objects on the road, some road areas are invisible. In addition, the performance of deep learning-based methods is closely related to the amount of training data, and obtaining training data by annotating real-world data is time-consuming and labor-intensive. Summary of the invention

[0004] The present invention aims to obtain semantic classification of visible categories within the current range through semantic understanding of images, and on this basis, predict the blocked road area to obtain the complete road area. The problem that it is difficult to obtain accurate complete road area detection effect by training the model with real training data alone is solved by joint training of simulation data and real data. On this basis, a road grid map is constructed to represent the complete road conditions in front of the vehicle. The specific technical solution is as follows.

[0005] As a first aspect, the present invention provides a method for constructing a road grid map enhanced by real and simulated data, characterized in that the steps include:

[0006] S1, constructing a real training dataset and a simulated training dataset for training an unobstructed road segmentation model;

[0007] S2, build an unobstructed road detection model;

[0008] S3, training the unobstructed road detection model according to the real training data set and the simulation training data set;

[0009] S4, constructing a road grid map according to the training results;

[0010] S5, deploying the road grid map construction method described in steps S1-S4 on the vehicle side.

[0011] Among them, as a further optimization of the above steps, each step is optimized as follows:

[0012] S1, construct training data for training the unobstructed road segmentation model, including:

[0013] Obtain image data in real-world driving scenarios, obtain real-world data through manual annotation, and construct a real training data set, denoted as D r ={(I r ,S r ,L r )|I r ∈Q H×W×3 ,S r ∈Q H×W×C ,L r ∈Q H×W×2}, where I r is an RGB image; S r is the corresponding semantic segmentation annotation result, C is the number of annotated semantic categories; L r is the corresponding road segmentation annotation result; H and W are the height and width of the image respectively;

[0014] By constructing a variety of driving scenarios in the simulation environment, the data collected by the same vehicle at the same location in a static environment and a dynamic traffic environment are compared to obtain the annotation results in different dynamic scenarios, and a simulation training data set is constructed, denoted as D s ={(I s ,S s ,L s )|I s ∈Q H×W×3 ,S s ∈Q H×W×C ,L s ∈Q H×W×2}, where I s is an RGB image; S s is the corresponding semantic segmentation annotation result, C is the number of annotated semantic categories; L s The corresponding road segmentation and labeling results;

[0015] S2, build an unobstructed road detection model, the input information form of the detection task is RGB image I∈Q H×W×3 The one-hot encoding of the semantic segmentation labels or the predicted semantic segmentation probability tensor S∈[0,1] H×W×C , where H is the width of the image, W is its height, and C is the number of semantic categories; the output of the detection task is a tensor O∈[0,1] H×W×2 , contains the probability of each pixel belonging to the road and non-road categories, regardless of whether the road is occluded;

[0016] S3, using the constructed real training data set and the simulated training data set to train the unobstructed road detection model constructed in step S2, the algorithm regards road detection as a pixel-level classification problem, sets training hyperparameters, and then sets an adaptive distance estimation Adam optimizer, using a cross entropy loss function as an objective function to improve the convergence result of the model, and obtain unobstructed road detection data;

[0017] S4, based on the unobstructed road detection data, a road grid map is constructed through inverse perspective transformation and the camera's external parameters.

[0018] In combination with the first aspect, in a first case where any of the cases may occur, constructing a real training data set includes the following steps:

[0019] Collecting image data in real-world driving scenarios r ∈Q H×W×3 ,The image data contains the color and texture information of the ,driving scenes under different locations, different weather conditions, different lighting conditions, etc.;

[0020] Manually annotate the semantic information of the collected data and construct the semantic label S r ∈Q H×W×C ;

[0021] The road segmentation label L of the collected data is manually annotated r ∈Q H×W×2 ,For the marked pixels, regardless of whether the pixels are occluded by the foreground objects, they are marked as road categories, and the rest are non-road categories;

[0022] Building a simulation training dataset includes the following steps:

[0023] Construct a simulated dynamic driving scene and a virtual vehicle with image acquisition function, control the vehicle to drive in the simulated dynamic driving scene, the dynamic driving scene refers to the virtual scene that also includes other virtual traffic participants, and record the image and position information of the virtual vehicle in the simulated environment. dy ={(I dy ,S dy ,G dy )};

[0024] Construct a simulated static driving scene, which is a simulated environment after removing dynamic traffic participants and foreground targets blocking the road from the simulated dynamic driving scene, and record the data of the virtual vehicle during driving, i.e., D st ={(I st ,L st ,G st )};

[0025] Matching is performed based on the position information. For data frames where the position error is less than the threshold, a simulation training data set D is constructed. s ={(I dy ,S dy ,L st )}.

[0026] In combination with the first aspect or the first situation above, in the second situation in any of the situations that may occur, the constructed unobstructed road detection model includes an input layer, a downsampling layer, a residual convolution layer, an upsampling layer and an output layer, wherein:

[0027] The input layer is the semantic segmentation result of the RGB image;

[0028] The downsampling layer downsamples the input features through a context convolution module with a step size of m. The downsampling layer adds a context branch and is parallel to the main convolution. First, the attention weight is obtained by using n×n convolution and Softmax function, and then attention pooling is performed to obtain the global context features of the feature layer. The global context features are then transformed through n×n convolution and added to the feature map of the main convolution.

[0029] Residual convolution, which reduces the vanishing gradient by adding skip connections at both ends of the convolution operation and extracts deeper features;

[0030] In the upsampling layer, the two feature maps are combined using the context convolution module and the bilinear upsampling operation. The two branches of the two feature maps are first merged along the channel, and the merged feature map is processed by a context convolution module.

[0031] In combination with the first aspect or the first and second situations, in a third situation in any of the possible situations, the objective function in step S3 is: L total =λ1L real +λ2L syth , where L total is the total loss, L real and L syth are the losses of real data samples and simulated data samples, respectively, and λ1 and λ2 are their weight values;

[0032] L real and L syth The cross entropy loss function is adopted, and its formula is: Among them, y is the true value, p is the predicted value, i and j are the coordinates of the pixel, H and W are the height and width of the image, respectively.

[0033] According to the designed objective function, the Adam optimizer is used to optimize the parameters of the unobstructed road detection model. First, the gradient of the t time step is calculated: Calculate the exponential moving average m of the gradient t , m0 = 0, and the weight distribution of momentum and current gradient is controlled by exponential decay rate β1, β1 is a preset value, and its more explicit formula is: m t =β1m t-1 +(1-β1)g t ;

[0034] Second, calculate the exponential moving average v of the squared gradient t , v0 = 0, β2 = 0.999 is the influence of controlling the square of the gradient at the previous moment, and the formula is as follows:

[0035] Again, for m t Correction of deviation Reduce the impact of bias in the early stages of training;

[0036] Then, for v t Correction is performed to reduce the impact of deviation on the initial stage of training. The formula is: Finally, update the parameters and set the default learning rate α = 0.001, ε = 10 -8 To avoid division by 0, the formula is:

[0037] In combination with the first aspect or the first, second, and third situations above, in a fourth situation in any of the situations that may occur, the method of step S4 is:

[0038] In the vehicle coordinate system, construct a grid map G={c ij |i=1,...h,j=1,...,w}, take the center coordinates of each cell as the index point, find its corresponding coordinates in the image pixel coordinates through conversion operation, and use its segmentation category as the classification category of the cell, that is, whether it belongs to road or non-road type, and assign unknown type to cells without corresponding pixels.

[0039] The conversion operation is:

[0040] Assume that the road is flat, that is, z v = 0, the next point [x c ,y c ,z c ] T The transformation relationship to the two-dimensional image coordinate system is:

[0041]

[0042] Among them, u and v are the horizontal and vertical coordinates of the corresponding pixel of the point in the image; d x and d y is the physical size of a pixel in the X and Y axes of the camera; f is the focal length of the camera.

[0043] In the process of constructing the grid map, the vehicle coordinate system is used as the reference. For a point [x v ,y v ,z v ] T , its corresponding point [x c ,y c ,z c ] T The conversion relationship is:

[0044]

[0045] Among them, R and t are the rotation matrix and translation vector from the vehicle coordinate system to the camera coordinate system, respectively.

[0046] In combination with the first aspect or one of the first to fourth cases mentioned above, the fifth case in any possible case is that the step size m is 2, the n×n convolution is 1×1 convolution; the region sizes of x∈[n1,m1] and y∈[n2,m2] are constructed as: x∈[6,50], y∈[-15,15]; the grid map with a resolution of n×n is: a grid map with a resolution of 0.1×0.1; β1 is taken as 0.9.

[0047] As a second aspect, the present invention discloses a method for road identification, the steps of the method are as follows:

[0048] Step 1, deploying the road grid map detection model trained by the method, which includes a trained unobstructed road detection model and a road grid map generation post-processing algorithm;

[0049] Step 2: collect the target road image in real time, and preprocess the collected image. The preprocessing method is to standardize the pixel values ​​of the collected image. The method is shown in the following formula:

[0050] x'=(x-μ) / σ

[0051] Among them, x is the pixel value, μ is its mean, and σ is its standard deviation;

[0052] Step 3, input the standardized processing results into the deployed unobstructed road detection model;

[0053] Step 4, detecting the road area ahead in real time, obtaining the probability of each pixel in the image belonging to the road or non-road category, and constructing a road grid map through step S4 of the method;

[0054] Step 5: According to the road grid map, identify the roads in the target road area and output them.

[0055] As a third aspect, the present invention provides a road grid map construction system enhanced by real and simulated data, the system comprising an image acquisition system, a training data set module, and an unobstructed road detection model, wherein:

[0056] An image acquisition system is used to acquire image information of a target road, perform manual annotation preprocessing on the acquired image information, and provide it for invocation by a training data set module; acquire images of a target road in real time, and perform preprocessing on the acquired images; detect the road area ahead in real time, and obtain the probability that each pixel in the image belongs to a road or non-road category, and the preprocessing result or the probability is provided for invocation by an unobstructed road detection model;

[0057] The training data set module obtains manually annotated image information and constructs a real training data set; obtains a variety of driving scenarios in a simulation environment, compares the data collected by the same vehicle at the same location in a static environment and a dynamic traffic environment, obtains annotated image information in different dynamic scenarios, and constructs a simulation training data set; the above training data set is used by the unobstructed road detection model;

[0058] The unobstructed road detection model is trained based on the above training data set to construct a road raster map.

[0059] As a fourth aspect, the present invention discloses a computer-readable storage medium storing one or more programs, wherein the computer-readable storage medium stores one or more program instructions, and when the program instructions are executed by a processor, any one of the above-mentioned methods is performed.

[0060] The beneficial effects of the present invention are:

[0061] 1. Through joint training of simulation data and real data, the problem that it is difficult to obtain accurate detection results of complete road areas by training models with real training data alone is solved. On this basis, a road grid map is constructed to represent the complete road conditions in front of the vehicle, providing effective road perception results for the planning and control system.

[0062] It can obtain the complete road area and reason about the road area blocked by foreground objects, so as to better serve the planning and control subsystem in the autonomous driving system. BRIEF DESCRIPTION OF THE DRAWINGS

[0063] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0064] Figure 1 It is a schematic diagram of the method flow of the present invention;

[0065] Figure 2 The road segmentation model framework of the present invention;

[0066] Figure 3 It is a logical schematic diagram of the context convolution module of the present invention;

[0067] Figure 4 It is a logical schematic diagram of the upsampling module of the present invention;

[0068] Figure 5 It is a logic diagram of the residual convolution module of the present invention;

[0069] Figure 6 A schematic diagram of a transparent viewing angle of a grid map of the present invention;

[0070] Figure 7 It is a schematic diagram of a top-down perspective of a grid map of the present invention. DETAILED DESCRIPTION

[0071] The technical solution of the present invention is further specifically described below through embodiments and in conjunction with the accompanying drawings. Obviously, the described embodiments are only partial embodiments of the present invention.

[0072] Example 1

[0073] like Figure 1 As shown, the present invention provides a method for constructing a road grid map enhanced by real and simulated data, and the method steps are as follows:

[0074] 1. Construction of training dataset

[0075] In order to achieve the purpose of detecting the complete road area, training data for training the unobstructed road segmentation model is first constructed.

[0076] On the one hand, we use the on-board camera to obtain image data in real-world driving scenarios, and then obtain real-world data through manual annotation, referred to as the real training data set, denoted as D r ={(I r ,S r ,L r )|I r ∈Q H×W×3,S r ∈Q H×W×C ,L r ∈Q H×W×2}, where I r is an RGB image; S r is the corresponding semantic segmentation annotation result, C is the number of annotated semantic categories; L r The corresponding road segmentation annotation result; H and W are the height and width of the image respectively.

[0077] On the other hand, in order to make up for the low efficiency of manual labeling and obtain sufficient training data, a variety of driving scenarios are constructed in the simulation environment. The data collected by the same vehicle at the same location are compared in static and dynamic traffic environments to obtain the labeling results in different dynamic scenarios. The constructed simulation training data set is denoted as D s ={(I s ,S s ,L s )|I s ∈Q H×W×3 ,S s ∈Q H×W×C ,L s ∈Q H×W×2}, where I s is an RGB image; S s is the corresponding semantic segmentation annotation result, C is the number of annotated semantic categories; L s The corresponding road segmentation labeling result.

[0078] Specifically, constructing a real training dataset includes the following steps:

[0079] (1) Using on-board cameras to collect image data in real-world driving scenarios r ∈Q H×W×3 ,The image data contains information such as color and texture of driving scenes under different ,locations, different weather conditions, different lighting conditions and ,other conditions.

[0080] (2) Manually label the semantic information of the collected data and construct the semantic label S r ∈Q H×W×C , where the annotated categories include most of the common and concerned categories in driving scenes, such as trees, pedestrians, cars, and roads.

[0081] (3) Manually label the road segmentation labels L of the collected data r ∈Q H×W×2 ,For the marked pixels, regardless of whether the pixel is occluded by the foreground object, it is marked as the road category, and the rest are non-road categories.

[0082] Specifically, constructing a simulation training data set includes the following steps:

[0083] (1) Construct a dynamic driving scene and a virtual vehicle equipped with a camera sensor in the simulation software, and control the vehicle to drive in the simulation environment. The dynamic driving scene refers to the virtual scene that also includes other virtual traffic participants, and records the image and position information of the virtual vehicle in the simulation environment. dy ={(I dy ,S dy ,G dy )}.

[0084] (2) Construct a static driving scene in the simulation software. The static driving scene is the simulation environment after removing dynamic traffic participants and foreground objects blocking the road from the dynamic driving scene in step (1). Record the data of the virtual vehicle during driving, i.e., D st ={(I st ,L st ,G st )}.

[0085] (3) Matching is performed based on the position information. For data frames where the position error is less than the threshold, a simulation training data set D is constructed. s ={(I dy ,S dy ,L st )}.

[0086] 2. Construction of unobstructed road detection model

[0087] The unobstructed road detection task is a pixel-level classification task like the traditional road segmentation task, but includes the occlusion reasoning process to obtain a complete representation of the road area. The input information of the task is in the form of an RGB image I∈Q H×W×3 The one-hot encoding of the semantic segmentation labels or the predicted semantic segmentation probability tensor S∈[0,1] H×W×C , where H is the width of the image, W is its height, and C is the number of semantic categories. The output of the unobstructed road detection task is a new tensor O∈[0,1] H×W×2 , contains the probability of each pixel belonging to the road and non-road classes, regardless of whether the road is occluded or not.

[0088] like Figure 2 As shown in the figure, the unobstructed road detection model constructed includes modules such as input layer, downsampling layer, residual convolution layer, upsampling layer and output layer. Among them:

[0089] The input layer is the semantic segmentation result of the RGB image.

[0090] The downsampling layer downsamples the input features through a context convolution module with a step size of 2. The downsampling layer is as follows: Figure 3 As shown in the figure, a context branch is added in parallel with the main convolution. First, 1×1 convolution and Softmax function are used to obtain the attention weight, and then attention pooling is performed to obtain the global context features of the feature layer; then the global context features are transformed through 1×1 convolution and added to the feature map of the main convolution.

[0091] The residual convolution module is as follows Figure 5 As shown in the figure, it alleviates the gradient vanishing problem by adding skip connections at both ends of the convolution operation, which is conducive to extracting deeper features.

[0092] The upsampling layer is Figure 4 As shown in the figure, it uses a context convolution module and a bilinear upsampling operation to effectively combine the two feature maps. First, the two branches of the two feature maps are merged along the channel, and the merged feature map is processed by a context convolution module.

[0093] In the entire model, the upsampling modules are cascaded from deep features to shallow features. The current upsampling module utilizes the results of the previous level and the corresponding decoding features, making the upsampling operation more efficient.

[0094] 3. Training the Unobstructed Road Segmentation Model

[0095] The unobstructed road detection model constructed in step 2 is trained using the training data set constructed in step 1. The algorithm regards road detection as a pixel-level classification problem, sets training hyperparameters, and then sets the adaptive distance estimation Adam optimizer. The cross entropy loss function is used as the objective function to improve the convergence result of the model. The adaptive distance estimation optimizer is used to improve the loss descent path and improve the performance of the training model. The objective function used is:

[0096] L total =λ1L real +λ2L syth ;

[0097] Among them, L total is the total loss, L real and L syth are the losses of real data samples and simulated data samples, respectively, and λ1 and λ2 are their weight values. real and L syth The cross entropy loss function is adopted, and its formula is as follows:

[0098]

[0099] Among them, y is the true value, p is the predicted value, i and j are the coordinates of the pixel, H and W are the height and width of the image, respectively.

[0100] According to the designed objective function, the Adam optimizer is used to optimize the parameters of the unobstructed road detection model.

[0101] Compute the gradient at time step t:

[0102]

[0103] First, calculate the exponential moving average m of the gradient at time step t t , m0 = 0, and the weight distribution of momentum and current gradient is controlled by exponential decay rate β1, β1 is set to 0.9, and the more explicit formula is as follows:

[0104] m t =β1m t-1 +(1-β1)g t ;

[0105] Second, calculate the exponential moving average v of the squared gradient t , v0 = 0, β2 = 0.999 is the influence of controlling the square of the gradient at the previous moment. The formula is as follows:

[0106]

[0107] Again, for m t Perform deviation correction to reduce the impact of deviation on the initial stage of training. The formula is as follows:

[0108]

[0109] Then, for v t Correction is performed to reduce the impact of deviation on the initial stage of training. The formula is as follows:

[0110]

[0111] Finally, update the parameters and set the default learning rate α = 0.001, ε = 10 -8 To avoid division by 0, the formula is as follows:

[0112]

[0113] 4. Road Grid Map Generation

[0114] Based on the unobstructed road detection obtained in step three, a road grid map is constructed through inverse perspective transformation and the camera's external parameters.

[0115] First, in the vehicle coordinate system, a grid map G = {c ij|i=1,...h,j=1,...,w}. The center coordinates of each cell are used as index points, and the corresponding coordinates in the image pixel coordinates are found through conversion operations, and the segmentation category is used as the classification category of the cell, that is, whether it belongs to the road or non-road type. For cells without corresponding pixels, they are assigned an unknown type.

[0116] The conversion operation is:

[0117] Assume that the road is flat, that is, z v = 0, the next point [x c ,y c ,z c ] T The transformation relationship to the two-dimensional image coordinate system is:

[0118]

[0119] Among them, u and v are the horizontal and vertical coordinates of the corresponding pixel of the point in the image; d x and d y is the physical size of a pixel in the X and Y axes of the camera; f is the focal length of the camera.

[0120] In the process of constructing the grid map, the vehicle coordinate system is used as the reference. For a point [x v ,y v ,z v ] T , its corresponding point [x c ,y c ,z c ] T The conversion relationship is:

[0121]

[0122] Among them, R and t are the rotation matrix and translation vector from the vehicle coordinate system to the camera coordinate system, respectively.

[0123] Example 2

[0124] This embodiment provides a method for road recognition, which uses the trained road grid map detection model described in Example 1 combined with real-time collected road images to construct a grid map, thereby identifying the road. The steps are as follows:

[0125] The road grid map detection model and its algorithm are deployed on the vehicle controller, which includes the trained unobstructed road detection model and the road grid map generation algorithm. The vehicle camera collects the road image in front of the vehicle, and the collected image is preprocessed. The preprocessing is to standardize the pixel values ​​of the input image. The method is shown in the following formula:

[0126] x'=(x-μ) / σ;

[0127] Among them, x is the pixel value, μ is its mean, and σ is its standard deviation.

[0128] Then, the image preprocessing results are input into the deployed unobstructed road detection model to detect the road area ahead in real time and obtain the probability that each pixel in the image belongs to the road or non-road category.

[0129] On this basis, a road grid map is constructed by the road grid map construction method described in step 4 of Example 1, and roads in the target road area are identified and output. Figure 6 The grid map perspective and Figure 7 Grid map shown from overhead perspective.

[0130] It should be understood that the above embodiments are only used to illustrate the present invention and are not used to limit the scope of the present invention. In addition, it should be understood that after reading the content of the present invention, those skilled in the art can make various changes and modifications to the present invention, and these equivalent forms also fall within the scope limited by the appended claims of the present application.

Claims

1. A method for constructing a road grid map enhanced by real and simulated data, characterized in that: The method comprises: S1, constructing a real training dataset and a simulated training dataset for training an unobstructed road segmentation model; S2, build an unobstructed road detection model; S3, training the unobstructed road detection model according to the real training data set and the simulation training data set; S4, constructing a road grid map according to the training results; S5, deploying the road grid map construction method described in steps S1-S4 on the vehicle side; The objective function in step S3 is: L total =λ1L real +λ2L syth , where L total is the total loss, L real and L syth are the losses of real data samples and simulated data samples, respectively, λ1 and λ2 are weight values; L real and L syth The cross entropy loss function is adopted, and the formula is: ， Among them, y is the true value, p is the predicted value, i and j are the coordinates of the pixel, H and W are the height and width of the image respectively; According to the designed objective function, the Adam optimizer is used to optimize the parameters of the unobstructed road detection model. First, the gradient of the t time step is calculated: g t =▽ θ J(θ t-1 ), calculate the exponential moving average m of the gradient t , m0 = 0, and the weight distribution of momentum and current gradient is controlled by exponential decay rate β1, β1 is the preset value, and the update formula is: t =β1m t-1 +(1-β1)g t ; Second, calculate the exponential moving average v of the squared gradient t , v0=0,β2=0.999,the formula is as follows: t =β2v t-1 +(1-β2)g 2 t ; Again, for m t To correct the deviation, the formula is: ； Then, for v t To correct the deviation, the formula is: ； Finally, update the parameters and set the default learning rate α = 0.001, ε = 10 -8 , the formula is: 。 2. The method for constructing a road grid map enhanced by real and simulated data according to claim 1, characterized in that: The method comprises: S1, constructing a real training dataset and a simulated training dataset for training the unobstructed road segmentation model, including: Obtain image data in real-world driving scenarios, obtain real-world data through manual annotation, and construct a real training data set, denoted as D r ={(I r ,S r ,L r )|I r ∈Q H×W×3 ,S r ∈Q H×W×C , L r ∈Q H×W×2 }, where I r is an RGB image; S r is the corresponding semantic segmentation annotation result, C is the number of annotated semantic categories; L r is the corresponding road segmentation annotation result; H and W are the height and width of the image respectively; By constructing a variety of driving scenarios in the simulation environment, the data collected by the same vehicle at the same location in a static environment and a dynamic traffic environment are compared, the annotation results in different dynamic scenarios are obtained, and a simulation training data set is constructed, which is denoted as D s ={(I s ,S s ,L s )|I s ∈Q H×W×3 ,S s ∈Q H×W×C ,L s ∈Q H×W×2 }, where I s is an RGB image; S s is the corresponding semantic segmentation annotation result, C is the number of annotated semantic categories; L s The corresponding road segmentation labeling result; S2, build an unobstructed road detection model, the input information form of the detection task is RGB image I∈Q H×W×3 The one-hot encoding of the semantic segmentation label or the predicted semantic segmentation probability tensor S∈[0,1] H×W×C , where H is the width of the image, W is the height of the image, and C is the number of semantic categories; the output of the detection task is a tensor O∈[0,1] H×W×2 , contains the probability of each pixel belonging to the road and non-road categories, regardless of whether the road is occluded; S3, using the constructed real training data set and the simulated training data set to train the unobstructed road detection model constructed in step S2, treating road detection as a pixel-level classification problem, setting training hyperparameters, and then setting an adaptive distance estimation Adam optimizer, using a cross entropy loss function as an objective function to improve the convergence result of the model, and obtaining unobstructed road detection data; S4, based on the unobstructed road detection data, a road grid map is constructed through inverse perspective transformation and the camera's external parameters.

3. The method for constructing a road grid map enhanced by real and simulated data according to claim 2, characterized in that: Building a real training dataset includes the following steps: Collecting image data in real-world driving scenarios r ∈Q H×W×3 ,The image data contains the color and texture information of driving scenes in different locations, different weather and ,lighting conditions; Manually annotate the semantic information of the collected data and construct the semantic label S r ∈Q H×W×C ; The road segmentation label L of the collected data is manually annotated r ∈Q H×W×2 ,For the marked pixels, regardless of whether the pixels are occluded by the foreground objects, they are marked as road categories, and the rest are non-road categories; Building a simulation training dataset includes the following steps: Construct a simulated dynamic driving scene and a virtual vehicle with image acquisition function, control the vehicle to drive in the simulated dynamic driving scene, the dynamic driving scene refers to a virtual scene that also includes other virtual traffic participants, and record the image and position information of the virtual vehicle in the simulated environment. dy ={(I dy ,S dy ,G dy )}; Construct a simulated static driving scene, which is a simulated environment after removing dynamic traffic participants and foreground targets blocking the road from the simulated dynamic driving scene, and record the data of the virtual vehicle during driving, i.e., D st ={(I st ,L st ,G st )}; Matching is performed based on the position information. For data frames where the position error is less than the threshold, a simulation training data set D is constructed. s ={(I dy ,S dy ,L st )}.

4. The method for constructing a road grid map enhanced by real and simulated data according to claim 3, characterized in that: The constructed unobstructed road detection model includes an input layer, a downsampling layer, a residual convolution layer, an upsampling layer, and an output layer, where: The downsampling layer downsamples the input features through a context convolution module with a step size of m. The downsampling layer adds a context branch and is parallel to the main convolution. First, the attention weight is obtained by using n×n convolution and Softmax function, and then attention pooling is performed to obtain the global context features of the feature layer. The global context features are then transformed through n×n convolution and added to the feature map of the main convolution. The residual convolution reduces the gradient disappearance by adding jump connections at both ends of the convolution operation and extracts deeper features. In the upsampling layer, the two feature maps are combined using the context convolution module and the bilinear upsampling operation. The two branches of the two feature maps are first merged along the channel, and the merged feature map is processed by a context convolution module.

5. The method for constructing a road grid map enhanced by real and simulated data according to claim 4, characterized in that: The method of step S4 is: In the vehicle coordinate system, construct a grid map G={c ij |i=1 ,...h ,j=1 ,...,w}, take the center coordinates of each cell as the index point, find its corresponding coordinates in the image pixel coordinates through conversion operation, and use its segmentation category as the classification category of the cell, that is, whether it belongs to the road or non-road type. For cells without corresponding pixels, they are assigned an unknown type; The conversion operation is: Assume that the road is flat, that is, z v = 0, the next point [x c ,y c ,z c ] T The transformation relationship to the two-dimensional image coordinate system is: ， Among them, u and v are the horizontal and vertical coordinates of the corresponding pixel of the point in the image; d x and d y is the physical size of a pixel in the X-axis and Y-axis directions of the camera; f is the focal length of the camera; in the process of building a grid map, the vehicle coordinate system is used as the reference, and for a point [x v ,y v ,z v ] T , its corresponding point [x c ,y c ,z c ] T The conversion relationship is: ， Among them, R and t are the rotation matrix and translation vector from the vehicle coordinate system to the camera coordinate system, respectively.

6. The method for constructing a road grid map enhanced by real and simulated data according to any one of claims 4-5, characterized in that: The step size m is 2, and the n×n convolution is 1×1 convolution; the region sizes of x∈[n1,m1] and y∈[n2,m2] are constructed as follows: x∈[6,50], y∈[-15,15]; the grid map with a resolution of n×n is: the grid map with a resolution of 0.1×0.1; The value of β1 is 0.

9.

7. A method for road recognition, characterized in that: The steps of the method are as follows: Step 1, deploying an unobstructed road detection model trained by the method described in any one of claims 1-6; Step 2: collect the target road image in real time and preprocess the collected image. The preprocessing method is to standardize the pixel values ​​of the collected image, as shown in the following formula: x'=(x-μ) / σ, where x is the pixel value, μ is the mean, and σ is the standard deviation; Step 3, input the standardized processing results into the deployed unobstructed road detection model; Step 4, detecting the road area ahead in real time, obtaining the probability that each pixel in the image belongs to the road or non-road category, and constructing a road grid map through step S4 of the method described in any one of claims 1 to 6; Step 5: According to the road grid map, identify the roads in the target road area and output them.

8. A road grid map construction system enhanced by real and simulated data, used to execute the road grid map construction method enhanced by real and simulated data as claimed in any one of claims 1 to 6, characterized in that: The system includes an image acquisition system, a training data set module, and an unobstructed road detection model, wherein: An image acquisition system is used to acquire image information of a target road, perform manual annotation preprocessing on the acquired image information, and provide it for invocation by a training data set module; acquire images of a target road in real time, and perform preprocessing on the acquired images; detect the road area ahead in real time, and obtain the probability that each pixel in the image belongs to a road or non-road category, and the preprocessing result or the probability is provided for invocation by an unobstructed road detection model; The training data set module obtains manually annotated image information and constructs a real training data set; constructs a variety of driving scenarios in a simulation environment, and compares the annotated image information of different dynamic scenarios collected by the same vehicle at the same location in a static environment and a dynamic traffic environment to construct a simulation training data set; the above training data set is used by the unobstructed road detection model; The unobstructed road detection model is trained based on the above training data set to construct a road raster map.

9. A computer-readable storage medium storing one or more programs, characterized in that: The computer-readable storage medium stores one or more computer programs, and when the computer programs are executed by the processor, the method described in any one of claims 1 to 6 is performed.

Images