Road detection method, device, storage medium and electronic equipment

Abstract

The present invention discloses a road detection method, device, storage medium and electronic device, the method comprising: obtaining visual image data and pseudo radar image data containing roads; extracting features and performing feature interaction calculation on the visual image data and pseudo radar image data based on the dual channels in the dual channel convolution-coding stacking network, respectively, to obtain calculation results; detecting the calculation results and the features extracted by the dual channel convolution-coding stacking network based on the detection network to obtain the road segmentation result. By implementing the present invention, pseudo radar image data and visual image data are used to complete the segmentation detection of roads; the application cost and the power consumption of the corresponding equipment are greatly reduced; and in the dual channel convolution-coding stacking network, the features extracted by another channel can provide reference for the calculation of the current channel during calculation, and compared with the method of directly processing the image data, the detection accuracy is improved by feature extraction and feature interaction calculation.

Description

Technical Field

[0001] The present invention relates to the field of deep learning technology, and in particular to a road detection method, device, storage medium and electronic equipment. Background Art

[0002] Road detection technology is a key part of assisted driving and unmanned driving technology, which can provide necessary road environment information for the driver or vehicle's driving decision-making. Due to the complexity of the road environment, the results of road detection are easily affected by shadows, vehicles, and various complex traffic conditions. Therefore, it is very important to explore road detection algorithms that can accurately detect roads.

[0003] The goal of road detection is to give all pixels in the detection image that belong to drivable roads. The current road detection algorithms are mainly divided into pure visual image road detection algorithms and road detection algorithms that rely on radar data. Pure visual image road detection algorithms are simple to apply, but have low accuracy. Although the road detection algorithm that relies on radar data has high accuracy, there is a problem that data collection needs to rely on radar collection equipment. On the one hand, radar collection equipment is expensive and relies on radar data, which increases the cost of using the algorithm when it is applied. On the other hand, radar collection equipment has high power consumption. In actual autonomous driving applications, the use of radar equipment greatly increases the power consumption of the vehicle. Summary of the invention

[0004] In view of this, an embodiment of the present invention provides a road detection method, device, storage medium and electronic device to solve the technical problem that the road detection algorithm in the prior art relies on radar data and has a high cost.

[0005] The technical solution proposed by the present invention is as follows:

[0006] A first aspect of an embodiment of the present invention provides a road detection method, comprising: acquiring visual image data and pseudo radar image data containing a road; performing feature extraction and feature interaction calculation on the visual image data and the pseudo radar image data based on dual channels in a dual-channel convolution-coding stacked network to obtain calculation results; and detecting the calculation results and the features extracted by the dual-channel convolution-coding stacked network based on a detection network to obtain a road segmentation result.

[0007] Optionally, the dual-channel convolution-coding stacked network includes multiple stacked dual-channel convolution-coding modules, each dual-channel convolution-coding module includes a first channel and a second channel, the first channel includes a convolution module and a coding module arranged in sequence, and the structure of the second channel is the same as that of the first channel.

[0008] Optionally, based on the dual channels in the dual-channel convolution-coding stacked network, feature extraction and feature interaction calculation are performed on the visual image data and the pseudo radar image data respectively to obtain a calculation result, including: inputting the visual image data and the pseudo radar image data into the first channel and the second channel of the first dual-channel convolution-coding module respectively for feature extraction and feature interaction calculation to obtain a first calculation result of the first channel and a first calculation result of the second channel; inputting the first calculation result of the first channel and the features extracted from the first channel into the first channel of the second dual-channel convolution-coding module, and inputting the first calculation result of the second channel and the features extracted from the second channel into the second channel of the second dual-channel convolution-coding module, so that the second dual-channel convolution-coding module performs feature extraction and feature interaction calculation; and so on, inputting the calculation result and the extracted features of the previous dual-channel convolution-coding module into the next dual-channel convolution-coding module for feature extraction and feature interaction calculation, until the calculation result of the last dual-channel convolution-coding module is obtained.

[0009] Optionally, the visual image data and the pseudo radar image data are respectively input into the first channel and the second channel of the first dual-channel convolution-coding module for feature extraction and feature interaction calculation to obtain a first calculation result of the first channel and a first calculation result of the second channel, including: performing feature extraction on the visual image based on the convolution module of the first channel to obtain a first feature and a second feature; performing feature extraction on the pseudo radar image based on the convolution module of the second channel to obtain a third feature; and inputting the first feature, the second feature and the third feature together into the encoding module of the first channel and the encoding module of the second channel for calculation to obtain a first calculation result of the first channel and a first calculation result of the second channel.

[0010] Optionally, the first feature, the second feature and the third feature are jointly input into the encoding module of the first channel and the encoding module of the second channel for calculation to obtain a first calculation result of the first channel and a first calculation result of the second channel, including: performing full connection calculation on the first feature and the second feature to obtain a fully connected output; calculating the output of a single attention head based on the fully connected output and the third feature; based on a mapping function, merging the outputs of a single attention head to obtain the output of a multi-head attention mechanism; post-processing the output of the multi-head attention mechanism using a post-processing layer, the post-processing layer including a first batch normalization module and a second batch normalization module; calculating the first calculation result or the second calculation result based on the output of the multi-head attention mechanism and the post-processing result.

[0011] Optionally, the pseudo radar image data is obtained in the following manner: performing depth prediction on the visual image data according to a depth prediction network to obtain depth image data; generating pseudo radar image data according to the depth image data; or, the pseudo radar image data is obtained in the following manner: processing sparse radar image data based on a data completion model to obtain pseudo radar image data, wherein the data completion model includes multiple convolutional layers.

[0012] Optionally, the calculation results and the features extracted by the dual-channel convolution-coding stacking network are detected based on the detection network to obtain a road segmentation result, including: merging the calculation results and the features extracted by the dual-channel convolution-coding stacking network to obtain new features; and detecting and calculating the new features according to the convolution module in the detection network to obtain a road segmentation result.

[0013] A second aspect of an embodiment of the present invention provides a road detection device, comprising: a data acquisition module, used to acquire visual image data and pseudo radar image data containing a road; a feature extraction and interaction module, used to perform feature extraction and feature interaction calculation on the visual image data and pseudo radar image data based on the dual channels in a dual-channel convolution-coding stacked network, respectively, to obtain a calculation result; and a detection module, used to detect the calculation result and the features extracted by the dual-channel convolution-coding stacked network based on a detection network to obtain a road segmentation result.

[0014] A third aspect of an embodiment of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the road detection method as described in the first aspect of the embodiment of the present invention and any one of the first aspects.

[0015] A fourth aspect of an embodiment of the present invention provides an electronic device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the computer instructions to execute the road detection method as described in the first aspect of the embodiment of the present invention and any one of the first aspects.

[0016] The technical solution provided by the present invention has the following effects:

[0017] The road detection method, device, storage medium and electronic device provided by the embodiment of the present invention use pseudo radar image data and visual image data to complete road segmentation detection without relying on radar data; compared with the method of using radar image data for detection, the application cost is greatly reduced, and the power consumption of the corresponding equipment is also reduced; and the road detection method uses a dual-channel convolution-coding stacking network and a detection network to perform road segmentation detection. In the dual-channel convolution-coding stacking network, feature extraction and interactive calculation are realized. During calculation, the features extracted by another channel can provide a reference for the calculation of the current channel. Compared with the method of directly processing image data, the dual-channel convolution-coding stacking network is used to improve the detection accuracy through feature extraction and feature interactive calculation.

[0018] The road detection method provided by the embodiment of the present invention uses pseudo radar image data and visual image data to perform road detection, wherein the pseudo radar image data can be obtained by using a sparsely dotted radar device. Compared with the existing radar device that can obtain radar data, the sparsely dotted radar device reduces power consumption and cost; in addition, the pseudo radar image data can also be obtained from visual image data, and no radar device is used at all, thereby further reducing cost and power consumption. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0020] Figure 1 is a flow chart of a road detection method according to an embodiment of the present invention;

[0021] Figure 2 is a flow chart of a road detection method according to another embodiment of the present invention;

[0022] Figure 3 is a flow chart of a road detection method according to another embodiment of the present invention;

[0023] Figure 4 is a flow chart of a road detection method according to another embodiment of the present invention;

[0024] Figure 5 is a flow chart of a road detection method according to another embodiment of the present invention;

[0025] Figure 6 is a structural block diagram of a road detection device according to an embodiment of the present invention;

[0026] Figure 7 is a schematic diagram of the structure of a computer-readable storage medium provided according to an embodiment of the present invention;

[0027] Figure 8 is a schematic diagram of the structure of an electronic device provided according to an embodiment of the present invention. DETAILED DESCRIPTION

[0028] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0029] As mentioned in the background technology, currently, most of the road detection algorithms used are pure visual image road detection algorithms or road detection algorithms that rely on radar images. However, no matter which algorithm is used, there are certain defects. Therefore, a road detection algorithm with higher detection accuracy and lower cost is needed.

[0030] In view of this, an embodiment of the present invention provides a road detection method, which is based on pseudo radar data and uses a dual-channel network to perform road detection. It not only has low application cost, but also has high detection accuracy, overcoming the problem that existing algorithms rely on radar data and equipment. The road detection method can be applied to the field of autonomous driving and can also be applied to other scenarios where road detection is required. The embodiment of the present invention does not specifically limit the application scenario of the road detection method.

[0031] According to an embodiment of the present invention, a road detection method is provided. It should be noted that the steps shown in the flowchart of the accompanying drawings can be executed in a computer system such as a set of computer executable instructions, and although a logical order is shown in the flowchart, in some cases, the steps shown or described can be executed in an order different from that shown here.

[0032] In this embodiment, a road detection method is provided, which can be used in electronic devices such as computers, mobile phones, tablet computers, etc. Figure 1 is a flow chart of a road detection method according to an embodiment of the present invention. Figure 1 As shown, the method comprises the following steps:

[0033] Step S101: Obtain visual image data and pseudo radar image data containing roads. Specifically, since the radar acquisition equipment used to obtain radar data is costly and consumes a lot of power, pseudo radar data can be used for road detection. The visual image data can be RGB image data containing roads obtained by a vehicle-mounted camera, and the visual image data can characterize the road information in front of the vehicle. In addition, other reasonable methods can also be used to obtain visual image data, which is not limited in the embodiments of the present invention.

[0034] When a road detection algorithm that relies on radar image data is used, the radar equipment used needs to be densely dotted to collect dense radar information, that is, 256*256 image data needs to be obtained. Therefore, in order to reduce costs, sparse radar image data obtained by sparse radar equipment can be used to generate pseudo radar image data. Specifically, the sparse radar image data is processed based on the data completion model to generate pseudo radar image data. The data completion model can be composed of a convolutional neural network. In order to enable the convolutional neural network to generate pseudo radar image data, the convolutional neural network can be trained with radar image data in the early stage to determine the parameters of the convolutional neural network, thereby realizing the processing of pseudo radar image data by the convolutional neural network. Specifically, some sparse radar image data and radar image data can be obtained first to train the convolutional neural network. During training, the sparse radar image data is used as the input of the convolutional neural network, and the radar image data is used as the output of the convolutional neural network. The parameters of the convolutional neural network can be determined during the training process. Thus, the data completion model can be obtained by determining the parameters of the convolutional neural network model. In addition, the data completion model can also use other methods to realize the processing of pseudo radar image data, and the embodiment of the present invention is not limited to this.

[0035] In one embodiment, in order to further reduce costs, radar equipment may not be used at all, that is, a visual image acquired by a camera is used to generate a pseudo radar image. Thus, the pseudo radar image data is obtained by: performing depth prediction on the visual image data according to a depth prediction network to obtain depth image data; and generating pseudo radar image data according to the depth image data.

[0036] Among them, the depth prediction network can adopt an existing depth prediction network, such as a convolutional neural network. Specifically, the depth prediction network can be trained in the following way: obtain a certain number of RGB image data and its corresponding depth image data, use the RGB image data as the input of the convolutional neural network, and its corresponding depth image data as the output of the convolutional neural network, train the convolutional neural network, and determine the parameters of the convolutional neural network, thereby obtaining a depth prediction network. In actual use, the obtained RGB image is input into the depth prediction network to output the depth image data. After generating the depth image data, the following formula can be used to calculate to generate pseudo radar image data,

[0037]

[0038] In this formula, the parameter δ h Using formula Calculate, and the H in the formula (u,v) Using formula Among them, (u, v) represents the pixel coordinate position in the RGB image data, D (u，v) Indicates the depth map value at this position, PL (u，v) Represents the pseudo radar image value of the location, (N u , N v ) represents a certain neighborhood range of the pixel, usually a 7×7 neighborhood is sufficient. v represents the focal length of the camera, (C u , C v ) represents the origin coordinates of the camera imaging.

[0039] Step S102: performing feature extraction and feature interaction calculation on the visual image data and the pseudo radar image data based on the dual channels in the dual-channel convolution-coding stacked network to obtain calculation results.

[0040] For the acquired pseudo radar image data and visual image data, they can be input into a pre-trained road detection model for road detection, and the road detection result can be obtained by segmentation. In this embodiment, the pre-trained road detection model includes a dual-channel convolution-coding stacked network and a detection network. It should be noted that the road detection model can be trained with pseudo radar data image data and visual image data with calibrated segmentation results, and after the model converges, the parameters in the model are fixed to obtain a pre-trained road detection model. Specifically, the pre-training process includes: obtaining certain skilled pseudo radar data image data and visual image data, which are annotated with labels of segmentation results, and inputting the pseudo radar data image data and visual image data into the road detection model for training until the model converges to obtain the parameters therein. Thus, the pre-trained road detection model can be determined. In actual application scenarios, the acquired pseudo radar data image data and visual image data containing roads can be directly input into the pre-trained road detection model, and forward propagation can be performed to obtain the road segmentation detection result.

[0041] In one embodiment, the acquired pseudo radar image data and visual image data are first input into a dual-channel convolution-coding stacked network of a road detection model for feature extraction and feature interaction calculation; wherein, since the dual-channel convolution-coding stacked network has dual channels, the features extracted from the image of each channel can be sent to the other channel to realize feature interaction calculation, so that the features extracted by the other channel can provide a reference for the calculation of the current channel during calculation. Compared with the method of directly processing image data, the road detection model uses a dual-channel convolution-coding stacked network to improve the detection accuracy through feature extraction and feature interaction calculation.

[0042] Step S103: Based on the detection network, the calculation results and the features extracted by the dual-channel convolution-coding stacking network are detected to obtain a road segmentation result. Specifically, after the dual-channel convolution-coding stacking network performs feature extraction and feature interactive calculation on the pseudo radar image data and the visual image data, the extracted features and the corresponding calculation results can be combined and input into the detection network for segmentation detection, thereby obtaining the final road segmentation result.

[0043] The road detection method provided by the embodiment of the present invention adopts pseudo radar image data and visual image data, and can complete the segmentation detection of the road without relying on radar data; compared with the method of using radar image data for detection, the application cost is greatly reduced, and the power consumption of the corresponding equipment is also reduced; and the road detection method adopts a dual-channel convolution-coding stacking network and a detection network to perform road segmentation detection. In the dual-channel convolution-coding stacking network, feature extraction and interactive calculation are realized. During calculation, the features extracted by another channel can provide a reference for the calculation of the current channel. Compared with the method of directly processing image data, the dual-channel convolution-coding stacking network is adopted to improve the detection accuracy through feature extraction and feature interactive calculation.

[0044] The road detection method provided by the embodiment of the present invention uses pseudo radar image data and visual image data to perform road detection, wherein the pseudo radar image data can be obtained by using a sparsely dotted radar device. Compared with the existing radar device that can obtain radar data, the sparsely dotted radar device reduces power consumption and cost; in addition, the pseudo radar image data can also be obtained from visual image data, and no radar device is used at all, thereby further reducing cost and power consumption.

[0045] In one embodiment, if Figure 2 As shown, the dual-channel convolution-coding stacked network includes a plurality of stacked dual-channel convolution-coding modules, each of which includes a first channel and a second channel, the first channel includes a convolution module and a coding module arranged in sequence, and the structure of the second channel is the same as that of the first channel. Among them, the convolution module is composed of a convolutional neural network (CNN), and in this embodiment, the convolution module includes a multi-layer convolution network. The coding module is composed of a TFM (transformer model), and in this embodiment, the coding module mainly includes the coding structure of the TFM.

[0046] In one embodiment, since the dual-channel convolution-coding stacked network includes a plurality of stacked dual-channel convolution-coding modules, the input pseudo radar image data and visual image data can be sequentially passed through each dual-channel convolution-coding module for feature extraction and feature interaction calculation.

[0047] like Figure 2As shown, first, the visual image data and the pseudo radar image data are respectively input into the first channel and the second channel of the first dual-channel convolution-coding module for feature extraction and feature interactive calculation, so as to obtain the first calculation result of the first channel and the first calculation result of the second channel; then, the first calculation result of the first channel and the feature extracted by the first channel are input into the first channel of the second dual-channel convolution-coding module, and the first calculation result of the second channel and the feature extracted by the second channel are input into the second channel of the second dual-channel convolution-coding module, and feature extraction and feature interactive calculation are performed in the first channel and the second channel to obtain the second calculation result of the first channel and the second calculation result of the second channel; and by analogy, the calculation result and the extracted feature of the previous dual-channel convolution-coding module are input into the next dual-channel convolution-coding module for feature extraction and feature interactive calculation, until the calculation result of the last dual-channel convolution-coding module is obtained.

[0048] Among them, since multiple dual-channel convolution-encoding modules perform feature extraction and feature interaction calculation in sequence, that is, the results in the previous module will be input into the next module for extraction and calculation, the data processed in the subsequent modules will be more redundant than the data in the previous modules. Based on this feature, different numbers of convolution network layers can be set for the convolution modules in the dual-channel convolution-encoding module according to the specific data to be processed. For example, the number of convolution layers of the subsequent convolution modules gradually increases compared to the number of convolution layers in the previous ones, thereby realizing the processing of more redundant data.

[0049] In one embodiment, if Figure 3 As shown, the visual image data and the pseudo radar image data are respectively input into the first channel and the second channel of the first dual-channel convolution-encoding module for feature extraction and feature interaction calculation, and a first calculation result of the first channel and a first calculation result of the second channel are obtained, including the following steps:

[0050] Step S201: extract features from the visual image based on the convolution module of the first channel to obtain a first feature and a second feature. The features actually included in the first feature and the second feature may be the same. Specifically, after the convolution module extracts features from the visual image data, it may copy the extracted features to obtain two copies of features, namely, a first feature Q and a second feature K.

[0051] Step S202: extracting features from the pseudo radar image based on the convolution module of the second channel to obtain a third feature. The feature extracted from the pseudo radar image data by the convolution module of the second channel can be used as the third feature V.

[0052] Step S203: The first feature, the second feature and the third feature are inputted into the encoding module of the first channel and the encoding module of the second channel for calculation, and the first calculation result of the first channel and the first calculation result of the second channel are obtained. The three features Q, K and V extracted by the above convolution module can be used as the three inputs of the encoding module TFM in each channel, that is, the convolution module of the first channel not only inputs the features Q and K into the encoding module of the first channel, but also inputs them into the encoding module of the second channel; similarly, the encoding module of the second channel inputs the feature V into the encoding module of the second channel and the encoding module of the first channel. In this way, the interactive calculation of features is realized.

[0053] Specifically, since the encoding module of the first channel and the encoding module of the second channel both receive three features Q, K and V for calculation, and the first channel and the second channel have the same structure, the encoding modules of the two channels have the same calculation process. Figure 4 and Figure 5 As shown, the calculation process of the encoding module of a channel is explained:

[0054] Step S301: Perform full connection calculation on the first feature and the second feature to obtain a full connection output. Before the full connection calculation, the mapping matrix W of the first feature Q is calculated. Q and the mapping matrix W of the second feature K K Then, a full connection calculation is performed based on the first feature, the second feature, and the corresponding mapping matrix. The full connection calculation is expressed by the following formula:

[0055]

[0056] Where d represents the feature dimension. T Represents a transpose operation.

[0057] Step S302: Calculate the output of a single attention head based on the fully connected output and the third feature. When the encoding module adopts the TFM model, since the encoding structure of the TFM model adopts a multi-head attention mechanism, it is necessary to first calculate the output of a single attention head based on the fully connected output and the third feature.

[0058] Among them, the output of a single attention head is expressed by the following formula:

[0059]

[0060] Among them, head i Represents the output of an attention head. represents the learnable parameters, that is, the mapping matrix of the corresponding features, R represents a real number, and d represents the feature dimension.

[0061] Step S303: Based on the mapping function, the output of a single attention head is merged to obtain the output of the multi-head attention mechanism. After determining the output of a single attention head, the output of the single attention head can be merged through the mapping function W hC The output of the multi-head attention mechanism is obtained by merging the output of the single attention head with the mapping function W hC Multiplying them together gives the output of the multi-head attention mechanism. Therefore, the output of the multi-head attention mechanism can be expressed as: h n Represents the number of attention heads.

[0062] Step S304: Post-process the output of the multi-head attention mechanism using a post-processing layer, the post-processing layer including a first batch normalization module and a second batch normalization module. After calculating the output of the multi-head attention mechanism, the post-processing layer can be used to post-process it. The post-processing layer includes a first batch normalization module Linear_BN and a second batch normalization module Linear_ABN.

[0063] Among them, the first batch normalization module includes a linear mapping layer and a batch normalization layer (BatchNormalization, BN), the linear mapping layer is used for linear transformation, or linear superposition; the batch normalization layer uses the mean and standard deviation on the small batch to continuously adjust the intermediate output, so that the numerical value of the intermediate output of the entire first batch normalization module in each layer is more stable; the second batch normalization module includes a linear mapping layer, an activation function and a batch normalization layer, the linear mapping layer has the same function as above, and the setting of the activation function and the batch normalization layer can normalize the input value as much as possible in the narrow area of ​​the activation function. Specifically, through the processing of the first batch normalization module, the output of the multi-head attention mechanism will be converted from d feature dimensions to 2d feature dimensions, and then, through the processing of the second batch normalization module, the 2d feature dimension can be converted back to d feature dimensions.

[0064] Step S305: Calculate the first calculation result or the second calculation result according to the output of the multi-head attention mechanism and the post-processing result. The data processed by the post-processing layer can be element-wise added to the output of the multi-head attention mechanism, and the result of the element-wise addition is output as the calculation result of the encoding module.

[0065] In one embodiment, when the calculation results and the features extracted by the dual-channel convolution-coding stacked network are detected based on the detection network to obtain the road segmentation result, the calculation results of the two channel encoding modules in the last dual-channel convolution-coding module and the features extracted by the two channel convolution modules can be merged to obtain new features, and then the convolution module in the detection network detects and calculates the new features to obtain the road segmentation result. Specifically, in the above step 102, the road detection model has been obtained through training, that is, the parameters in the detection network have been determined. Therefore, the new features can be directly input into the detection network to obtain the road segmentation result.

[0066] The embodiment of the present invention also provides a road detection device, such as Figure 6 As shown, the device comprises:

[0067] The data acquisition module is used to acquire visual image data and pseudo radar image data containing roads; the specific content refers to the corresponding part of the above method embodiment, which will not be repeated here.

[0068] The feature extraction and interaction module is used to perform feature extraction and feature interaction calculation on the visual image data and the pseudo radar image data based on the dual channels in the dual-channel convolution-coding stacked network to obtain calculation results; the specific content refers to the corresponding part of the above method embodiment, which will not be repeated here.

[0069] The detection module is used to detect the calculation result and the features extracted by the dual-channel convolution-coding stack network based on the detection network to obtain the road segmentation result. For details, please refer to the corresponding part of the above method embodiment, which will not be repeated here.

[0070] The road detection device provided by the embodiment of the present invention adopts pseudo radar image data and visual image data, and can complete the segmentation detection of the road without relying on radar data; compared with the method of adopting radar image data detection, the application cost is greatly reduced, and the power consumption of the corresponding equipment is also reduced; and the road detection device adopts a dual-channel convolution-coding stacking network and a detection network to perform road segmentation detection. In the dual-channel convolution-coding stacking network, feature extraction and interactive calculation are realized. During calculation, the feature extracted by another channel can provide a reference for the calculation of the current channel. Compared with the method of directly processing image data, the dual-channel convolution-coding stacking network is adopted to improve the detection accuracy through feature extraction and feature interactive calculation.

[0071] For a detailed description of the functions of the road detection device provided in the embodiment of the present invention, please refer to the description of the road detection method in the above embodiment.

[0072] The embodiment of the present invention also provides a storage medium, such as Figure 7As shown, a computer program 601 is stored thereon, and when the instruction is executed by the processor, the steps of the road detection method in the above embodiment are implemented. The storage medium also stores audio and video stream data, feature frame data, interaction request signaling, encrypted data, and preset data size, etc. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory (Flash Memory), a hard disk (Hard Disk Drive, abbreviated: HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0073] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. Among them, the storage medium can be a disk, an optical disk, a read-only memory (ROM), a random access memory (RAM), a flash memory, a hard disk drive (HDD) or a solid-state drive (SSD), etc.; the storage medium can also include a combination of the above-mentioned types of memory.

[0074] The embodiment of the present invention further provides an electronic device, such as Figure 8 As shown, the electronic device may include a processor 51 and a memory 52, wherein the processor 51 and the memory 52 may be connected via a bus or other means. Figure 8 The example of connecting through bus is taken in the following.

[0075] The processor 51 may be a central processing unit (CPU). The processor 51 may also be other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field-programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or a combination of the above chips.

[0076] The memory 52 is a non-transitory computer-readable storage medium that can be used to store non-transitory software programs, non-transitory computer executable programs and modules, such as the corresponding program instructions / modules in the embodiment of the present invention. The processor 51 executes various functional applications and data processing of the processor by running the non-transitory software programs, instructions and modules stored in the memory 52, that is, the road detection method in the above method embodiment is implemented.

[0077] The memory 52 may include a program storage area and a data storage area, wherein the program storage area may store an application required for operating the device and at least one function; the data storage area may store data created by the processor 51, etc. In addition, the memory 52 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other non-volatile solid-state storage device. In some embodiments, the memory 52 may optionally include a memory remotely arranged relative to the processor 51, and these remote memories may be connected to the processor 51 via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0078] The one or more modules are stored in the memory 52 and when executed by the processor 51, perform the following steps: Figure 1 -5 is a road detection method in the embodiment shown in FIG.

[0079] For details of the above electronic equipment, please refer to Figures 1 to 5 The corresponding related descriptions and effects in the illustrated embodiments can be understood and will not be repeated here.

[0080] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A road detection method, characterized in that: include: Obtain visual image data and pseudo radar image data containing a road, wherein the pseudo radar image data is obtained by: performing depth prediction on the visual image data according to a depth prediction network to obtain depth image data; generating pseudo radar image data according to the depth image data; or, the pseudo radar image data is obtained by: processing sparse radar image data based on a data completion model to obtain pseudo radar image data, wherein the data completion model includes a plurality of convolutional layers; Based on the dual channels in the dual-channel convolution-coding stack network, feature extraction and feature interaction calculation are performed on the visual image data and the pseudo radar image data respectively to obtain calculation results; Based on the detection network, the calculation results and the features extracted by the dual-channel convolution-coding stacking network are detected to obtain a road segmentation result; The dual-channel convolution-coding stack network includes a plurality of dual-channel convolution-coding modules stacked in a stacked manner, each dual-channel convolution-coding module includes a first channel and a second channel, the first channel includes a convolution module and a coding module arranged in sequence, and the structure of the second channel is the same as that of the first channel; Based on the dual channels in the dual-channel convolution-coding stack network, feature extraction and feature interaction calculation are performed on the visual image data and the pseudo radar image data respectively to obtain calculation results, including: Inputting the visual image data and the pseudo radar image data into the first channel and the second channel of the first dual-channel convolution-encoding module respectively for feature extraction and feature interaction calculation, to obtain a first calculation result of the first channel and a first calculation result of the second channel; Inputting the first calculation result of the first channel and the features extracted by the first channel into the first channel of the second dual-channel convolution-encoding module, and inputting the first calculation result of the second channel and the features extracted by the second channel into the second channel of the second dual-channel convolution-encoding module, so that the second dual-channel convolution-encoding module performs feature extraction and feature interaction calculation; By analogy, the calculation results and extracted features of the previous dual-channel convolution-encoding module are input into the next dual-channel convolution-encoding module for feature extraction and feature interaction calculation, until the calculation results of the last dual-channel convolution-encoding module are obtained.

2. The road detection method according to claim 1, characterized in that: The visual image data and the pseudo radar image data are respectively input into the first channel and the second channel of the first dual-channel convolution-encoding module for feature extraction and feature interaction calculation, and a first calculation result of the first channel and a first calculation result of the second channel are obtained, including: Performing feature extraction on the visual image based on the convolution module of the first channel to obtain a first feature and a second feature; Performing feature extraction on the pseudo radar image based on the convolution module of the second channel to obtain a third feature; The first feature, the second feature, and the third feature are input into the encoding module of the first channel and the encoding module of the second channel for calculation, so as to obtain a first calculation result of the first channel and a first calculation result of the second channel.

3. The road detection method according to claim 2, characterized in that: The first feature, the second feature, and the third feature are inputted into the encoding module of the first channel and the encoding module of the second channel for calculation, to obtain a first calculation result of the first channel and a first calculation result of the second channel, including: Performing full connection calculation on the first feature and the second feature to obtain a full connection output; Calculate the output of a single attention head according to the fully connected output and the third feature; Based on the mapping function, the output of a single attention head is merged to obtain the output of the multi-head attention mechanism; Post-processing the output of the multi-head attention mechanism using a post-processing layer, wherein the post-processing layer includes a first batch normalization module and a second batch normalization module; The first calculation result or the second calculation result is calculated based on the output of the multi-head attention mechanism and the post-processing result.

4. The road detection method according to claim 1, characterized in that: The calculation results and the features extracted by the dual-channel convolution-coding stacking network are detected based on the detection network to obtain the road segmentation results, including: The calculation results and the features extracted by the dual-channel convolution-coding stacked network are combined to obtain new features; The new features are detected and calculated according to the convolution module in the detection network to obtain a road segmentation result.

5. A road detection device, characterized in that: include: A data acquisition module, used to acquire visual image data and pseudo radar image data containing a road, wherein the pseudo radar image data is acquired by: performing depth prediction on the visual image data according to a depth prediction network to obtain depth image data; generating pseudo radar image data according to the depth image data; or, the pseudo radar image data is acquired by: processing sparse radar image data based on a data completion model to obtain pseudo radar image data, wherein the data completion model includes a plurality of convolutional layers; A feature extraction and interaction module is used to perform feature extraction and feature interaction calculation on the visual image data and the pseudo radar image data based on the dual channels in the dual-channel convolution-coding stack network to obtain calculation results; A detection module, used for detecting the calculation result and the features extracted by the dual-channel convolution-coding stack network based on a detection network to obtain a road segmentation result; The dual-channel convolution-coding stack network includes a plurality of dual-channel convolution-coding modules stacked in a stacked manner, each dual-channel convolution-coding module includes a first channel and a second channel, the first channel includes a convolution module and a coding module arranged in sequence, and the structure of the second channel is the same as that of the first channel; Based on the dual channels in the dual-channel convolution-coding stack network, feature extraction and feature interaction calculation are performed on the visual image data and the pseudo radar image data respectively to obtain calculation results, including: Inputting the visual image data and the pseudo radar image data into the first channel and the second channel of the first dual-channel convolution-encoding module respectively for feature extraction and feature interaction calculation, to obtain a first calculation result of the first channel and a first calculation result of the second channel; Inputting the first calculation result of the first channel and the features extracted by the first channel into the first channel of the second dual-channel convolution-encoding module, and inputting the first calculation result of the second channel and the features extracted by the second channel into the second channel of the second dual-channel convolution-encoding module, so that the second dual-channel convolution-encoding module performs feature extraction and feature interaction calculation; By analogy, the calculation results and extracted features of the previous dual-channel convolution-encoding module are input into the next dual-channel convolution-encoding module for feature extraction and feature interaction calculation, until the calculation results of the last dual-channel convolution-encoding module are obtained.

6. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable the computer to execute the road detection method as described in any one of claims 1 to 4.

7. An electronic device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the road detection method according to any one of claims 1 to 4 by executing the computer instructions.

Images