A method and device for detecting a drivable area

Abstract

The present invention provides a method and device for detecting a drivable area, wherein the method comprises: obtaining a road traffic image using a monocular camera, constructing an initial tensor containing a color channel according to the obtained road traffic image; obtaining coordinate information of each pixel point in the road traffic image for each road traffic image, adding the coordinate information of the pixel point to the color channel corresponding to the pixel point, and obtaining a tensor to be processed corresponding to the road traffic image; inputting the tensor to be processed into a pre-trained neural network containing an attention module, and obtaining a feature tensor representing a drivable area and a non-drivable area; constructing the outline of the drivable area according to the pixel points representing the drivable area in the feature tensor, and controlling the vehicle to travel within the outline of the drivable area. The detection accuracy of the drivable area can be improved.

Description

Technical Field

[0001] The present invention relates to the field of road traffic technology, and in particular to a drivable area detection method and device. Background Art

[0002] The drivable area detection technology is a key technology in the assisted driving system and the autonomous driving system. It is used to detect whether a certain area in the vehicle driving environment is a drivable area or a non-drivable area, and provide technical support for the driving route for assisted driving and autonomous driving. Among them, the non-drivable area mainly includes static obstacles on the road, such as areas where cones and guardrails are located, and dynamic obstacles, such as areas where vehicles and pedestrians are located, curbs on both sides of the road, and construction areas in the middle of the road.

[0003] At present, the drivable area detection technology generally adopts a drivable area detection method based on the disparity image of a binocular camera. However, this drivable area detection method requires the use of a binocular camera to capture two left and right images, and obtain a disparity image based on the two left and right images, so as to perform feasible area detection based on the disparity image. Since the disparity image is obtained based on the two left and right images, the calibration accuracy of the binocular camera is required to be high, which makes the equipment cost for drivable area detection high and the detection process complicated. The bumps and shakes during vehicle driving will affect the detection accuracy, resulting in low drivable area detection accuracy, affecting vehicle driving. For example, the original drivable area is detected as a non-drivable area, making the vehicle unable to drive, causing traffic congestion. Summary of the invention

[0004] In view of this, an object of the present invention is to provide a drivable area detection method and device to improve the detection accuracy of the drivable area.

[0005] In a first aspect, an embodiment of the present invention provides a drivable area detection method, comprising:

[0006] A monocular camera is used to obtain a road traffic image, and an initial tensor including a color channel is constructed based on the obtained road traffic image;

[0007] For each road traffic image, obtain the coordinate information of each pixel in the road traffic image, add the coordinate information of the pixel to the color channel corresponding to the pixel, and obtain the tensor to be processed corresponding to the road traffic image;

[0008] Inputting the tensor to be processed into a pre-trained neural network including an attention module to obtain a feature tensor representing a drivable area and a non-drivable area;

[0009] According to the pixel points representing the drivable area in the feature tensor, the outline of the drivable area is constructed, and the vehicle is controlled to travel within the outline of the drivable area.

[0010] In combination with the first aspect, an embodiment of the present invention provides a first possible implementation of the first aspect, wherein the neural network is a semantic segmentation network, and the tensor to be processed is input into a pre-trained neural network including an attention module to obtain a feature tensor representing a drivable area and a non-drivable area, including:

[0011] Inputting the tensor to be processed into the first sublayer in the first branch layer in the semantic segmentation network to obtain a first tensor;

[0012] Inputting the first tensor into the second sublayer in the first branch layer and the first subbranch layer in the second branch layer respectively, to obtain a second tensor and a third tensor, wherein the second sublayer includes: a third convolutional layer, a third batch normalization layer, a third activation function layer, a first basic module and a second basic module in sequence, and the first subbranch layer includes: a fourth convolutional layer, a fourth batch normalization layer, a fourth activation function layer, a third basic module and a fourth basic module in sequence;

[0013] Input the second tensor and the third tensor into the first feature fusion layer to obtain the fourth tensor and the fifth tensor;

[0014] Input the fourth tensor into the fifth basic module in the first branch layer, input the output of the fifth basic module into the sixth basic module to obtain a sixth tensor, input the fifth tensor into the seventh basic module in the second branch layer, input the output of the seventh basic module into the eighth basic module to obtain a seventh tensor, input the fifth tensor into the first branch sub-layer in the third branch layer to obtain an eighth tensor, wherein the first branch sub-layer sequentially includes: a fifth convolutional layer, a fifth batch normalization layer, a fifth activation function layer, a ninth basic module, and a tenth basic module;

[0015] Input the sixth tensor, the seventh tensor, and the eighth tensor into the second feature fusion layer to obtain the ninth tensor, the tenth tensor, and the eleventh tensor respectively;

[0016] Input the ninth tensor into the eleventh basic module in the first branch layer, input the output of the eleventh basic module into the twelfth basic module, and obtain the twelfth tensor; input the tenth tensor into the thirteenth basic module in the second branch layer, input the output of the thirteenth basic module into the fourteenth basic module, and obtain the thirteenth tensor; input the eleventh tensor into the fifteenth basic module in the third branch layer, input the output of the fifteenth basic module into the sixteenth basic module, and obtain the fourteenth tensor; input the eleventh tensor into the fourth branch layer, and obtain the fifteenth tensor; the fourth branch layer sequentially includes: a sixth convolutional layer, a sixth batch normalization layer, a sixth activation function layer, a seventeenth basic module, and an eighteenth basic module;

[0017] The twelfth tensor, the thirteenth tensor, the fourteenth tensor, and the fifteenth tensor are input into the third feature fusion layer to obtain the sixteenth tensor, the seventeenth tensor, the eighteenth tensor, and the nineteenth tensor respectively;

[0018] Input the tensor output by the third feature fusion layer into the tensor concatenation layer to obtain a concatenated tensor;

[0019] The concatenated tensor is input into the seventh convolutional layer, and the output of the seventh convolutional layer is input into the eighth convolutional layer to obtain the output tensor;

[0020] Perform Argmax operation on the output tensor along the 0th dimension to obtain the tensor to be processed, and upsample the tensor to be processed to obtain the feature tensor.

[0021] In combination with the first possible implementation manner of the first aspect, an embodiment of the present invention provides a second possible implementation manner of the first aspect, wherein the step of inputting the tensor to be processed into the first sublayer in the first branch layer in the semantic segmentation network to obtain the first tensor includes:

[0022] Inputting the tensor to be processed into the first convolutional layer for convolution processing;

[0023] The output of the first convolutional layer is input to the first batch normalization layer for batch normalization;

[0024] The output of the first batch normalization layer is input to the first activation function layer;

[0025] The output of the first activation function layer is input to the second convolution layer for convolution processing;

[0026] The output of the second convolutional layer is input to the second batch normalization layer for batch normalization processing;

[0027] The output of the second batch normalization layer is input to the second activation function layer;

[0028] The output of the second activation function layer is input to the neck module, and the neck module outputs the first tensor.

[0029] In combination with the second possible implementation manner of the first aspect, the embodiment of the present invention provides a third possible implementation manner of the first aspect, wherein the output of the second activation function layer is input to the neck module, and the neck module outputs a first tensor, including:

[0030] The outputs of the second activation function layer are input to the first convolution sublayer and the fourth convolution sublayer respectively;

[0031] The output of the first convolution sublayer is input to the first batch normalization sublayer, the output of the first batch normalization sublayer is input to the first activation function sublayer, the output of the first activation function sublayer is input to the second convolution sublayer, the output of the second convolution sublayer is input to the second batch normalization sublayer, the output of the second batch normalization sublayer is input to the second activation function sublayer, the output of the second activation function sublayer is input to the third convolution sublayer, and the output of the third convolution sublayer is input to the third batch normalization sublayer;

[0032] The output of the third batch normalization sublayer is input to the first channel attention submodule, and the output of the first channel attention submodule is input to the first spatial attention submodule;

[0033] The output of the fourth convolution sublayer is input to the fourth batch normalization sublayer;

[0034] The output of the first spatial attention submodule and the output of the fourth batch normalization sublayer are input to the incremental submodule;

[0035] The output of the incremental submodule is input to the third activation function sublayer, and the output of the third activation function sublayer is the first tensor.

[0036] In combination with the third possible implementation manner of the first aspect, an embodiment of the present invention provides a fourth possible implementation manner of the first aspect, wherein the output of the third batch normalization sublayer is input to the first channel attention submodule, including:

[0037] The outputs of the third batch normalization sublayer are respectively input to the average pooling layer, the maximum pooling layer and the first multiplication operation layer;

[0038] The output of the average pooling layer is input to the first tensor shape change layer, the output of the first tensor shape change layer is input to the first linear layer, the output of the first linear layer is input to the fourth activation function sublayer, and the output of the fourth activation function sublayer is input to the second linear layer;

[0039] The output of the maximum pooling layer is input to the second tensor shape change layer, the output of the second tensor shape change layer is input to the third linear layer, the output of the third linear layer is input to the fifth activation function sublayer, and the output of the fifth activation function sublayer is input to the fourth linear layer;

[0040] The output of the second linear layer and the output of the fourth linear layer are input to the first addition operation layer for addition operation, the output of the first addition operation layer is input to the sixth activation function sublayer, and the output of the sixth activation function sublayer is input to the tensor expansion operation layer;

[0041] The output of the tensor expansion operation layer and the output of the third batch normalization sub-layer are input to the first multiplication operation layer to obtain the tensor output to the first sub-module of spatial attention.

[0042] In combination with the fourth possible implementation of the first aspect, an embodiment of the present invention provides a fifth possible implementation of the first aspect, wherein the output of the first channel attention submodule is input to the first spatial attention submodule, including:

[0043] The outputs of the first multiplication operation layer are respectively input into the maximum operation layer, the average operation layer and the second multiplication operation layer;

[0044] The output of the maximum operation layer and the output of the average operation layer are input to the first tensor splicing sublayer, and the output of the first tensor splicing sublayer is input to the convolution sublayer;

[0045] The output of the convolution sublayer is input to the batch normalization sublayer, and the output of the batch normalization sublayer is input to the seventh activation function sublayer;

[0046] The output of the seventh activation function sublayer and the output of the first multiplication operation layer are input to the second multiplication operation layer.

[0047] In combination with the first aspect and any possible implementation manner from the first possible implementation manner to the fifth possible implementation manner of the first aspect, an embodiment of the present invention provides a sixth possible implementation manner of the first aspect, wherein the number of channels of the initial tensor is 3, and the number of channels of the tensor to be processed is 5.

[0048] In a second aspect, an embodiment of the present invention further provides a drivable area detection device, comprising:

[0049] An initial tensor construction module is used to obtain a road traffic image using a monocular camera, and to construct an initial tensor containing a color channel based on the obtained road traffic image;

[0050] A tensor construction module to be processed is used to obtain the coordinate information of each pixel in each road traffic image, add the coordinate information of the pixel to the color channel corresponding to the pixel, and obtain the tensor to be processed corresponding to the road traffic image;

[0051] A feature tensor acquisition module, used for inputting the tensor to be processed into a pre-trained neural network including an attention module to obtain a feature tensor representing a drivable area and a non-drivable area;

[0052] The drivable area detection module is used to construct the outline of the drivable area based on the pixel points representing the drivable area in the feature tensor, and control the vehicle to travel within the outline of the drivable area.

[0053] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the above method when executing the computer program.

[0054] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above method are executed.

[0055] The method and device for detecting a drivable area provided by the embodiment of the present invention obtains a road traffic image by using a monocular camera, and constructs an initial tensor containing a color channel based on the obtained road traffic image; for each road traffic image, obtains the coordinate information of each pixel point in the road traffic image, adds the coordinate information of the pixel point to the color channel corresponding to the pixel point, and obtains a tensor to be processed corresponding to the road traffic image; inputs the tensor to be processed into a pre-trained neural network containing an attention module to obtain a feature tensor representing the drivable area and the non-drivable area; constructs the outline of the drivable area based on the pixel points representing the drivable area in the feature tensor, and controls the vehicle to travel within the outline of the drivable area. In this way, by adding the pixel coordinates of the image taken by the monocular camera to the three-channel tensor, the neural network's perception of position information is improved, and by designing a convolutional neural network that integrates the visual attention mechanism, the influence of interference information on the image on the detection result is reduced, thereby effectively improving the detection accuracy and reliability of the drivable area.

[0056] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, preferred embodiments are given below and described in detail with reference to the accompanying drawings. BRIEF DESCRIPTION OF THE DRAWINGS

[0057] In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for use in the embodiments are briefly introduced below. It should be understood that the following drawings only show certain embodiments of the present invention and therefore should not be regarded as limiting the scope. For ordinary technicians in this field, other related drawings can be obtained based on these drawings without creative work.

[0058] Figure 1 A schematic diagram of the flow of a drivable area detection method provided in an embodiment of the present invention is shown;

[0059] Figure 2 A schematic diagram of the semantic segmentation network structure provided by an embodiment of the present invention is shown;

[0060] Figure 3A schematic diagram of the structure of a neck module provided by an embodiment of the present invention is shown;

[0061] Figure 4 A schematic diagram of the structure of the first submodule of the channel attention provided in an embodiment of the present invention is shown;

[0062] Figure 5 A schematic diagram of the structure of the first submodule of spatial attention provided by an embodiment of the present invention is shown;

[0063] Figure 6 A schematic diagram of the basic module structure provided by an embodiment of the present invention is shown;

[0064] Figure 7 A schematic diagram of the structure of a drivable area detection device provided in an embodiment of the present invention is shown;

[0065] Figure 8 A schematic diagram of the structure of a computer device 800 provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] In order to make the purpose, technical solutions and advantages of the embodiments of the present invention clearer, the technical solutions in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. The components of the embodiments of the present invention generally described and shown in the drawings here can be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of the present invention provided in the drawings is not intended to limit the scope of the claimed invention, but merely represents selected embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without making creative work belong to the scope of protection of the present invention.

[0067] The existing drivable area detection method based on binocular camera parallax images requires the use of binocular cameras, and has high requirements for the calibration accuracy of binocular cameras, which makes the equipment cost for drivable area detection high; in addition, the bumps and shakes during vehicle driving will affect the detection accuracy, resulting in false detection of drivable area detection, detecting the original drivable area as a non-drivable area, making the vehicle unable to drive, causing traffic congestion, or detecting the original non-drivable area as a drivable area, causing the vehicle to have a traffic accident. In the embodiment of the present invention, a monocular camera is used to capture images, and by adding the pixel coordinates of the captured image to a three-channel tensor to form a five-channel tensor, the neural network's ability to perceive position information can be improved, and by designing a convolutional neural network that integrates a visual attention mechanism, the influence of irrelevant interference information on the image on the detection results can be reduced, thereby effectively improving the detection accuracy and reliability of the drivable area.

[0068] The embodiments of the present invention provide a method and device for detecting a drivable area, which will be described below through embodiments.

[0069] Figure 1 FIG. 2 is a flow chart of a method for detecting a drivable area provided in an embodiment of the present invention. Figure 1 As shown, the method includes:

[0070] Step 101, using a monocular camera to acquire a road traffic image, and constructing an initial tensor including a color channel based on the acquired road traffic image;

[0071] In an embodiment of the present invention, a monocular camera installed in a vehicle is used to shoot road traffic to obtain a road traffic image. As an optional embodiment, the road traffic image is an initial tensor of size 3*H*W, where 3 is the number of RGB color channels, H is the height of the road traffic image, and W is the width of the road traffic image. As an optional embodiment, H*W is 2048*1024.

[0072] Step 102, for each road traffic image, obtaining coordinate information of each pixel in the road traffic image, adding the coordinate information of the pixel to the color channel corresponding to the pixel, and obtaining a tensor to be processed corresponding to the road traffic image;

[0073] In the embodiment of the present invention, the coordinate information of the pixel point is assumed to be (X, Y), where X is the horizontal coordinate information of the pixel point, and Y is the vertical coordinate information of the pixel point. The horizontal coordinate information and the vertical coordinate information of each pixel point are respectively added to the RGB color channels of the pixel point to obtain new channels (R, G, B, X, Y), thereby expanding the initial tensor (3*H*W) to a tensor to be processed of a size of 5*H*W, where 5 is the number of channels (R, G, B, X, Y).

[0074] In the embodiment of the present invention, a road traffic image corresponds to a tensor to be processed.

[0075] Step 103, inputting the tensor to be processed into a pre-trained neural network including an attention module to obtain a feature tensor representing the drivable area and the non-drivable area;

[0076] In the embodiment of the present invention, a tensor to be processed of size 5*H*W is input into a neural network including an attention module. For each tensor to be processed, a binary tensor of size H*W is output. In the binary tensor, if the value of a pixel is 0, it means that the pixel belongs to a drivable area, and if the value is 1, it means that the pixel belongs to a non-drivable area.

[0077] In the embodiment of the present invention, batch processing can be performed on the tensors to be processed, and multiple tensors to be processed (B, C, H, W) can be input at one time. Wherein, B is the number of input tensors to be processed, and C is the number of channels. In the embodiment of the present invention, C=5.

[0078] In an embodiment of the present invention, as an optional embodiment, the neural network is a semantic segmentation network.

[0079] Step 104: construct an outline of the drivable area according to the pixel points representing the drivable area in the feature tensor, and control the vehicle to travel within the outline of the drivable area.

[0080] In the embodiment of the present invention, the outline of the drivable area, that is, the boundary points of the drivable area, is found based on the binary tensor output by the neural network.

[0081] In an embodiment of the present invention, as an optional embodiment, the neural network outputs a binary tensor image, and the binary tensor image is scanned from left to right and from top to bottom using a scanning method to obtain the contour of the drivable area, or the contour finding algorithm interface in OpenCV is used to calculate the contour of the drivable area.

[0082] The following is a detailed description of step 103.

[0083] Figure 2 FIG. 4 shows a schematic diagram of a semantic segmentation network structure provided by an embodiment of the present invention. Figure 2 As shown, the semantic segmentation network is a multi-branch and multi-stage network, including: a first branch layer, a second branch layer, a third branch layer and a fourth branch layer. The size and number of channels in the tensor feature map corresponding to different branch layers are different. Each branch layer is responsible for the corresponding tensor processing. Then, the tensors output by all branch layers are fused and spliced, and each pixel in the spliced ​​tensor is classified to determine whether the pixel belongs to a drivable area, thereby outputting a feature tensor.

[0084] In an embodiment of the present invention, as an optional embodiment, the neural network is a semantic segmentation network, and the tensor to be processed is input into a pre-trained neural network including an attention module to obtain a feature tensor representing the drivable area and the non-drivable area, including:

[0085] A11, inputting the tensor to be processed into the first sublayer of the first branch layer in the semantic segmentation network to obtain a first tensor;

[0086] In an embodiment of the present invention, a set of road traffic images to be processed (tensor set to be processed) with a tensor size of B*5*H*W is input into a semantic segmentation network, and convolutional layers at different depths in the semantic segmentation network perform convolution operations on the tensors output by the previous layer, output new tensors, and the new tensors are forward propagated in different branch layers to output feature tensors of different resolutions.

[0087] In an embodiment of the present invention, the first sublayer includes in sequence: a first convolutional layer, a first batch normalization layer, a first activation function layer, a second convolutional layer, a second batch normalization layer, a second activation function layer and a neck module. Among them, the first convolutional layer, the first batch normalization layer, and the first activation function layer are called the first convolution, normalization, and activation layer, and the second convolutional layer, the second batch normalization layer, and the second activation function layer are called the second convolution, normalization, and activation layer. As an optional embodiment, the tensor to be processed is input into the first sublayer of the first branch layer in the semantic segmentation network to obtain the first tensor, including:

[0088] B11, inputting the tensor to be processed into the first convolutional layer for convolution processing;

[0089] In the embodiment of the present invention, there may be multiple input tensors to be processed, forming a set of tensors to be processed (B*5*H*W).

[0090] B12, the output of the first convolutional layer is input to the first batch normalization layer for batch normalization processing;

[0091] B13, the output of the first batch normalization layer is input to the first activation function layer;

[0092] In the embodiment of the present invention, the activation function used by the first activation function layer is the ReLU activation function.

[0093] B14, the output of the first activation function layer is input to the second convolution layer for convolution processing;

[0094] B15, the output of the second convolutional layer is input to the second batch normalization layer for batch normalization processing;

[0095] B16, the output of the second batch normalization layer is input to the second activation function layer;

[0096] In the embodiment of the present invention, the activation function used by the second activation function layer is the ReLU activation function.

[0097] B17, the output of the second activation function layer is input to the neck module, and the neck module outputs the first tensor.

[0098] In the embodiment of the present invention, the convolution layer (Conv) performs convolution processing on the input tensor, the batch normalization layer (BN) performs a normalization operation, the activation function layer (ReLU) performs an activation operation, and the neck module (Bottleneck) uses a 1*1 convolutional neural network.

[0099] Figure 3 FIG. 2 shows a schematic diagram of the neck module structure provided by an embodiment of the present invention. Figure 3 As shown, in an embodiment of the present invention, the neck module (Bottleneck) includes: a first convolution sublayer, a first batch normalization sublayer, a first activation function sublayer, a second convolution sublayer, a second batch normalization sublayer, a second activation function sublayer, a third convolution sublayer, a third batch normalization sublayer, a fourth convolution sublayer, a fourth batch normalization sublayer, a channel attention first submodule, a spatial attention first submodule, a gradual addition submodule and a third activation function sublayer. As an optional embodiment, the output of the second activation function layer is input to the neck module, and the neck module outputs a first tensor, including:

[0100] C11, the output of the second activation function layer is input to the first convolution sublayer and the fourth convolution sublayer respectively;

[0101] C12, the output of the first convolution sublayer is input to the first batch normalization sublayer, the output of the first batch normalization sublayer is input to the first activation function sublayer, the output of the first activation function sublayer is input to the second convolution sublayer, the output of the second convolution sublayer is input to the second batch normalization sublayer, the output of the second batch normalization sublayer is input to the second activation function sublayer, the output of the second activation function sublayer is input to the third convolution sublayer, and the output of the third convolution sublayer is input to the third batch normalization sublayer;

[0102] C13, the output of the third batch normalization sublayer is input to the first submodule of channel attention, and the output of the first submodule of channel attention is input to the first submodule of spatial attention;

[0103] C14, the output of the fourth convolutional sublayer is input to the fourth batch normalization sublayer;

[0104] C15, the output of the first submodule of spatial attention and the output of the fourth batch normalization sublayer are input to the incremental submodule;

[0105] C16, the output of the incremental submodule is input to the third activation function sublayer, and the output of the third activation function sublayer is the first tensor.

[0106] In the embodiment of the present invention, the activation functions used by the first activation function sublayer, the second activation function sublayer and the third activation function sublayer are all ReLU activation functions.

[0107] In an embodiment of the present invention, the first channel attention submodule (Channel Attention Module) calculates the output of the third batch normalization sublayer of the input, so that different channels contribute to the characteristics of the output tensor to different degrees, thereby obtaining a tensor containing channel attention features, and then the tensor containing the channel attention features is input into the first spatial attention submodule (Spatial Attention Module). After calculation, different positions contribute to the characteristics of the output tensor to different degrees, thereby obtaining a tensor containing spatial attention features.

[0108] Figure 4 FIG. 4 shows a schematic diagram of the structure of the first submodule of the channel attention provided by an embodiment of the present invention. Figure 4 As shown, in the embodiment of the present invention, the first submodule of the channel attention includes: an average pooling layer (AvgPool), a maximum pooling layer (MaxPool), a first tensor shape change layer (Reshape), a second tensor shape change layer (Reshape), a first linear layer (Linear), a second linear layer (Linear), a third linear layer (Linear), a fourth linear layer (Linear), a fourth activation function sublayer, a fifth activation function sublayer, a sixth activation function sublayer, a tensor expansion operation layer, a first addition operation layer and a first multiplication operation layer. As an optional embodiment, the output of the third batch normalization sublayer is input to the first submodule of the channel attention, including:

[0109] D11, the output of the third batch normalization sublayer is input to the average pooling layer, the maximum pooling layer and the first multiplication operation layer respectively;

[0110] D12, the output of the average pooling layer is input to the first tensor shape change layer, the output of the first tensor shape change layer is input to the first linear layer, the output of the first linear layer is input to the fourth activation function sublayer, and the output of the fourth activation function sublayer is input to the second linear layer;

[0111] D13, the output of the maximum pooling layer is input to the second tensor shape change layer, the output of the second tensor shape change layer is input to the third linear layer, the output of the third linear layer is input to the fifth activation function sublayer, and the output of the fifth activation function sublayer is input to the fourth linear layer;

[0112] D14, the output of the second linear layer and the output of the fourth linear layer are input to the first addition operation layer for addition operation, the output of the first addition operation layer is input to the sixth activation function sublayer, and the output of the sixth activation function sublayer is input to the tensor expansion operation layer;

[0113] D15, the output of the tensor expansion operation layer and the output of the third batch normalization sub-layer are input to the first multiplication operation layer to obtain the tensor output to the first sub-module of spatial attention.

[0114] In the embodiment of the present invention, the activation function used by the fourth activation function sublayer and the fifth activation function sublayer is the ReLU activation function, and the activation function used by the sixth activation function sublayer is the Sigmoid activation function. The output (tensor) of the third batch normalization sublayer is (B, C, H, W), the outputs of the average pooling layer and the maximum pooling layer are both (B, C, 1, 1), and the first linear layer to the fourth connection layer are all fully connected layers.

[0115] In this embodiment of the present invention, the outputs of the second linear layer and the fourth linear layer are both (B, C), the output of the sixth activation function sublayer is (B, C), the output of the tensor expansion operation layer is (B, C, H, W), and the output of the first multiplication operation layer is (B, C, H, W).

[0116] In an embodiment of the present invention, the first multiplication operation layer performs a point-by-point multiplication operation on the elements in the two input tensors (the tensor output by the tensor expansion operation layer and the tensor output by the third batch normalization sublayer), so that in the first submodule of the channel attention, a tensor of size B*C is obtained through learning, which represents the learned weight of each channel, and is expanded to a tensor of B*C*H*W, and is element-wise multiplied with the output of the third batch normalization sublayer. In this way, the different degrees of contribution of different channels to the feature tensor can be learned.

[0117] Figure 5 FIG. 4 shows a schematic diagram of the structure of the first submodule of spatial attention provided by an embodiment of the present invention. Figure 5 As shown, in this embodiment of the present invention, the first submodule of spatial attention includes: a maximum operation layer, an average operation layer, a first tensor splicing sublayer, a convolution sublayer, a batch normalization sublayer, a seventh activation function sublayer and a second multiplication operation layer, and the output of the first submodule of channel attention is input to the first submodule of spatial attention, including:

[0118] E11, the outputs of the first multiplication operation layer are respectively input to the maximum operation layer, the average operation layer and the second multiplication operation layer;

[0119] E12, the output of the maximum operation layer and the output of the average operation layer are input to the first tensor splicing sublayer, and the output of the first tensor splicing sublayer is input to the convolution sublayer;

[0120] E13, the output of the convolution sublayer is input to the batch normalization sublayer, and the output of the batch normalization sublayer is input to the seventh activation function sublayer;

[0121] E14, the output of the seventh activation function sublayer and the output of the first multiplication operation layer are input to the second multiplication operation layer.

[0122] In the embodiment of the present invention, the activation function used by the seventh activation function sublayer is the Sigmoid activation function. The tensor size of the input maximum operation layer is (B, C, H, W), the output tensor sizes of the maximum operation layer and the average operation layer are (B, 1, H, W), the output tensor size of the first tensor splicing sublayer is (B, 2, H, W), the output tensor size of the seventh activation function sublayer is (B, 1, H, W), and the output tensor size of the second multiplication operation layer is (B, C, H, W).

[0123] In an embodiment of the present invention, the first submodule of spatial attention learns a tensor of size B*1*H*W, which represents the weight of each pixel position, and multiplies it element-wise with the original input of the first submodule of spatial attention (the output of the first multiplication operation layer). In this way, it is possible to learn that different spatial positions in the tensor contribute to the features to different degrees.

[0124] A12, input the first tensor into the second sublayer in the first branch layer and the first subbranch layer in the second branch layer, respectively, to obtain the second tensor and the third tensor, wherein the second sublayer includes: the third convolution layer, the third batch normalization layer, the third activation function layer, the first basic module and the second basic module in sequence, and the first subbranch layer includes: the fourth convolution layer, the fourth batch normalization layer, the fourth activation function layer, the third basic module and the fourth basic module in sequence. Among them, the third convolution layer, the third batch normalization layer, and the third activation function layer are called the third convolution, normalization, and activation layer, and the fourth convolution layer, the fourth batch normalization layer, and the fourth activation function layer are called the fourth convolution, normalization, and activation layer.

[0125] In the embodiment of the present invention, the basic block has an inlet and an outlet.

[0126] Figure 6 FIG. 1 shows a schematic diagram of the basic module structure provided by an embodiment of the present invention. Figure 6 As shown, the basic module includes in sequence: a fifth convolution sublayer, a fifth batch normalization sublayer, an eighth activation function sublayer, a sixth convolution sublayer, a sixth batch normalization sublayer, a channel attention second submodule, a spatial attention second submodule, a second gradual addition submodule and a ninth activation function sublayer, wherein the structure of the channel attention second submodule is the same as that of the channel attention first submodule, and the structure of the spatial attention second submodule is the same as that of the spatial attention first submodule.

[0127] A13, inputting the second tensor and the third tensor into the first feature fusion layer to obtain a fourth tensor and a fifth tensor;

[0128] In the implementation of the present invention, the first feature fusion layer performs different processing on the second and third input tensors to obtain the fourth and fifth tensors, respectively. The fourth tensor is the output corresponding to the second input tensor, and the fifth tensor is the output corresponding to the third input tensor.

[0129] A14, input the fourth tensor into the fifth basic module in the first branch layer, input the output of the fifth basic module into the sixth basic module to obtain the sixth tensor, input the fifth tensor into the seventh basic module in the second branch layer, input the output of the seventh basic module into the eighth basic module to obtain the seventh tensor, input the fifth tensor into the first branch sub-layer in the third branch layer to obtain the eighth tensor, and the first branch sub-layer includes: a fifth convolutional layer, a fifth batch normalization layer, a fifth activation function layer, a ninth basic module and a tenth basic module in sequence; among which, the fifth convolutional layer, the fifth batch normalization layer, and the fifth activation function layer are called the fifth convolutional, normalization, and activation layers.

[0130] A15, input the sixth tensor, the seventh tensor, and the eighth tensor into the second feature fusion layer to obtain the ninth tensor, the tenth tensor, and the eleventh tensor respectively;

[0131] In an embodiment of the present invention, a feature fusion layer (Fuse Layer) is used to fuse the features of tensors. The second feature fusion layer performs different processing on the sixth, seventh, and eighth tensors of the input, and obtains the ninth, tenth, and eleventh tensors, respectively. Among them, the ninth tensor is the output corresponding to the sixth input tensor, the tenth tensor is the output corresponding to the seventh input tensor, and the eleventh tensor is the output corresponding to the eighth input tensor.

[0132] A16, input the ninth tensor into the eleventh basic module in the first branch layer, input the output of the eleventh basic module into the twelfth basic module to obtain the twelfth tensor, input the tenth tensor into the thirteenth basic module in the second branch layer, input the output of the thirteenth basic module into the fourteenth basic module to obtain the thirteenth tensor, input the eleventh tensor into the fifteenth basic module in the third branch layer, input the output of the fifteenth basic module into the sixteenth basic module to obtain the fourteenth tensor, input the eleventh tensor into the fourth branch layer to obtain the fifteenth tensor, and the fourth branch layer sequentially includes: a sixth convolutional layer, a sixth batch normalization layer, a sixth activation function layer, a seventeenth basic module and an eighteenth basic module; among which, the sixth convolutional layer, the sixth batch normalization layer, and the sixth activation function layer are called the sixth convolution, normalization, and activation layers.

[0133] In an embodiment of the present invention, the feature resolution of the tensor output by the first branch layer is H / 4*W / 4, the feature resolution of the tensor output by the second branch layer is H / 8*W / 8, the feature resolution of the tensor output by the third branch layer is H / 16*W / 16, and the feature resolution of the tensor output by the fourth branch layer is H / 32*W / 32.

[0134] A17, input the twelfth tensor, the thirteenth tensor, the fourteenth tensor, and the fifteenth tensor into the third feature fusion layer to obtain the sixteenth tensor, the seventeenth tensor, the eighteenth tensor, and the nineteenth tensor respectively;

[0135] In the embodiment of the present invention, the sixteenth tensor is the output corresponding to the input twelfth tensor, the seventeenth tensor is the output corresponding to the input thirteenth tensor, the eighteenth tensor is the output corresponding to the input fourteenth tensor, and the nineteenth tensor is the output corresponding to the input fifteenth tensor.

[0136] In an embodiment of the present invention, the feature fusion layer (first feature fusion layer, second feature fusion layer, third feature fusion layer) in the semantic segmentation network fuses the features of tensors from different branch layers, upsamples the features output by the low-resolution branch layer to make them consistent with the resolution of a higher layer, and then adds the two pixel by pixel to obtain a fused new feature. After the last fusion (fusion of the third feature fusion layer), the output size is B*C0*H / 4*W / 4 (the sixteenth tensor), B*C1*H / 8*W / 8 (the seventeenth tensor), B*C2*H / 16*W / 16 (the eighteenth tensor), B*C3*H / 32*W / 32 (the nineteenth tensor), where C0, C1, C2, and C3 are the number of channels of the final output tensors corresponding to the four preset branch layers.

[0137] A18, inputting the tensor output by the third feature fusion layer into the tensor concatenation layer to obtain a concatenated tensor;

[0138] In the embodiment of the present invention, the tensor concatenation layer (Concat) is used to perform a tensor concatenation operation on tensors.

[0139] In the embodiment of the present invention, the four tensors output by the third feature fusion layer are concatenated in the tensor concatenation layer to obtain a concatenated tensor of size B*(C0+C1+C2+C3)*H / 4*W / 4.

[0140] A19, inputs the concatenated tensor into the seventh convolutional layer, and inputs the output of the seventh convolutional layer into the eighth convolutional layer to obtain the output tensor;

[0141] In an embodiment of the present invention, the concatenated tensor output by the tensor concatenation layer is input into the seventh convolutional layer, and the output of the seventh convolutional layer is input into the eighth convolutional layer for classification to obtain an output tensor of a size of B*K*H / 4*W / 4, where K is the number of classification categories.

[0142] A20, performs Argmax operation on the output tensor along the 0th dimension to obtain the tensor to be processed, and upsamples the tensor to be processed to obtain the feature tensor.

[0143] In an embodiment of the present invention, an Argmax operation is performed on the output tensor along the 0th dimension, i.e., the dimension corresponding to K, to obtain a tensor of a size of B*1*H / 4*W / 4, and the obtained tensor is upsampled to obtain a feature tensor of a size of B*1*H*W, which contains the classification results of each pixel in the road traffic image to be processed.

[0144] In an embodiment of the present invention, the semantic segmentation network is pre-trained to complete the drivable area detection by the trained semantic segmentation network. As an optional embodiment, the training is supervised training, the training data is collected and labeled images containing drivable areas, and the training data is enhanced, including horizontal flipping, multi-scale transformation, random cropping, etc.

[0145] In the embodiment of the present invention, the purpose of training the neural network using training data is to find suitable parameters for the neural network, that is, the weights of the neural network. The training process is to update the weights of the neural network until the optimal weights are found. In the process of updating the weights, the loss function is used to measure the difference (loss) between the output predicted by the neural network and the true value of the training data. The loss function is a function that is only related to the weights of the neural network. In order to improve the accuracy of the prediction results of the neural network, the loss needs to be as small as possible. In the embodiment of the present invention, the gradient descent method is used to update the weights of the neural network, that is, starting from the last layer of the network, the chain rule of the composite function derivation is used from back to front to find the gradient of the function at this point, calculate the gradient of the loss function at this point, and then update the weights of the neural network along the negative gradient direction. In this way, by transmitting training data to the neural network round by round, the neural network will continuously update the weights to make the prediction results better and better.

[0146] In the embodiment of the present invention, there are different optimization algorithms, called different optimizers, according to the different specific implementation details of gradient descent. As an optional embodiment, the optimizer selects stochastic gradient descent (SGD), and the learning rate strategy is cosine annealing strategy (Cosine Annealing Warm Restart).

[0147] In the embodiment of the present invention, the loss function is as follows:

[0148]

[0149]

[0150] In the formula,

[0151] loss(x,class) represents the loss of training data x;

[0152] class is the ground truth label of the training data x, that is, the sample x in the training data is labeled as class, including drivable areas and non-drivable areas;

[0153] w class is the weight of the class, and the weight of the class is the preset value;

[0154] p class is the predicted value of the sample in the category class;

[0155] It means to sum the predicted values ​​of all K categories, and the subscript j ranges from 1 to K, where K is the total number of categories;

[0156] L represents the total loss;

[0157] It represents the sum of the losses of all N samples, where N is the total number of samples;

[0158] x i represents the i-th sample, i ranges from 1 to N;

[0159] class / x i For sample x i The truth value category of

[0160] Represents the sum of the weights of the true value categories corresponding to all samples;

[0161] For sample x i The weight of the corresponding true value category.

[0162] In the embodiment of the present invention, the image is captured by a monocular camera, and no additional sensors are required, which is low in cost, and has high feasibility and scalability; by adding pixel coordinates to the input data, the neural network's ability to perceive position information can be improved. By designing a convolutional neural network that integrates a visual attention mechanism, the impact of irrelevant interference information on the image on the detection results is reduced, with high real-time performance, fast detection speed, low video memory usage, and low computing resource consumption, which can effectively improve the segmentation accuracy of the drivable area and the reliability of detection.

[0163] Figure 7FIG. 2 shows a schematic diagram of the structure of a drivable area detection device provided in an embodiment of the present invention. Figure 7 As shown, the device comprises:

[0164] An initial tensor construction module 701 is used to obtain a road traffic image using a monocular camera, and construct an initial tensor including a color channel based on the obtained road traffic image;

[0165] The to-be-processed tensor construction module 702 is used to obtain the coordinate information of each pixel in each road traffic image, add the coordinate information of the pixel to the color channel corresponding to the pixel, and obtain the to-be-processed tensor corresponding to the road traffic image;

[0166] In the embodiment of the present invention, the number of channels of the initial tensor is 3, and the number of channels of the tensor to be processed is 5.

[0167] A feature tensor acquisition module 703 is used to input the tensor to be processed into a pre-trained neural network including an attention module to obtain a feature tensor representing a drivable area and a non-drivable area;

[0168] In the embodiment of the present invention, as an optional embodiment, the neural network is a semantic segmentation network, and the feature tensor acquisition module 703 includes:

[0169] A first tensor acquisition unit (not shown in the figure), used for inputting the tensor to be processed into a first sublayer in a first branch layer in a semantic segmentation network to obtain a first tensor;

[0170] A first tensor processing unit, used for inputting the first tensor into the second sublayer in the first branch layer and the first subbranch layer in the second branch layer, respectively, to obtain a second tensor and a third tensor, wherein the second sublayer includes in sequence: a third convolutional layer, a third batch normalization layer, a third activation function layer, a first basic module and a second basic module, and the first subbranch layer includes in sequence: a fourth convolutional layer, a fourth batch normalization layer, a fourth activation function layer, a third basic module and a fourth basic module;

[0171] A first feature fusion unit is used to input the second tensor and the third tensor into the first feature fusion layer to obtain a fourth tensor and a fifth tensor;

[0172] A second tensor processing unit is used to input the fourth tensor into a fifth basic module in the first branch layer, input the output of the fifth basic module into a sixth basic module, obtain a sixth tensor, input the fifth tensor into a seventh basic module in the second branch layer, input the output of the seventh basic module into an eighth basic module, obtain a seventh tensor, input the fifth tensor into a first branch sub-layer in the third branch layer, and obtain an eighth tensor, wherein the first branch sub-layer sequentially includes: a fifth convolutional layer, a fifth batch normalization layer, a fifth activation function layer, a ninth basic module, and a tenth basic module;

[0173] A second feature fusion unit is used to input the sixth tensor, the seventh tensor, and the eighth tensor into the second feature fusion layer to obtain a ninth tensor, a tenth tensor, and an eleventh tensor respectively;

[0174] A third tensor processing unit is used to input the ninth tensor into the eleventh basic module in the first branch layer, the output of the eleventh basic module is input into the twelfth basic module, and the twelfth tensor is obtained; the tenth tensor is input into the thirteenth basic module in the second branch layer, the output of the thirteenth basic module is input into the fourteenth basic module, and the thirteenth tensor is obtained; the eleventh tensor is input into the fifteenth basic module in the third branch layer, the output of the fifteenth basic module is input into the sixteenth basic module, and the fourteenth tensor is obtained; the eleventh tensor is input into the fourth branch layer to obtain the fifteenth tensor, and the fourth branch layer sequentially includes: a sixth convolutional layer, a sixth batch normalization layer, a sixth activation function layer, a seventeenth basic module, and an eighteenth basic module;

[0175] A third feature fusion unit is used to input the twelfth tensor, the thirteenth tensor, the fourteenth tensor, and the fifteenth tensor into the third feature fusion layer to obtain a sixteenth tensor, a seventeenth tensor, an eighteenth tensor, and a nineteenth tensor respectively;

[0176] A splicing unit, used for inputting the tensor output by the third feature fusion layer into the tensor splicing layer to obtain a spliced ​​tensor;

[0177] A fourth tensor processing unit is used to input the concatenated tensor into the seventh convolution layer, and the output of the seventh convolution layer is input into the eighth convolution layer to obtain an output tensor;

[0178] The output unit is used to perform Argmax operation on the output tensor along the 0th dimension to obtain the tensor to be processed, and upsample the tensor to be processed to obtain the feature tensor.

[0179] In the embodiment of the present invention, as an optional embodiment, the tensor first acquisition unit is specifically used to:

[0180] Inputting the tensor to be processed into the first convolutional layer for convolution processing;

[0181] The output of the first convolutional layer is input to the first batch normalization layer for batch normalization;

[0182] The output of the first batch normalization layer is input to the first activation function layer;

[0183] The output of the first activation function layer is input to the second convolution layer for convolution processing;

[0184] The output of the second convolutional layer is input to the second batch normalization layer for batch normalization processing;

[0185] The output of the second batch normalization layer is input to the second activation function layer;

[0186] The output of the second activation function layer is input to the neck module, and the neck module outputs the first tensor.

[0187] In the embodiment of the present invention, the output of the second activation function layer is input to the neck module, and the neck module outputs a first tensor, including:

[0188] The outputs of the second activation function layer are input to the first convolution sublayer and the fourth convolution sublayer respectively;

[0189] The output of the first convolution sublayer is input to the first batch normalization sublayer, the output of the first batch normalization sublayer is input to the first activation function sublayer, the output of the first activation function sublayer is input to the second convolution sublayer, the output of the second convolution sublayer is input to the second batch normalization sublayer, the output of the second batch normalization sublayer is input to the second activation function sublayer, the output of the second activation function sublayer is input to the third convolution sublayer, and the output of the third convolution sublayer is input to the third batch normalization sublayer;

[0190] The output of the third batch normalization sublayer is input to the first channel attention submodule, and the output of the first channel attention submodule is input to the first spatial attention submodule;

[0191] The output of the fourth convolution sublayer is input to the fourth batch normalization sublayer;

[0192] The output of the first spatial attention submodule and the output of the fourth batch normalization sublayer are input to the incremental submodule;

[0193] The output of the incremental submodule is input to the third activation function sublayer, and the output of the third activation function sublayer is the first tensor.

[0194] In the embodiment of the present invention, the output of the third batch normalization sublayer is input to the first submodule of the channel attention, including:

[0195] The outputs of the third batch normalization sublayer are respectively input to the average pooling layer, the maximum pooling layer and the first multiplication operation layer;

[0196] The output of the average pooling layer is input to the first tensor shape change layer, the output of the first tensor shape change layer is input to the first linear layer, the output of the first linear layer is input to the fourth activation function sublayer, and the output of the fourth activation function sublayer is input to the second linear layer;

[0197] The output of the maximum pooling layer is input to the second tensor shape change layer, the output of the second tensor shape change layer is input to the third linear layer, the output of the third linear layer is input to the fifth activation function sublayer, and the output of the fifth activation function sublayer is input to the fourth linear layer;

[0198] The output of the second linear layer and the output of the fourth linear layer are input to the first addition operation layer for addition operation, the output of the first addition operation layer is input to the sixth activation function sublayer, and the output of the sixth activation function sublayer is input to the tensor expansion operation layer;

[0199] The output of the tensor expansion operation layer and the output of the third batch normalization sub-layer are input to the first multiplication operation layer to obtain the tensor output to the first sub-module of spatial attention.

[0200] In the embodiment of the present invention, the output of the first channel attention submodule is input to the first spatial attention submodule, including:

[0201] The outputs of the first multiplication operation layer are respectively input into the maximum operation layer, the average operation layer and the second multiplication operation layer;

[0202] The output of the maximum operation layer and the output of the average operation layer are input to the first tensor splicing sublayer, and the output of the first tensor splicing sublayer is input to the convolution sublayer;

[0203] The output of the convolution sublayer is input to the batch normalization sublayer, and the output of the batch normalization sublayer is input to the seventh activation function sublayer;

[0204] The output of the seventh activation function sublayer and the output of the first multiplication operation layer are input to the second multiplication operation layer.

[0205] The drivable area detection module 704 is used to construct the outline of the drivable area according to the pixel points representing the drivable area in the feature tensor, and control the vehicle to travel within the outline of the drivable area.

[0206] like Figure 8 As shown, an embodiment of the present application provides a computer device 800 for executing Figure 1 The device includes a memory 801, a processor 802, and a computer program stored in the memory 801 and executable on the processor 802, wherein the processor 802 implements the steps of the drivable area detection method when executing the computer program.

[0207] Specifically, the above-mentioned memory 801 and processor 802 can be general-purpose memories and processors, which are not specifically limited here. When the processor 802 runs the computer program stored in the memory 801, the above-mentioned drivable area detection method can be executed.

[0208] Corresponds to Figure 1 The drivable area detection method in the embodiment of the present application also provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the steps of the above-mentioned drivable area detection method are executed.

[0209] Specifically, the storage medium can be a general storage medium, such as a mobile disk, a hard disk, etc. When the computer program on the storage medium is executed, the above-mentioned drivable area detection method can be executed.

[0210] In the embodiments provided in the present application, it should be understood that the disclosed systems and methods can be implemented in other ways. The system embodiments described above are merely schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation. For example, multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some communication interfaces, indirect coupling or communication connection of systems or units, which can be electrical, mechanical or other forms.

[0211] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0212] In addition, each functional unit in the embodiments provided in the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0213] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk.

[0214] It should be noted that similar numbers and letters represent similar items in the following figures. Therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. In addition, the terms "first", "second", "third", etc. are only used to distinguish the description and are not to be understood as indicating or implying relative importance.

[0215] Finally, it should be noted that the above-described embodiments are only specific implementation methods of the present application, which are used to illustrate the technical solutions of the present application, rather than to limit them. The protection scope of the present application is not limited thereto. Although the present application is described in detail with reference to the above-mentioned embodiments, ordinary technicians in the field should understand that any technician familiar with the technical field can still modify the technical solutions recorded in the above-mentioned embodiments within the technical scope disclosed in the present application, or can easily think of changes, or make equivalent replacements for some of the technical features therein; and these modifications, changes or replacements do not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application. They should all be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for detecting a drivable area, characterized in that: include: A monocular camera is used to obtain a road traffic image, and an initial tensor including a color channel is constructed based on the obtained road traffic image; For each road traffic image, obtain the coordinate information of each pixel in the road traffic image, add the coordinate information of the pixel to the color channel corresponding to the pixel, and obtain the tensor to be processed corresponding to the road traffic image; Inputting the tensor to be processed into a pre-trained neural network including an attention module to obtain a feature tensor representing a drivable area and a non-drivable area; Constructing the outline of the drivable area according to the pixel points representing the drivable area in the feature tensor, and controlling the vehicle to travel within the outline of the drivable area; The neural network is a semantic segmentation network. The tensor to be processed is input into a pre-trained neural network including an attention module to obtain a feature tensor representing a drivable area and a non-drivable area, including: Inputting the tensor to be processed into the first sublayer in the first branch layer in the semantic segmentation network to obtain a first tensor; Inputting the first tensor into the second sublayer in the first branch layer and the first subbranch layer in the second branch layer respectively, to obtain a second tensor and a third tensor, wherein the second sublayer includes: a third convolutional layer, a third batch normalization layer, a third activation function layer, a first basic module and a second basic module in sequence, and the first subbranch layer includes: a fourth convolutional layer, a fourth batch normalization layer, a fourth activation function layer, a third basic module and a fourth basic module in sequence; Input the second tensor and the third tensor into the first feature fusion layer to obtain the fourth tensor and the fifth tensor; Input the fourth tensor into the fifth basic module in the first branch layer, input the output of the fifth basic module into the sixth basic module to obtain a sixth tensor, input the fifth tensor into the seventh basic module in the second branch layer, input the output of the seventh basic module into the eighth basic module to obtain a seventh tensor, input the fifth tensor into the first branch sub-layer in the third branch layer to obtain an eighth tensor, wherein the first branch sub-layer sequentially includes: a fifth convolutional layer, a fifth batch normalization layer, a fifth activation function layer, a ninth basic module, and a tenth basic module; Input the sixth tensor, the seventh tensor, and the eighth tensor into the second feature fusion layer to obtain the ninth tensor, the tenth tensor, and the eleventh tensor respectively; Input the ninth tensor into the eleventh basic module in the first branch layer, input the output of the eleventh basic module into the twelfth basic module, and obtain the twelfth tensor; input the tenth tensor into the thirteenth basic module in the second branch layer, input the output of the thirteenth basic module into the fourteenth basic module, and obtain the thirteenth tensor; input the eleventh tensor into the fifteenth basic module in the third branch layer, input the output of the fifteenth basic module into the sixteenth basic module, and obtain the fourteenth tensor; input the eleventh tensor into the fourth branch layer, and obtain the fifteenth tensor; the fourth branch layer sequentially includes: a sixth convolutional layer, a sixth batch normalization layer, a sixth activation function layer, a seventeenth basic module, and an eighteenth basic module; The twelfth tensor, the thirteenth tensor, the fourteenth tensor, and the fifteenth tensor are input into the third feature fusion layer to obtain the sixteenth tensor, the seventeenth tensor, the eighteenth tensor, and the nineteenth tensor respectively; Input the tensor output by the third feature fusion layer into the tensor concatenation layer to obtain a concatenated tensor; The concatenated tensor is input into the seventh convolutional layer, and the output of the seventh convolutional layer is input into the eighth convolutional layer to obtain the output tensor; Perform Argmax operation on the output tensor along the 0th dimension to obtain the tensor to be processed, and upsample the tensor to be processed to obtain the feature tensor.

2. The method according to claim 1, characterized in that The step of inputting the tensor to be processed into a first sublayer in a first branch layer in a semantic segmentation network to obtain a first tensor includes: Inputting the tensor to be processed into the first convolutional layer for convolution processing; The output of the first convolutional layer is input to the first batch normalization layer for batch normalization; The output of the first batch normalization layer is input to the first activation function layer; The output of the first activation function layer is input to the second convolution layer for convolution processing; The output of the second convolutional layer is input to the second batch normalization layer for batch normalization processing; The output of the second batch normalization layer is input to the second activation function layer; The output of the second activation function layer is input to the neck module, and the neck module outputs the first tensor.

3. The method according to claim 2, characterized in that The output of the second activation function layer is input to the neck module, and the neck module outputs a first tensor, including: The outputs of the second activation function layer are input to the first convolution sublayer and the fourth convolution sublayer respectively; The output of the first convolution sublayer is input to the first batch normalization sublayer, the output of the first batch normalization sublayer is input to the first activation function sublayer, the output of the first activation function sublayer is input to the second convolution sublayer, the output of the second convolution sublayer is input to the second batch normalization sublayer, the output of the second batch normalization sublayer is input to the second activation function sublayer, the output of the second activation function sublayer is input to the third convolution sublayer, and the output of the third convolution sublayer is input to the third batch normalization sublayer; The output of the third batch normalization sublayer is input to the first channel attention submodule, and the output of the first channel attention submodule is input to the first spatial attention submodule; The output of the fourth convolution sublayer is input to the fourth batch normalization sublayer; The output of the first spatial attention submodule and the output of the fourth batch normalization sublayer are input to the incremental submodule; The output of the incremental submodule is input to the third activation function sublayer, and the output of the third activation function sublayer is the first tensor.

4. The method according to claim 3, characterized in that The output of the third batch normalization sublayer is input to the first channel attention submodule, including: The outputs of the third batch normalization sublayer are respectively input to the average pooling layer, the maximum pooling layer and the first multiplication operation layer; The output of the average pooling layer is input to the first tensor shape change layer, the output of the first tensor shape change layer is input to the first linear layer, the output of the first linear layer is input to the fourth activation function sublayer, and the output of the fourth activation function sublayer is input to the second linear layer; The output of the maximum pooling layer is input to the second tensor shape change layer, the output of the second tensor shape change layer is input to the third linear layer, the output of the third linear layer is input to the fifth activation function sublayer, and the output of the fifth activation function sublayer is input to the fourth linear layer; The output of the second linear layer and the output of the fourth linear layer are input to the first addition operation layer for addition operation, the output of the first addition operation layer is input to the sixth activation function sublayer, and the output of the sixth activation function sublayer is input to the tensor expansion operation layer; The output of the tensor expansion operation layer and the output of the third batch normalization sub-layer are input to the first multiplication operation layer to obtain the tensor output to the first sub-module of spatial attention.

5. The method according to claim 4, characterized in that The output of the first channel attention submodule is input to the first spatial attention submodule, including: The outputs of the first multiplication operation layer are respectively input into the maximum operation layer, the average operation layer and the second multiplication operation layer; The output of the maximum operation layer and the output of the average operation layer are input to the first tensor splicing sublayer, and the output of the first tensor splicing sublayer is input to the convolution sublayer; The output of the convolution sublayer is input to the batch normalization sublayer, and the output of the batch normalization sublayer is input to the seventh activation function sublayer; The output of the seventh activation function sublayer and the output of the first multiplication operation layer are input to the second multiplication operation layer.

6. The method according to any one of claims 1 to 5, characterized in that: The number of channels of the initial tensor is 3, and the number of channels of the tensor to be processed is 5.

7. A drivable area detection device, characterized in that: include: An initial tensor construction module is used to obtain a road traffic image using a monocular camera, and to construct an initial tensor containing a color channel based on the obtained road traffic image; A tensor construction module to be processed is used to obtain the coordinate information of each pixel in each road traffic image, add the coordinate information of the pixel to the color channel corresponding to the pixel, and obtain the tensor to be processed corresponding to the road traffic image; A feature tensor acquisition module, used for inputting the tensor to be processed into a pre-trained neural network including an attention module to obtain a feature tensor representing a drivable area and a non-drivable area; A drivable area detection module, used to construct an outline of the drivable area based on the pixel points representing the drivable area in the feature tensor, and control the vehicle to travel within the outline of the drivable area; The neural network is a semantic segmentation network. The tensor to be processed is input into a pre-trained neural network including an attention module to obtain a feature tensor representing a drivable area and a non-drivable area, including: Inputting the tensor to be processed into the first sublayer in the first branch layer in the semantic segmentation network to obtain a first tensor; Inputting the first tensor into the second sublayer in the first branch layer and the first subbranch layer in the second branch layer respectively, to obtain a second tensor and a third tensor, wherein the second sublayer includes: a third convolutional layer, a third batch normalization layer, a third activation function layer, a first basic module and a second basic module in sequence, and the first subbranch layer includes: a fourth convolutional layer, a fourth batch normalization layer, a fourth activation function layer, a third basic module and a fourth basic module in sequence; Input the second tensor and the third tensor into the first feature fusion layer to obtain the fourth tensor and the fifth tensor; Input the fourth tensor into the fifth basic module in the first branch layer, input the output of the fifth basic module into the sixth basic module to obtain a sixth tensor, input the fifth tensor into the seventh basic module in the second branch layer, input the output of the seventh basic module into the eighth basic module to obtain a seventh tensor, input the fifth tensor into the first branch sub-layer in the third branch layer to obtain an eighth tensor, wherein the first branch sub-layer sequentially includes: a fifth convolutional layer, a fifth batch normalization layer, a fifth activation function layer, a ninth basic module, and a tenth basic module; Input the sixth tensor, the seventh tensor, and the eighth tensor into the second feature fusion layer to obtain the ninth tensor, the tenth tensor, and the eleventh tensor respectively; Input the ninth tensor into the eleventh basic module in the first branch layer, input the output of the eleventh basic module into the twelfth basic module, and obtain the twelfth tensor; input the tenth tensor into the thirteenth basic module in the second branch layer, input the output of the thirteenth basic module into the fourteenth basic module, and obtain the thirteenth tensor; input the eleventh tensor into the fifteenth basic module in the third branch layer, input the output of the fifteenth basic module into the sixteenth basic module, and obtain the fourteenth tensor; input the eleventh tensor into the fourth branch layer, and obtain the fifteenth tensor; the fourth branch layer sequentially includes: a sixth convolutional layer, a sixth batch normalization layer, a sixth activation function layer, a seventeenth basic module, and an eighteenth basic module; The twelfth tensor, the thirteenth tensor, the fourteenth tensor, and the fifteenth tensor are input into the third feature fusion layer to obtain the sixteenth tensor, the seventeenth tensor, the eighteenth tensor, and the nineteenth tensor respectively; Input the tensor output by the third feature fusion layer into the tensor concatenation layer to obtain a concatenated tensor; The concatenated tensor is input into the seventh convolutional layer, and the output of the seventh convolutional layer is input into the eighth convolutional layer to obtain the output tensor; Perform Argmax operation on the output tensor along the 0th dimension to obtain the tensor to be processed, and upsample the tensor to be processed to obtain the feature tensor.

8. A computer device, characterized in that: include: A processor, a memory and a bus, wherein the memory stores machine-readable instructions executable by the processor, and when the computer device is running, the processor and the memory communicate via the bus, and when the machine-readable instructions are executed by the processor, the steps of the drivable area detection method as described in any one of claims 1 to 6 are performed.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the steps of the drivable area detection method as claimed in any one of claims 1 to 6 are executed.

Images