Obstacle detection method, system, device and storage medium

Abstract

The application discloses an obstacle detection method, system, device and storage medium, and belongs to the technical field of image recognition. The method comprises the following steps: extracting a deep feature map of a target image; performing multi-task learning on the deep feature map to obtain a depth map and a semantic segmentation map of the target image; and performing obstacle detection based on the depth map and the semantic segmentation map to determine the obstacles in the target image. The application aims to improve the universality of the obstacle detection method and solve the technical problem that the existing obstacle detection method cannot detect obstacles outside the label category and has low universality.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of image recognition, and particularly relates to a barrier detection method, system, device and storage medium. BACKGROUND

[0002] Environment perception is the basis and core of an automatic driving system, and effective detection of general barriers is related to the safety of the automatic driving system. The general barriers can include pedestrians, vehicles, railings and signs.

[0003] At present, barrier detection based on vision is usually realized based on a target detection algorithm and an image semantic segmentation algorithm. In the target detection algorithm, the target to be recognized is artificially limited, and only the object to be detected is recognized. In the image semantic segmentation algorithm, a specified target is pixel-labeled according to image content, and the specified target is also artificially limited, and only the object to be detected is labeled. Therefore, the target detection algorithm and the image semantic segmentation algorithm can only give barrier information of a specific label category, and cannot detect barriers outside the label category, and the general applicability is low. SUMMARY

[0004] Therefore, the present application provides a barrier detection method, system, device and storage medium, which aims to improve the general applicability of the barrier detection method and solve the technical problem that the existing barrier detection method cannot detect barriers outside the label category and has low general applicability.

[0005] The present application provides a barrier detection method, system, device and storage medium, and the barrier detection method comprises the following steps:

[0006] extracting a deep feature map of a target image;

[0007] performing multi-task learning on the deep feature map to obtain a depth map and a semantic segmentation map of the target image;

[0008] performing barrier detection based on the depth map and the semantic segmentation map to determine barriers in the target image.

[0009] In a possible implementation of the present application, the step of performing multi-task learning on the deep feature map to obtain a depth map and a semantic segmentation map of the target image comprises:

[0010] inputting the deep feature map as the input of a first layer network of a multi-task learning network, and performing depth task learning and semantic segmentation task learning using the deep feature map to generate a depth feature map and a semantic segmentation feature map;

[0011] The deep feature map and the semantic segmentation feature map are used as inputs to the next layer of the multi-task learning network. Feature fusion is performed on the deep feature map and the semantic segmentation feature map to obtain a first fused feature map and a second fused feature map.

[0012] The first fused feature map is used to learn deep tasks, and a new deep feature map is generated.

[0013] The second fused feature map is used to learn the semantic segmentation task, and a new semantic segmentation feature map is generated.

[0014] Return to the execution steps: Use the deep feature map and the semantic segmentation feature map as inputs to the next layer of the multi-task learning network, perform feature fusion on the deep feature map and the semantic segmentation feature map to obtain the first fused feature map and the second fused feature map, until the deep map and semantic segmentation map output by the last layer of the multi-task learning network are obtained.

[0015] In one possible implementation of this application, the step of fusing the deep feature map and the semantic segmentation feature map to obtain a first fused feature map and a second fused feature map includes:

[0016] Based on the self-attention mechanism, inter-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map to obtain a first inter-class feature map and a second inter-class feature map. Intra-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map to obtain a first intra-class feature map and a second intra-class feature map.

[0017] The first inter-class feature map and the first intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is convolved to obtain the first fused feature map.

[0018] The second inter-class feature map and the second intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is then convolved to obtain the second fused feature map.

[0019] In one possible implementation of this application, after the step of learning deep tasks using the first fused feature map and regenerating the deep feature map in the last layer of the multi-task learning network, the method further includes:

[0020] The depth feature map is convolved to obtain a single-channel depth feature map;

[0021] The depth feature map of the single channel is mapped to obtain the mapping value of each depth point;

[0022] Based on the mapping value of each depth point and the preset maximum distance, the depth value of each 5-degree depth point is calculated, and the depth feature map of the single channel is updated to obtain a depth map.

[0023] In one possible implementation of this application, after the step of learning the semantic segmentation task using the second fused feature map and regenerating the semantic segmentation feature map in the last layer of the multi-task learning network, the method further includes: generating a semantic segmentation map based on the semantic segmentation feature map using the argmax function.

[0024] In one possible implementation of this application, the category labels in the semantic segmentation map include sky, ground and background. The step of detecting obstacles in the target image based on the depth map and the semantic segmentation map includes: 5. Filtering out depth points in the depth map that belong to sky, ground and background according to the semantic segmentation map.

[0025] The remaining depth points in the depth map are projected into space to generate 3D obstacles.

[0026] This application also provides an obstacle detection system, the system comprising: a feature extraction module for extracting deep feature maps of a target image;

[0027] A multi-task learning module is used to perform multi-task learning on the deep feature map to obtain the depth map and semantic segmentation map of the target image;

[0028] An obstacle target recognition module is used to detect obstacles based on the depth map and the semantic segmentation map, and to determine the obstacles in the target image.

[0029] In one possible implementation of this application, the multi-task learning module includes a multi-task learning network, which comprises a first layer network, multiple intermediate layer networks, and a last layer network. The first layer network includes a depth decoder and a segmentation decoder.

[0030] The deep decoder in the network uses the deep feature map to learn deep tasks and generate a deep feature map. The segmentation decoder in the first layer network uses the deep feature map to learn semantic segmentation tasks and generate a semantic segmentation feature map. The intermediate layer network and the last layer network each include a feature fusion unit, a deep decoder, and a segmentation decoder. The feature fusion unit of the intermediate layer network and the last layer network performs feature fusion on the deep feature map and the semantic segmentation feature map to obtain a first fused feature map and a second fused feature map. The deep decoder of the intermediate layer network and the last layer network uses the first fused feature map to learn deep tasks and regenerates the deep feature map. The segmentation decoder of the intermediate layer network and the last layer network uses the second fused feature map to learn semantic segmentation tasks and regenerates the semantic segmentation feature map. The deep feature map generated by the deep decoder of the last layer network is used as the depth map, and the semantic segmentation feature map generated by the segmentation decoder of the last layer network is used as the semantic segmentation map.

[0031] And / or the feature fusion unit is used to specifically implement: based on a self-attention mechanism, inter-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map to obtain a first inter-class feature map and a second inter-class feature map; intra-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map to obtain a first intra-class feature map and a second intra-class feature map; the first inter-class feature map and the first intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is convolved to obtain a first fused feature map; the second inter-class feature map and the second intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is convolved to obtain a second fused feature map;

[0032] And / or the last layer of the network further includes a depth output unit, which is specifically used to: perform convolution processing on the depth feature map to obtain a single-channel depth feature map; perform mapping processing on the single-channel depth feature map to obtain a mapping value for each depth point; calculate the depth value of each depth point based on the mapping value of each depth point and a preset maximum distance, and update the single-channel depth feature map to obtain a depth map;

[0033] And / or the last layer of the network further includes a semantic segmentation output unit, which is used to specifically implement: generating a semantic segmentation map based on the semantic segmentation feature map using the argmax function;

[0034] And / or the category labels in the semantic segmentation map include sky, ground and background, and the obstacle target recognition module includes: a filtering unit, used to filter and remove depth points in the depth map belonging to sky, ground and background according to the semantic segmentation map; and a projection unit, used to project the remaining depth points in the depth map into space to generate 3D obstacles.

[0035] And / or, the category labels in the semantic segmentation graph may also include pedestrians, non-motorized vehicles, and motorized vehicles.

[0036] Optionally, both the depth decoder and the segmentation decoder are configured with convolutional blocks and bilinear interpolation.

[0037] This application also provides an obstacle detection device, the device comprising: a memory, a processor, and an obstacle detection program stored in the memory and executable on the processor, the obstacle detection program being configured to implement the steps of the obstacle detection method as described above.

[0038] This application also provides a storage medium storing an obstacle detection program, which, when executed by a processor, implements the steps of the obstacle detection method described above.

[0039] This application provides an obstacle detection method, system, device, and storage medium. Compared with existing obstacle detection methods based on object detection and image semantic segmentation algorithms, which cannot detect obstacles outside the labeled categories and have low versatility, this application extracts a deep feature map of the target image; performs multi-task learning on the deep feature map to obtain a depth map and a semantic segmentation map of the target image; and performs obstacle detection based on the depth map and the semantic segmentation map to determine the obstacles in the target image. In this application, the depth map represents the depth of all objects in the target image. By combining the depth map and the semantic segmentation map, the 3D shape of all objects in the target image can be constructed while giving the category of each pixel in the target image, thereby identifying and detecting all obstacles in the target image. There are no obstacles that cannot be identified and detected, thus improving the versatility of the obstacle detection method. Attached Figure Description

[0040] Figure 1 This is a flowchart illustrating the first embodiment of the obstacle detection method of this application;

[0041] Figure 2 This is a schematic diagram of the main framework structure of the obstacle detection method of this application;

[0042] Figure 3 This is a schematic diagram of the feature fusion unit structure of this application;

[0043] Figure 4This is a schematic diagram of the obstacle detection device structure in the hardware operating environment involved in the embodiments of this application.

[0044] The realization of the purpose, functional features and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation

[0045] It should be understood that the specific embodiments described herein are merely illustrative of this application and are not intended to limit this application.

[0046] This application provides an obstacle detection method, such as... Figure 1 As shown, in one embodiment of the obstacle detection method of this application, the obstacle detection method includes the following steps:

[0047] Step S10: Extract the deep feature map of the target image;

[0048] Step S20: Perform multi-task learning on the deep feature map to obtain the depth map and semantic segmentation map of the target image;

[0049] Step S30: Based on the depth map and the semantic segmentation map, perform obstacle detection to determine the obstacles in the target image.

[0050] This embodiment aims to improve the versatility of obstacle detection methods.

[0051] Specifically, in this application, the depth map represents the depth of all objects in the target image. By combining the depth map and the semantic segmentation map, the 3D shape of all objects in the target image can be constructed while giving the category of each pixel in the target image. This allows for the identification and detection of all obstacles in the target image, ensuring that no obstacles are undetectable and improving the versatility of the obstacle detection method. Simultaneously, the depth map accurately reflects the distance between obstacles in the target image and the image capture location, thus accurately obtaining the position of obstacles in 3D space, i.e., the 3D shape of obstacles in the target image.

[0052] Furthermore, in this application, during multi-task learning, feature fusion based on a self-attention mechanism is performed on the feature information of sub-tasks (deep feature maps and semantic segmentation feature maps), which promotes the exchange of information between deep feature maps and semantic segmentation feature maps and effectively improves the expressive power of the remaining sub-tasks.

[0053] Furthermore, in this application, during multi-task learning, the deep feature map and semantic segmentation feature map are fused by performing inter-class feature enhancement between the deep feature map and the semantic segmentation feature map, intra-class feature enhancement of the deep feature map itself, and intra-class feature enhancement of the speech segmentation feature map itself. This allows for the exchange and fusion of information from different locations of the sub-task features, further enhancing the expressive power of the remaining sub-tasks.

[0054] Furthermore, since the depth subtask is a regression task, its output is a continuous floating-point variable. Floating-point variables may lose precision during runtime. Therefore, in this application, the depth feature map is convolved to obtain a single-channel depth feature map. The single-channel depth feature map is then mapped to obtain the mapping value of each depth point. Based on the mapping value of each depth point and the preset maximum distance, the depth value of each depth point is calculated, and the single-channel depth feature map is updated to obtain a depth map. This effectively ensures the accuracy of the depth map in obstacle recognition, thereby improving the accuracy of obstacle detection and recognition.

[0055] Furthermore, because the use of a sensor that requires a reflective signal (LiDAR) can lead to the emitted signal not reflecting when it encounters a background, sky, or ground area at infinity, the accuracy of the depth value predicted by the neural network is generally low, and even false detections of obstacles may occur. Therefore, in this application, based on the semantic segmentation map, depth points belonging to the sky and ground in the depth map are filtered out, and the remaining depth points in the depth map are projected into space to generate 3D obstacles, thus filtering out false detections and improving the accuracy of obstacle detection and recognition.

[0056] Furthermore, in this application, a deep decoder is used when performing deep subtasks and a semantic decoder is used when performing semantic segmentation subtasks. Both the deep decoder and the semantic decoder use the results of cross-pollination of convolutional blocks and bilinear interpolation, thereby improving feature resolution.

[0057] In this embodiment, the specific application scenario is:

[0058] In the field of autonomous driving, environmental perception is the foundation and core of autonomous driving systems, and the effective detection of general obstacles is crucial to the safety of these systems. General obstacles can include pedestrians, vehicles, guardrails, signs, etc. Currently, vision-based obstacle detection is typically implemented using object detection and image semantic segmentation algorithms. In object detection algorithms, the targets to be identified are subject to manually defined restrictions, only identifying objects that need to be detected. Image semantic segmentation algorithms label specified targets pixel by pixel based on image content; these specified targets are also subject to manually defined restrictions, only labeling objects that need to be detected. Therefore, object detection and image semantic segmentation algorithms can only provide obstacle information for specific label categories and cannot detect obstacles outside of these categories, resulting in low versatility.

[0059] The specific steps are as follows:

[0060] Step S10: Extract the deep feature map of the target image.

[0061] As an example, the target image ( Figure 2 The RGB image shown can be a 2D image captured by a monocular camera or a 2D image obtained by a single-line lidar scan.

[0062] As an example, the step of extracting the deep feature map of the target image includes the following steps:

[0063] The target image is input into a pre-trained deep convolutional neural network. Figure 2 The backbone network shown is used to obtain a feature map downsampled by 32 times, which is then used as the deep feature map of the target image. Figure 2 The backbone network features shown are described below. The pre-trained deep convolutional neural network includes several convolutional layers, activation function layers, and batch regularization layers.

[0064] As an example, a pre-trained deep convolutional neural network can be a ResNet50 network.

[0065] Step S20: Perform multi-task learning on the deep feature map to obtain the depth map and semantic segmentation map of the target image.

[0066] As an example, the step of performing multi-task learning on the deep feature map to obtain the depth map and semantic segmentation map of the target image includes the following steps:

[0067] Step S21: Use the deep feature map as the input to the first layer of the multi-task learning network, and use the deep feature map to learn the deep task and the semantic segmentation task respectively, to generate deep feature map and semantic segmentation feature map.

[0068] As an example, such as Figure 2 As shown, the first layer of the multi-task learning network includes a deep decoder and a segmentation decoder. The deep feature map is input into the deep decoder, and the output is a deep feature map. Figure 2 The deep feature map (shown in the image) is input into the segmentation decoder to obtain the semantic segmentation feature map. Figure 2 (Segmentation features shown).

[0069] Step S22: Use the deep feature map and the semantic segmentation feature map as inputs to the next layer of the multi-task learning network, and perform feature fusion on the deep feature map and the semantic segmentation feature map to obtain a first fused feature map and a second fused feature map.

[0070] As an example, such as Figure 2 In the multi-task learning network, there are three intermediate layers, each including a feature fusion unit, a deep decoder, and a segmentation decoder. The feature fusion unit receives the deep feature map and semantic segmentation map from the previous layer of the current layer, and performs feature fusion on the deep feature map and semantic segmentation map to obtain a first fused feature map and a second fused feature map.

[0071] As an example, the step of fusing the deep feature map and the semantic segmentation feature map to obtain a first fused feature map and a second fused feature map includes the following steps:

[0072] Step A1: Based on the self-attention mechanism, perform inter-class feature enhancement on the deep feature map and the semantic segmentation feature map to obtain a first inter-class feature map and a second inter-class feature map. Perform intra-class feature enhancement on the deep feature map and the semantic segmentation feature map to obtain a first intra-class feature map and a second intra-class feature map.

[0073] As an example, such as Figure 3 As shown, the steps of performing inter-class feature enhancement on the deep feature map and the semantic segmentation feature map based on the self-attention mechanism to obtain the first inter-class feature map and the second inter-class feature map include:

[0074] Step A101: Perform a linear transformation on the deep feature map and the semantic segmentation feature map to obtain the query-key-value triplet feature map (Q) corresponding to the deep feature map. D K D V D ), and the query-key-value triple feature map (Q) corresponding to the semantic segmentation feature map. S K S V S ).

[0075] As an example, the query-key-value triplet feature map (Q) corresponding to the deep feature map is... D K D V D This is obtained through the following linear formula:

[0076] Q D =W θ F D +W θ ′;

[0077] K D =W φ F D +W φ ′;

[0078] V D =W ξ F D +W ξ ′;

[0079] Among them, F D It is a depth feature map, W θ W θ ′、W φ W φ ′、W ξ W ξ ′ is a learnable matrix, which is implemented using 1*1 convolutions in convolutional neural networks.

[0080] It should be noted that the semantic segmentation feature map corresponds to the query-key-value triplet feature map (Q). S K S V S The method for obtaining the deep feature map and the corresponding query-key-value triplet feature map (Q) D K D V D The method for obtaining ) is the same, and will not be repeated here.

[0081] Step A102: In the channel dimension, extract the query-key-value triplet feature map (Q) corresponding to the depth feature map. D K D V D The query-key-value triple feature map (Q) corresponding to the semantic segmentation feature map. S K S V S It is divided into multiple heads.

[0082] As an example, if the deep feature map corresponds to the query-key-value triplet feature map (Q... D K D V D The query-key-value triplet feature map (Q) corresponding to the semantic segmentation feature map. S K S V S Each head has 32 channels, and there are 4 heads, so each head has 8 channels.

[0083] Step A103: Based on the self-attention mechanism, calculate the key feature map K corresponding to the depth feature map in each head. D The query feature map Q corresponding to the semantic segmentation feature map S The similarity between them is used to obtain the first-class inter-attention matrix.

[0084] Step A104: Based on the first inter-class attention matrix, process the value feature map V corresponding to the deep feature map. D Weighted calculations yield the first-class feature map.

[0085] As an example, steps A103 and A104 can be represented as follows:

[0086]

[0087] Where j is the spatial index of the feature map. This is the query feature map corresponding to the semantic segmentation feature map of the current header; This is the key feature map corresponding to the depth feature map of the current head; The query feature map is the semantic segmentation feature map corresponding to the remaining heads, where the remaining heads are the heads remaining after removing the current head from the multiple heads that are divided. The key feature map corresponding to the depth feature map of the remaining heads; This represents the value feature map corresponding to the depth feature map of the current head; T denotes the transpose operation, and C′ is the number of channels. It should be noted that... This is the inter-class feature map corresponding to the current head, and the inter-class feature maps corresponding to all heads. This forms the first type of inter-class feature map.

[0088] Step A105: Based on the self-attention mechanism, calculate the key feature map K corresponding to the semantic segmentation feature map in each head. S The query feature map Q corresponding to the depth feature map D The similarity between them is used to obtain the second type of inter-class attention matrix.

[0089] Step A106: Based on the second inter-class attention matrix, process the value feature map V corresponding to the semantic segmentation feature map. S Weighted calculations yield the second type of feature map.

[0090] As an example, steps A105 and A106 can be represented as follows:

[0091]

[0092] Where j is the spatial index of the feature map. This is the query feature map corresponding to the depth feature map of the current head; This is the key feature map corresponding to the semantic segmentation feature map of the current header; The query feature map is the depth feature map corresponding to the remaining heads, where the remaining heads are the heads remaining after removing the current head from the multiple heads that are divided. The key feature map corresponding to the semantic segmentation feature map of the remaining heads; This represents the value feature map corresponding to the semantic segmentation feature map of the current header; T denotes the transpose operation, and C′ is the number of channels. It should be noted that... This is the inter-class feature map corresponding to the current head, and the inter-class feature maps corresponding to all heads. This forms the second type of inter-class feature map.

[0093] As an example, such as Figure 3 As shown, the steps of performing intra-class feature enhancement on the deep feature map and the semantic segmentation feature map based on the self-attention mechanism to obtain the first intra-class feature map and the second intra-class feature map include:

[0094] Step A107: In the spatial dimension, the query-key-value triplet feature map (Q) corresponding to the depth feature map is... D K D V D The query-key-value triple feature map (Q) corresponding to the semantic segmentation feature map. S K S V S The data is divided into multiple local windows; in the channel dimension, the query-key-value triplet feature map corresponding to the depth feature map in each local window is... The query-key-value triple feature map corresponding to the semantic segmentation feature map It is divided into multiple heads.

[0095] Step A108: For the query-key-value triplet feature map corresponding to the depth feature map. Each head in the process is divided into multiple windows of preset sizes; the preset sizes of the windows in each head are not equal.

[0096] Step A109: Based on the self-attention mechanism, calculate the key feature map K corresponding to the depth feature map in each head. D The query feature map Q corresponding to the depth feature map D The similarity between them is used to obtain the first type of internal attention matrix.

[0097] Step A110: Based on the first intra-class attention matrix, process the value feature map V corresponding to the depth feature map. D Weighted calculations yield the first type of internal feature map.

[0098] As an example, steps A109 and A110 can be represented as follows:

[0099]

[0100] Where i is the spatial index of the feature map in the local window where the current head is located. This is the query feature map corresponding to the depth feature map of the current head; This is the key feature map corresponding to the depth feature map of the current head; The query feature map is the depth feature map of the other heads in the local window where the current head is located, and the preset size of the window in the other heads is smaller than the preset size of the window in the current head; It is the key feature map corresponding to the depth feature map of the other heads in the local window where the current head is located, and the preset size of the window in the other heads is smaller than the preset size of the window in the current head; This represents the value feature map corresponding to the depth feature map of the current head; T denotes the transpose operation, and C′ is the number of channels. It should be noted that... This is the intra-class feature map corresponding to the current head, and the intra-class feature maps corresponding to all heads. This forms the first type of internal feature map.

[0101] Step A111: For the query-key-value triplet feature map corresponding to the semantic segmentation feature map... Each head in the process is divided into multiple windows of preset sizes; the preset sizes of the windows in each head are not equal.

[0102] Step A112: Based on the self-attention mechanism, calculate the key feature map K corresponding to the semantic segmentation feature map in each head. S The query feature map Q corresponding to the semantic segmentation feature map S The similarity between them is used to obtain the second type of internal attention matrix.

[0103] Step A113: Based on the second type of intra-attention matrix, process the value feature map V corresponding to the semantic segmentation feature map. S Weighted calculations yield the second type of internal feature map.

[0104] As an example, steps A112 and A113 can be represented as:

[0105]

[0106] Where i is the spatial index of the feature map in the local window where the current head is located. This is the query feature map corresponding to the semantic segmentation feature map of the current header; This is the key feature map corresponding to the semantic segmentation feature map of the current header; The query feature map is the semantic segmentation feature map of the other headers in the local window where the current header is located. The preset size of the window in the other header is smaller than the preset size of the window in the current header. It is the key feature map corresponding to the semantic segmentation feature maps of the other heads in the local window where the current head is located, and the preset size of the window in the other heads is smaller than the preset size of the window in the current head; This represents the value feature map corresponding to the semantic segmentation feature map of the current head; T denotes the transpose operation, and C′ is the number of channels. It should be noted that the current head is the head used to calculate the second type of intra-class attention in step A112. It should be noted that... This is the intra-class feature map corresponding to the current head, and the intra-class feature maps corresponding to all heads. This forms the second type of internal feature map.

[0107] Step A2: Concatenate the first inter-class feature map and the first intra-class feature map based on the channel dimension, and perform convolution processing on the concatenated feature map to obtain the first fused feature map.

[0108] As an example, if the inter-class feature map has C channels and the intra-class feature map also has C channels, then the concatenated feature map has 2C channels. Convolution processing is performed on the concatenated feature map to obtain the first fused feature map with C channels.

[0109] Step A4: Concatenate the second inter-class feature map and the second intra-class feature map based on the channel dimension, and perform convolution processing on the concatenated feature map to obtain the second fused feature map.

[0110] As an example, if the inter-class feature map has C channels and the intra-class feature map also has C channels, then the concatenated feature map has 2C channels. Convolution processing is performed on the concatenated feature map to obtain the second fused feature map with C channels.

[0111] Step S23: Use the first fused feature map to learn the deep task and regenerate the deep feature map.

[0112] As an example, the first fused feature map is input into the depth decoder, and the output is a depth feature map.

[0113] Step S24: Use the second fused feature map to learn the semantic segmentation task and regenerate the semantic segmentation feature map.

[0114] As an example, the second fused feature map is input into the segmentation decoder to obtain the semantic segmentation feature map.

[0115] Step S25: Return to step S22: Use the deep feature map and the semantic segmentation feature map as inputs to the next layer of the multi-task learning network, perform feature fusion on the deep feature map and the semantic segmentation feature map to obtain the first fused feature map and the second fused feature map, until the depth map and semantic segmentation map output by the last layer of the multi-task learning network are obtained.

[0116] As an example, in a multi-task learning network, there is a final layer, which includes a feature fusion unit, a deep decoder, and a segmentation decoder. The operation of the feature fusion unit, deep decoder, and segmentation decoder in the final layer is the same as that in the intermediate layers, and will not be repeated here. The deep feature map output by the deep decoder in the final layer is the depth map, and the semantic segmentation map output by the segmentation decoder is the semantic segmentation map.

[0117] As an example, both the depth decoder and the segment decoder are structured with convolutional blocks interleaved with bilinear interpolation.

[0118] As an example, the last layer of the network includes, in addition to the feature fusion unit, deep decoder, and segmentation decoder mentioned in the example above, a deep output unit and a semantic segmentation output unit.

[0119] The depth output unit is used for:

[0120] Step B1: Perform convolution processing on the depth feature map to obtain a single-channel depth feature map.

[0121] As an example, a depth feature map with C channels is convolved to obtain a single-channel depth feature map with 1 channel.

[0122] Step B2: Perform mapping processing on the depth feature map of the single channel to obtain the mapping value of each depth point.

[0123] As an example, the Sigmoid function is used to map the depth value of each depth point in a single-channel depth feature map to 0-1.

[0124] Step B3: Based on the mapping value of each depth point and the preset maximum distance, calculate the depth value of each depth point, and update the depth feature map of the single channel to obtain a depth map.

[0125] As an example, the mapping value of each depth point is multiplied by a preset maximum distance, and the resulting value is used as the depth value of each depth point. This is then used to update the single-channel depth feature map, resulting in a depth map. Figure 2 (The monocular depth estimation results are shown).

[0126] The semantic segmentation output unit is used for:

[0127] Based on the semantic segmentation feature map, a semantic segmentation map is generated using the argmax function. Figure 2 (Image semantic segmentation result shown).

[0128] As an example, during the training phase, the multi-task learning network uses the softmax function to generate a semantic segmentation prediction map and calculates the network loss of the semantic segmentation sub-task learning network using the cross-entropy loss function. The semantic segmentation sub-task learning network is then trained until its network loss converges.

[0129] Step S30: Based on the depth map and the semantic segmentation map, perform obstacle detection to determine the obstacles in the target image.

[0130] As an example, the category labels in the semantic segmentation map include sky, ground, and background. The step of detecting obstacles in the target image based on the depth map and the semantic segmentation map includes the following steps:

[0131] Step S31: Based on the semantic segmentation map, filter out depth points in the depth map that belong to the sky, ground, and background;

[0132] Step S32: Project the remaining depth points in the depth map into space to generate 3D obstacles.

[0133] As an example, each depth point in the depth map can be represented as in This is represented by the x-coordinate of each depth point in the depth map. This is represented by the ordinate of each depth point in the depth map. Represented as the depth value of each depth point, after filtering out depth points belonging to the sky, ground, and background in the depth map, the remaining depth points can be represented as... These remaining depth points are the depth points of the obstacles. Using camera intrinsic and extrinsic parameters... Projecting into space yields 3D point cloud coordinates. The formula for calculating the 3D point cloud coordinates in space is as follows:

[0134]

[0135] Where K is the product matrix of camera intrinsic and extrinsic parameters, and T is the product matrix of camera intrinsic and extrinsic parameters.

[0136] As an example, the category labels in the semantic segmentation graph also include pedestrians, non-motorized vehicles, and motorized vehicles. In this example, the inclusion of pedestrians, non-motorized vehicles, and motorized vehicles in the category labels of the semantic segmentation graph can enhance the function of the semantic segmentation graph in relation to the depth map.

[0137] This application provides an obstacle detection method, system, device, and storage medium. Compared with existing obstacle detection methods based on object detection and image semantic segmentation algorithms, which cannot detect obstacles outside the labeled categories and have low versatility, this application extracts a deep feature map of the target image; performs multi-task learning on the deep feature map to obtain a depth map and a semantic segmentation map of the target image; and performs obstacle detection based on the depth map and the semantic segmentation map to determine the obstacles in the target image. In this application, the depth map represents the depth of all objects in the target image. By combining the depth map and the semantic segmentation map, the 3D shape of all objects in the target image can be constructed while giving the category of each pixel in the target image, thereby identifying and detecting all obstacles in the target image. There are no obstacles that cannot be identified and detected, thus improving the versatility of the obstacle detection method.

[0138] Reference Figure 4 , Figure 4 This is a schematic diagram of the obstacle detection device structure in the hardware operating environment involved in the embodiments of this application.

[0139] like Figure 4 As shown, the obstacle detection device may include: a processor 1001, such as a central processing unit (CPU), a communication bus 1002, a user interface 1003, a network interface 1004, and a memory 1005. The communication bus 1002 is used to enable communication between these components. The user interface 1003 may include a display screen or an input unit such as a keyboard; optionally, the user interface 1003 may also include a standard wired interface or a wireless interface. The network interface 1004 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface). The memory 1005 may be a high-speed random access memory (RAM) or a stable non-volatile memory (NVM), such as a disk drive. The memory 1005 may also optionally be a storage device independent of the aforementioned processor 1001.

[0140] Those skilled in the art will understand that Figure 4The structure shown does not constitute a limitation on the obstacle detection device and may include more or fewer components than shown, or combine certain components, or have different component arrangements.

[0141] like Figure 4 As shown, the memory 1005, which serves as a storage medium, may include an operating system, a data storage module, a network communication module, a user interface module, and an obstacle detection program.

[0142] exist Figure 4 In the obstacle detection device shown, the network interface 1004 is mainly used for data communication with other devices; the user interface 1003 is mainly used for data interaction with the user; the processor 1001 and the memory 1005 in the obstacle detection device of this application can be set in the obstacle detection device, and the obstacle detection device implements the steps of the obstacle detection method described above by calling the obstacle detection program stored in the memory 1005 through the processor 1001.

[0143] The specific implementation of the obstacle detection device in this application is basically the same as the embodiments of the obstacle detection method described above, and will not be repeated here.

[0144] This application also provides an obstacle detection system, the system comprising:

[0145] The feature extraction module is used to extract deep feature maps of the target image;

[0146] A multi-task learning module is used to perform multi-task learning on the deep feature map to obtain the depth map and semantic segmentation map of the target image;

[0147] An obstacle target recognition module is used to detect obstacles based on the depth map and the semantic segmentation map, and to determine the obstacles in the target image.

[0148] In one possible implementation of this application, the multi-task learning module includes a multi-task learning network, which comprises a first-layer network, multiple intermediate-layer networks, and a final-layer network, wherein...

[0149] The first layer network includes a deep decoder and a segmentation decoder. The deep decoder in the first layer network uses the deep feature map to learn deep tasks and generate a deep feature map. The segmentation decoder in the first layer network uses the deep feature map to learn semantic segmentation tasks and generate a semantic segmentation feature map. The intermediate layer network and the last layer network each include a feature fusion unit, a deep decoder, and a segmentation decoder. The feature fusion unit of the intermediate layer network and the last layer network performs feature fusion on the deep feature map and the semantic segmentation feature map to obtain a first fused feature map and a second fused feature map. The deep decoder of the intermediate layer network and the last layer network uses the first fused feature map to learn deep tasks and regenerate a deep feature map. The segmentation decoder of the intermediate layer network and the last layer network uses the second fused feature map to learn semantic segmentation tasks and regenerate a semantic segmentation feature map. The deep feature map generated by the deep decoder of the last layer network is used as the deep map, and the semantic segmentation feature map generated by the segmentation decoder of the last layer network is used as the semantic segmentation map.

[0150] And / or the feature fusion unit is used to specifically implement: based on a self-attention mechanism, inter-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map to obtain a first inter-class feature map and a second inter-class feature map; intra-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map respectively to obtain a first intra-class feature map and a second intra-class feature map; the first inter-class feature map and the first intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is convolved to obtain a first fused feature map; the second inter-class feature map and the second intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is convolved to obtain a second fused feature map;

[0151] And / or the last layer of the network further includes a depth output unit, which is specifically used to: perform convolution processing on the depth feature map to obtain a single-channel depth feature map; perform mapping processing on the single-channel depth feature map to obtain a mapping value for each depth point; calculate the depth value of each depth point based on the mapping value of each depth point and a preset maximum distance, and update the single-channel depth feature map to obtain a depth map;

[0152] And / or the last layer of the network further includes a semantic segmentation output unit, which is used to specifically implement: generating a semantic segmentation map based on the semantic segmentation feature map using the argmax function;

[0153] And / or the category labels in the semantic segmentation map include sky, ground and background, and the obstacle target recognition module includes: a filtering unit, used to filter and remove depth points in the depth map belonging to sky, ground and background according to the semantic segmentation map; and a projection unit, used to project the remaining depth points in the depth map into space to generate 3D obstacles.

[0154] In one possible implementation of this application, both the depth decoder and the segmentation decoder are structured with convolutional blocks and bilinear interpolation.

[0155] The specific implementation of the obstacle detection system in this application is basically the same as the embodiments of the obstacle detection method described above, and will not be repeated here.

[0156] This application provides a storage medium that stores one or more programs, which can be executed by one or more processors to implement the steps of the obstacle detection method described in any of the above claims.

[0157] The specific implementation of the storage medium in this application is basically the same as the embodiments of the obstacle detection method described above, and will not be repeated here.

[0158] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the obstacle detection method described above.

[0159] The specific implementation of the computer program product in this application is basically the same as the various embodiments of the obstacle detection method described above, and will not be repeated here.

[0160] It should be noted that, in this document, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or system that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or system. Unless otherwise specified, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or system that includes that element.

[0161] The sequence numbers of the embodiments in this application are for descriptive purposes only and do not represent the superiority or inferiority of the embodiments.

[0162] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0163] The above are merely preferred embodiments of this application and do not limit the patent scope of this application. Any equivalent structural or procedural transformations made using the content of this application's specification and drawings, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of this application.

Claims

1. An obstacle detection method, characterized in that, The obstacle detection method includes the following steps: Extract deep feature maps from the target image; Multi-task learning is performed on the deep feature map to obtain the depth map and semantic segmentation map of the target image; Obstacle detection is performed based on the depth map and the semantic segmentation map to determine obstacles in the target image; The step of performing multi-task learning on the deep feature map to obtain the depth map and semantic segmentation map of the target image includes: The deep feature map is used as the input to the first layer of the multi-task learning network. The deep feature map is used to learn the deep task and the semantic segmentation task respectively, and the deep feature map and the semantic segmentation feature map are generated. The deep feature map and the semantic segmentation feature map are used as inputs to the next layer of the multi-task learning network. Feature fusion is performed on the deep feature map and the semantic segmentation feature map to obtain a first fused feature map and a second fused feature map. The first fused feature map is used to learn deep tasks, and a new deep feature map is generated. The second fused feature map is used to learn the semantic segmentation task, and a new semantic segmentation feature map is generated. Return to the execution steps: Use the deep feature map and the semantic segmentation feature map as input to the next layer of the multi-task learning network, perform feature fusion on the deep feature map and the semantic segmentation feature map to obtain the first fused feature map and the second fused feature map, until the deep map and semantic segmentation map output by the last layer of the multi-task learning network are obtained; The step of fusing the deep feature map and the semantic segmentation feature map to obtain a first fused feature map and a second fused feature map includes: Based on the self-attention mechanism, inter-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map to obtain a first inter-class feature map and a second inter-class feature map. Intra-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map to obtain a first intra-class feature map and a second intra-class feature map. The first inter-class feature map and the first intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is convolved to obtain the first fused feature map. The second inter-class feature map and the second intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is then convolved to obtain the second fused feature map.

2. The obstacle detection method as described in claim 1, characterized in that, In the last layer of the multi-task learning network, after the step of learning deep tasks using the first fused feature map and regenerating the deep feature map, the network further includes: The depth feature map is convolved to obtain a single-channel depth feature map; The depth feature map of the single channel is mapped to obtain the mapping value of each depth point; Based on the mapping value of each depth point and the preset maximum distance, the depth value of each depth point is calculated, and the depth feature map of the single channel is updated to obtain a depth map.

3. The obstacle detection method as described in claim 1, characterized in that, In the last layer of the multi-task learning network, after the step of learning the semantic segmentation task using the second fused feature map and regenerating the semantic segmentation feature map, the network further includes: Based on the semantic segmentation feature map, a semantic segmentation map is generated using the argmax function.

4. The obstacle detection method as described in claim 1, characterized in that, The category labels in the semantic segmentation map include sky, ground, and background. The step of detecting obstacles in the target image based on the depth map and the semantic segmentation map includes: Based on the semantic segmentation map, depth points in the depth map belonging to the sky, ground, and background are filtered out. The remaining depth points in the depth map are projected into space to generate 3D obstacles.

5. An obstacle detection system, characterized in that, The system includes: The feature extraction module is used to extract deep feature maps of the target image; A multi-task learning module is used to perform multi-task learning on the deep feature map to obtain the depth map and semantic segmentation map of the target image; An obstacle target recognition module is used to detect obstacles based on the depth map and the semantic segmentation map, and to determine obstacles in the target image; The multi-task learning module includes a multi-task learning network, which comprises a first-layer network, multiple intermediate-layer networks, and a final-layer network. The first layer network includes a deep decoder and a segmentation decoder. The deep decoder in the first layer network uses the deep feature map to learn deep tasks and generate a deep feature map. The segmentation decoder in the first layer network uses the deep feature map to learn semantic segmentation tasks and generate a semantic segmentation feature map. The intermediate layer network and the last layer network each include a feature fusion unit, a deep decoder, and a segmentation decoder. The feature fusion unit of the intermediate layer network and the last layer network fuses the deep feature map and the semantic segmentation feature map to obtain a first fused feature map and a second fused feature map. The deep decoder of the intermediate layer network and the last layer network uses the first fused feature map to learn deep tasks and regenerate a deep feature map. The segmentation decoder of the intermediate layer network and the last layer network uses the second fused feature map to learn semantic segmentation tasks and regenerate a semantic segmentation feature map. The deep feature map generated by the deep decoder of the last layer network is used as the deep map, and the semantic segmentation feature map generated by the segmentation decoder of the last layer network is used as the semantic segmentation map. The feature fusion unit is specifically implemented as follows: based on a self-attention mechanism, inter-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map to obtain a first inter-class feature map and a second inter-class feature map; intra-class feature enhancement is performed on the deep feature map and the semantic segmentation feature map to obtain a first intra-class feature map and a second intra-class feature map; the first inter-class feature map and the first intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is subjected to convolution processing to obtain a first fused feature map; the second inter-class feature map and the second intra-class feature map are concatenated based on the channel dimension, and the concatenated feature map is subjected to convolution processing to obtain a second fused feature map.

6. The obstacle detection system as described in claim 5, characterized in that: The last layer of the network also includes a depth output unit, which is specifically used to: perform convolution processing on the depth feature map to obtain a single-channel depth feature map; perform mapping processing on the single-channel depth feature map to obtain a mapping value for each depth point; calculate the depth value of each depth point based on the mapping value of each depth point and a preset maximum distance, and update the single-channel depth feature map to obtain a depth map; And / or the last layer of the network further includes a semantic segmentation output unit, which is used to specifically implement: generating a semantic segmentation map based on the semantic segmentation feature map using the argmax function; And / or the category labels in the semantic segmentation map include sky, ground and background, and the obstacle target recognition module includes: a filtering unit, used to filter and remove depth points in the depth map belonging to sky, ground and background according to the semantic segmentation map; and a projection unit, used to project the remaining depth points in the depth map into space to generate 3D obstacles. And / or, the category labels in the semantic segmentation graph may also include pedestrians, non-motorized vehicles, and motorized vehicles.

7. The obstacle detection system as described in claim 5, characterized in that, Both the depth decoder and the segmentation decoder have a structure that alternates between convolutional blocks and bilinear interpolation.

8. An obstacle detection device, characterized in that, The device includes: a memory, a processor, and an obstacle detection program stored in the memory and executable on the processor, the obstacle detection program being configured to implement the steps of the obstacle detection method as described in any one of claims 1 to 4.

9. A storage medium, characterized in that, The storage medium stores an obstacle detection program, which, when executed by a processor, implements the steps of the obstacle detection method as described in any one of claims 1 to 4.

Images