Driving area detection method, system and device and readable storage medium

Abstract

A drivable area detection method, system and device and a readable storage medium relate to the field of off-road vehicle aided driving, and specifically comprise: when it is detected that a road scene is unstructured, respectively extracting target text features and target image features in the unstructured road scene; performing cross-modal feature fusion processing on the target text feature and the target image feature through a preset cross-modal feature fusion model to obtain a target fusion feature; and performing learning processing on the target fusion feature through a preset multi-task learning module to obtain a detection result of the travelable area. According to the method and the device, the drivable area can be detected in a scene in which the data set does not appear.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of off-road vehicle assisted driving, in particular to a drivable area detection method, system, device and readable storage medium. BACKGROUND

[0002] With the increasing demand for off-road environment applications, the scene analysis and drivable area detection are generally performed by using a supervised learning method in the related art. However, the data set of the current supervised learning method cannot cover complex and variable unstructured scenes, such as scenes including muddy, stone, pothole, steep slope and other obstacles, such as country roads, mountains, snow fields and other scenes, which leads to the inability to effectively detect the drivable area in scenes that do not appear in the data set.

[0003] Therefore, how to detect the drivable area in scenes that do not appear in the data set is a problem to be solved at present. SUMMARY

[0004] The present application provides a drivable area detection method, system, device and readable storage medium, which can detect the drivable area in scenes that do not appear in the data set.

[0005] In a first aspect, the present application provides a drivable area detection method, which comprises: When it is detected that the road scene is unstructured, target text features and target image features under the unstructured road scene are extracted respectively; The target text features and the target image features are processed by a preset cross-modal feature fusion model to obtain target fusion features; The target fusion features are processed by a preset multi-task learning module to obtain a detection result of the drivable area.

[0006] In combination with the first aspect, in an implementation mode, the cross-modal feature fusion model comprises a self-attention mechanism, a cross-attention mechanism, a cross-fusion enhancement layer and a feedforward neural network; The self-attention mechanism is used to output a first target feature vector corresponding to the target text features and a second target feature vector corresponding to the target image features; The cross-attention mechanism is used to fuse the first target feature vector and the second target feature vector to obtain a first fusion feature and a second fusion feature respectively; The cross-fusion enhancement layer is used to weight process the first fusion feature and the second fusion feature to obtain a third fusion feature; The feedforward neural network is used to perform feature mapping and nonlinear conversion on the third fusion feature to obtain the target fusion features.

[0007] With reference to the first aspect, in an implementation form of the first aspect, the first target feature vector comprises a first query vector, a first key vector and a first value vector, the second target feature vector comprises a second query vector, a second key vector and a second value vector, and the fusing the first target feature vector and the second target feature vector to obtain a first fused feature and a second fused feature respectively comprises: determining the first fused feature based on the first query vector, the second key vector and the second value vector; determining the second fused feature based on the second query vector, the first key vector and the first value vector.

[0008] With reference to the first aspect, in an implementation form of the first aspect, the determining the first fused feature based on the first query vector, the second key vector and the second value vector comprises: performing similarity calculation on the first query vector and the second key vector to obtain a weight corresponding to the second key vector; determining the first fused feature based on the weight and the second value vector.

[0009] With reference to the first aspect, in an implementation form of the first aspect, before the step of learning the target fused feature by using the preset multi-task learning module, the method further comprises: decoding an image feature in the target fused feature based on an image decoder to obtain a new image feature; inputting the new image feature and a target text feature into a preset cross-modal feature fusion model to obtain a new target fused feature, and continuing to perform the step of learning the target fused feature by using the preset multi-task learning module according to the new target fused feature.

[0010] With reference to the first aspect, in an implementation form of the first aspect, the method further comprises: when it is detected that the road scene is structured, inputting image data and text data corresponding to the structured road scene into a preset multi-task network general model to obtain a detection result of the drivable area.

[0011] With reference to the first aspect, in an implementation form of the first aspect, the respectively extracting the target text feature and the target image feature in the unstructured road scene comprises: extracting the target text feature in the unstructured road scene from text data corresponding to the unstructured road scene based on a text backbone network; extracting the target image feature in the unstructured road scene from image data corresponding to the unstructured road scene based on an image encoder.

[0012] Secondly, the embodiments of the present application provide a drivable area detection system. The first processing module is configured to extract target text features and target image features in an unstructured road scene respectively when it is detected that the road scene is unstructured. The second processing module is configured to perform cross-modal feature fusion processing on the target text features and the target image features through a preset cross-modal feature fusion model to obtain target fusion features. The third processing module is configured to perform learning processing on the target fusion features through a preset multi-task learning module to obtain a detection result of the drivable area.

[0013] In a third aspect, an embodiment of the present application provides a drivable area detection device, which comprises a processor, a memory, and a drivable area detection program stored in the memory and executable by the processor, wherein the drivable area detection program, when executed by the processor, implements the steps of the drivable area detection method according to any one of the preceding aspects.

[0014] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a drivable area detection program, wherein the drivable area detection program, when executed by a processor, implements the steps of the drivable area detection method according to any one of the preceding aspects.

[0015] The technical scheme provided by the embodiments of the present application has the following beneficial effects: When it is detected that the road scene is unstructured, target text features and target image features in the unstructured road scene are extracted respectively. Cross-modal feature fusion processing is performed on the target text features and the target image features through a preset cross-modal feature fusion model to obtain target fusion features, so that the system can utilize the relationship between different modal features to effectively infer and predict the scene without a data set covering all obstacle scenes. Learning processing is performed on the target fusion features through a preset multi-task learning module to obtain a detection result of the drivable area, which realizes the detection of the drivable area in a scene that does not appear in the data set, and thus improves the accuracy of the detection of the drivable area in a complex environment. BRIEF DESCRIPTION OF DRAWINGS

[0016] Figure 1 FIG. 1 is a flowchart of a drivable area detection method according to an embodiment of the present application; Figure 2 FIG. 2 is a schematic diagram of a cross-modal feature fusion model in the drivable area detection method according to an embodiment of the present application; Figure 3 FIG. 3 is a schematic diagram of a visual language model in the drivable area detection method according to an embodiment of the present application; Figure 4A function module schematic diagram of the driving area detection system of the present application; Figure 5 A hardware structure schematic diagram of the driving area detection device involved in the embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work fall within the scope of protection of the present application.

[0018] In order to make the purpose, technical solutions and advantages of the present application clearer, the embodiments of the present application will be described in further detail below with reference to the drawings.

[0019] In a first aspect, the embodiments of the present application provide a driving area detection method.

[0020] In an embodiment, with reference to Figure 1 , Figure 1 A flowchart schematic diagram of the driving area detection method of the present application is shown in FIG. 1. As shown in FIG. 1, the driving area detection method comprises the following steps. Figure 1 Step S10: When it is detected that the road scene is unstructured, target text features and target image features under the unstructured road scene are extracted respectively.

[0021] Exemplarily, in the embodiments of the present application, the unstructured road scene refers to a road scene with complex road environment information and which cannot be described by simple rules or map structure, such as a rural road, a mountainous area, a snowy area and the like. Such a road contains obstacles that are not seen in the data set, such as stones, tree stumps and the like. The target text features refer to the text information extracted from the unstructured road scene, including but not limited to the text features related to traffic signs and indication signs. The target image features refer to the image information extracted from the unstructured road scene image, including but not limited to the visual features of roads, objects and obstacles.

[0022] Specifically, the text data under the unstructured road scene can be acquired by a navigation system, and the target text features are extracted from the text data for subsequent driving area detection based on the target text features. Meanwhile, the image data under the unstructured road scene can be acquired by a visual sensor, and the target image features are extracted from the image data for subsequent driving area detection based on the target image features.

[0023] ​Step S20: performing cross-modal feature fusion processing on the target text feature and the target image feature through the preset cross-modal feature fusion model to obtain a target fusion feature.

[0024] Exemplarily, in the embodiment of the present application, the preset cross-modal feature fusion model is used for fusion processing of image features and text features for subsequent detection of drivable areas based on the results of the fusion processing; specifically, the target text feature and the target image feature are input into the preset cross-modal feature fusion model, and the features of the two modalities are sequentially processed by each component of the model to output the final target fusion feature.

[0025] Step S30: performing learning processing on the target fusion feature through the preset multi-task learning module to obtain a detection result of the drivable area.

[0026] Exemplarily, in the embodiment of the present application, the multi-task learning module includes a shared network and a specific task branch, the shared network includes a convolution layer and a pooling layer, and the purpose is to extract high-level semantic features in the image; specifically, the convolution layer can be used to perform convolution operation on the input target fusion feature to output a feature map containing local information extracted from the target fusion feature; and the pooling layer is used to perform down-sampling processing on the feature map to retain important features and reduce the resolution, and finally output the pooled feature map.

[0027] It should be noted that the specific task branch includes a classification loss function, a positioning loss function and a contrast loss function, and the purpose is to output the drivable area. After the pooled feature map is input into the specific task branch, the classification loss function, the positioning loss function and the contrast loss function play different roles, and by optimizing the features in different aspects, the drivable area is finally output.

[0028] Specifically, the contrast loss function is used to enhance the classification accuracy between image information and text information, which pulls the image features and corresponding text features of similar regions in the pooled feature map closer, and pulls the image features and corresponding text features of dissimilar regions in the pooled feature map farther, to help enhance the understanding and differentiation ability of the region information, and finally output the optimized fusion feature, so that the classification results of image and text information are more consistent, thereby improving the classification accuracy, so as to ensure the accurate division of drivable areas and non-drivable areas; wherein, the contrast loss function in the embodiment of the present application can be preferably selected as Contrastive Loss.

[0029] The classification loss function is used to improve the recognition accuracy of the model on the drivable area and the non-drivable area, especially for the areas that are difficult to classify. Specifically, the classification loss function can give higher weights to the features in the pooled feature map that are difficult to classify, so as to more accurately distinguish the drivable area and the non-drivable area. In the embodiments of the present application, the classification loss function can be preferably Focal Loss (focal loss function). The positioning loss function is used to locate the position of the obstacle, which can optimize the difference between the bounding box of the predicted pooled feature map and the real obstacle, so as to more accurately find the boundary of the obstacle or the drivable area. The positioning loss function can accurately determine the position of the obstacle by optimizing the error of the bounding box overlap, center distance and size. In the embodiments of the present application, the positioning loss function can be preferably CIOULoss (Complete Intersection over Union Loss, complete intersection over union loss function). It can be understood that the optimization results of the three loss functions are combined to compare, classify and locate the pooled feature map in the three branches, and finally output the accurate drivable area.

[0030] When the present application detects that the road scene is unstructured, target text features and target image features under the unstructured road scene are extracted respectively; the target text features and the target image features are subjected to cross-modal feature fusion processing through a preset cross-modal feature fusion model to obtain target fusion features, so that the system can utilize the relationship between different modal features to effectively infer and predict the scene without the data set covering all obstacle scenes; the target fusion features are subjected to learning processing through a preset multi-task learning module to obtain the detection result of the drivable area, realizing the detection of the drivable area in a scene that does not appear in the data set, and thus improving the accuracy of the detection of the drivable area in a complex environment.

[0031] Further, in an embodiment, the cross-modal feature fusion model comprises a self-attention mechanism, a cross-attention mechanism, a cross-fusion enhancement layer and a feedforward neural network. The self-attention mechanism is used to output a first target feature vector corresponding to the target text features and a second target feature vector corresponding to the target image features. The cross-attention mechanism is used to fuse the first target feature vector and the second target feature vector to obtain a first fusion feature and a second fusion feature respectively. The cross-fusion enhancement layer is used to weight the first fusion feature and the second fusion feature to obtain a third fusion feature. The feedforward neural network is used to perform feature mapping and non-linear conversion on the third fusion feature to obtain the target fusion feature.

[0032] Exemplarily, in the embodiments of the present application, with reference to Figure 2 As shown in the figure, the cross-modal feature fusion model includes a self-attention mechanism, a cross-attention mechanism, a cross-fusion enhancement layer, and a feedforward neural network; wherein the self-attention mechanism is a mechanism for extracting the relationship between input features, for example, the self-attention mechanism can preferably be a scaled dot-product attention; the cross-attention mechanism is an attention mechanism capable of processing different modal information (such as text and image); it can strengthen the connection between modalities by cross-pairing and fusing feature vectors from different modalities; the cross-fusion enhancement layer is to enhance the representation ability of features from different modalities (such as text and image features) by further fusing them, thereby improving the performance of the model; the feedforward neural network (FFNN) further processes these fused features to generate the final target fusion features.

[0033] Specifically, the first target feature vector refers to the text feature vector obtained by the self-attention mechanism, which is used to represent the features of the text; the second target feature vector refers to the vector related to the target image feature, which is used to represent the features of the image; the first target feature vector corresponding to the target text feature can be generated by linear transformation of the input target text feature through the self-attention mechanism; similarly, the second target feature vector corresponding to the target image feature can be generated by linear transformation of the input target image feature.

[0034] It can be understood that in the self-attention mechanism, Q (Query, query vector), K (Key, key vector), and V (Value, value vector) are core components, and the three achieve adaptive fusion of features through dynamic interaction; in the cross-attention mechanism, there are two processes, wherein the first process is to perform cross-attention calculation with the text as the query and the image as the key-value pair to obtain the first fusion feature; the second process is to perform cross-attention calculation with the image as the query and the text as the key-value pair to obtain the second fusion feature; it should be noted that the similarity between the first fusion feature and the second fusion feature can be calculated first through the cross-fusion enhancement layer to obtain the attention weight of each fusion feature; then the first fusion feature and the second fusion feature are weighted and summed through the calculated attention weight to finally generate the third fusion feature, wherein the third fusion feature fuses key information from two modalities and can more accurately reflect the importance of each modality feature in the current drivable area detection task.

[0035] It should be noted that the feedforward neural network includes a fully connected layer and an activation function, the input third fusion feature can be linearly transformed through the fully connected layer to change the dimension and structure of the input feature vector, thereby helping the model better understand the fused features; and the activation function is used to introduce a nonlinear relationship, so that the model can fit more complex data patterns, thereby generating a corresponding target fusion feature for drivable area detection, thereby providing a basis for the final detection task; preferably, the activation function can be ReLU (Rectified Linear Unit) or Sigmoid (logical function).

[0036] Further, in an embodiment, the first target feature vector includes a first query vector, a first key vector, and a first value vector, the second target feature vector includes a second query vector, a second key vector, and a second value vector, and the fusion processing of the first target feature vector and the second target feature vector respectively obtains a first fusion feature and a second fusion feature, including: determining the first fusion feature based on the first query vector, the second key vector, and the second value vector; determining the second fusion feature based on the second query vector, the first key vector, and the first value vector.

[0037] For example, in the cross-attention mechanism in the embodiment of the present application, the first query vector Q1 represents the text information (such as a word or feature in the input sequence) currently to be processed, which is used to inquire the relevance of image information, the first key vector K1 represents the semantic representation of each word in the input sequence, and the first value vector V1 represents the specific content representation of the text feature, such as the semantics, syntax, or context information of each word in the input sequence; the second query vector Q2 represents the image information currently to be processed, which is used to inquire the relevance of the text information, the second key vector K2 represents the local region or feature in the image information, and the second value vector V2 represents the actual content of the local region or feature of the image; wherein the first target feature vector includes the first query vector, the first key vector, and the first value vector, and the second target feature vector includes the second query vector, the second key vector, and the second value vector.

[0038] Specifically, the cross-modal attention weight distribution can be generated by calculating the scaled dot product similarity of the first query vector Q1 and the second key vector K2 of the image modality, and then the weight distribution result is applied to the second value vector V2 (carrying the actual content information of the image region) of the image modality, and finally the text-guided image feature aggregation result, i.e., the first fusion feature, is obtained by weighted summation; it should be noted that the generation method of the second fusion feature is the same as that of the first fusion feature, and for the sake of brevity, it will not be described here.

[0039] Further, in one embodiment, determining the first fusion feature based on the first query vector, the second key vector, and the second value vector includes: The similarity between the first query vector and the second key vector is calculated to obtain the weight corresponding to the second key vector; The first fusion feature is determined based on the weights and the second value vector.

[0040] As an example, in this embodiment, the similarity between the second key vector and the first query vector can be calculated by a dot product operation, and the similarity result can be normalized by the Softmax function (a function that converts input real numbers into a probability distribution) to finally obtain the weight of the second key vector. The higher the weight of the second key vector, the higher its semantic relevance to the first query vector. Then, by weighting the weight of the second key vector and the second value vector, the first fusion feature is finally obtained, which can fully reflect the semantic information of the text and the visual content of the image.

[0041] Furthermore, in one embodiment, before the step of learning and processing the target fusion features through a preset multi-task learning module, the method further includes: Image features in the target fusion feature are decoded based on the image decoder to obtain new image features; The new image features and target text features are input into a preset cross-modal feature fusion model to obtain new target fusion features, and the step of learning and processing the target fusion features through a preset multi-task learning module is continued based on the new target fusion features.

[0042] As an example, in the embodiments of this application, the image decoder transforms the target fusion features into a more discriminative and structured image representation (i.e., new image features) through deconvolution layers or upsampling techniques. This decoding process not only restores the spatial distribution of the image but also enhances the expressiveness of the fusion features, which helps to coordinate and fuse image features with text features, thereby improving the final model performance.

[0043] Specifically, the new image features and target text features are input into a pre-defined cross-modal feature fusion model. The model obtains new target fusion features by fusing the two. These new target fusion features can then be further processed by a multi-task learning module to enhance their representational capabilities and obtain more accurate drivable area detection results.

[0044] Furthermore, in one embodiment, the method further includes: When it is detected that the road scene is structured, the image data and the text data corresponding to the structured road scene are input into a preset multi-task network general model to obtain a detection result of the drivable area.

[0045] Exemplarily, in the embodiments of the present application, the structured road scene refers to elements and information in the road environment being sorted and marked, which can clearly present the structure of the scene, including road signs, lane lines, traffic signs, signal lights, obstacles, road surface states, etc.; the multi-task network general model is used for identifying the drivable area in the structured road scene, for example, in the embodiments of the present application, YOLOX (You Only Look Once eXtreme, target detection model) is preferably selected as the multi-task network general model, which transmits the image data and the text data corresponding to the structured road scene to a convolutional neural network, and performs a series of feature extraction, prediction and detection operations inside the convolutional neural network, and finally outputs the detection result of the drivable area; wherein the working principle and implementation process of the YOLOX model for identifying the drivable area are well known in the art, and for the sake of brevity of description, will not be described here.

[0046] Further, in an embodiment, the respective extraction of the target text feature and the target image feature in the unstructured road scene comprises: extracting the target text feature in the unstructured road scene from the text data corresponding to the unstructured road scene based on a text backbone network; extracting the target image feature in the unstructured road scene from the image data corresponding to the unstructured road scene based on an image encoder.

[0047] Exemplarily, in the embodiments of the present application, the text data (composed of multiple words, such as sentences or paragraphs) corresponding to the unstructured road scene can be obtained through a navigation system, and the text backbone network is composed of an embedding layer, an encoding layer and a feature extraction layer; specifically, each word in the text data is converted into a word vector of fixed dimension through the embedding layer, and finally a matrix composed of these word vectors, i.e. an embedding vector matrix, is output; the embedding vector matrix is subjected to context modeling through the encoding layer, and a vector matrix after context modeling is output; the vector matrix after context modeling is subjected to high-level semantic extraction and compression of the text through the feature extraction layer, and finally the target text feature is output.

[0048] It should be noted that the image data corresponding to the unstructured road scene can be obtained by a visual sensor, and the image encoder includes a convolutional layer, a pooling layer, a fully connected layer and a feature extraction layer. Specifically, the convolutional layer extracts low-level features in the image data through convolution operation and an activation function to obtain a feature map matrix, and then down-samples the feature map matrix through the pooling layer to obtain a pooled feature map matrix. The pooled feature map matrix is flattened and input to the fully connected layer to further extract high-level features, so as to obtain a vector (i.e., a one-dimensional vector) output by the fully connected layer. Then, the vector output by the fully connected layer is input to the feature extraction layer to generate high-level semantic features of the target image, i.e., target image features.

[0049] It can be understood that, as shown in Figure 3 , the text backbone network, the image encoder, the cross-modal feature fusion model and the multi-task learning module in the present application jointly constitute a visual language model, which is used for detecting the drivable area of the unstructured road scene. Specifically, after obtaining the text data corresponding to the unstructured road scene, the text data is input to the text backbone network for text feature extraction. At the same time, after obtaining the image data corresponding to the unstructured road scene, the image data is input to the image encoder for image feature extraction. Then, the results of text feature extraction and image feature extraction are input to the cross-modal feature fusion model to obtain target fusion features. The target fusion features are decoded by the image decoder to obtain new fusion features. The new fusion features and the results of text feature extraction are input to the cross-modal feature fusion model to obtain new target fusion features. The new target fusion features are processed by the multi-task learning module to obtain the drivable area and the obstacle target.

[0050] It should be noted that, in the unstructured road scene, the new target fusion features are processed by the multi-task learning module to obtain not only the drivable area but also the obstacle target. This way enables the model to improve the recognition accuracy while processing multiple tasks, thereby providing more comprehensive reference for driving decision-making. When combined with the drivable area detection, the obstacle target can also be used as a limiting condition to exclude unsafe areas in the drivable area, thereby ensuring that the drivable area output by the model is smooth and safe.

[0051] In a second aspect, the embodiments of the present application also provide a drivable area detection system.

[0052] In an embodiment, as shown in Figure 4 , Figure 4 is a schematic diagram of the functional modules of the drivable area detection system embodiment of the present application. As Figure 4 shown, the drivable area detection system includes: a first processing module configured to extract target text features and target image features in an unstructured road scene respectively when it is detected that the road scene is unstructured; a second processing module configured to perform cross-modal feature fusion processing on the target text features and the target image features by using a preset cross-modal feature fusion model to obtain target fusion features; a third processing module configured to perform learning processing on the target fusion features by using a preset multi-task learning module to obtain a detection result of a drivable area.

[0053] Further, in an embodiment, the cross-modal feature fusion model includes a self-attention mechanism, a cross-attention mechanism, a cross-fusion enhancement layer, and a feedforward neural network, and the second processing module is specifically configured to: the self-attention mechanism is configured to output a first target feature vector corresponding to the target text features and a second target feature vector corresponding to the target image features; the cross-attention mechanism is configured to perform fusion processing on the first target feature vector and the second target feature vector to obtain first fusion features and second fusion features respectively; the cross-fusion enhancement layer is configured to perform weighted processing on the first fusion features and the second fusion features to obtain third fusion features; the feedforward neural network is configured to perform feature mapping and non-linear conversion on the third fusion features to obtain the target fusion features.

[0054] Further, in an embodiment, the first target feature vector includes a first query vector, a first key vector, and a first value vector, the second target feature vector includes a second query vector, a second key vector, and a second value vector, and the second processing module is specifically further configured to: determine the first fusion features based on the first query vector, the second key vector, and the second value vector; determine the second fusion features based on the second query vector, the first key vector, and the first value vector.

[0055] Further, in an embodiment, the determination of the first fusion features based on the first query vector, the second key vector, and the second value vector includes: perform similarity calculation on the first query vector and the second key vector to obtain a weight corresponding to the second key vector; determine the first fusion features based on the weight and the second value vector.

[0056] Further, in an embodiment, the third processing module is specifically configured to: perform decoding processing on image features in the target fusion features by using an image decoder to obtain new image features; The new image feature and the target text feature are input into a preset cross-modal feature fusion model to obtain a new target fusion feature, and the step of learning and processing the target fusion feature by using the preset multi-task learning module is continued.

[0057] Further, in an embodiment, the first processing module is specifically configured to: When it is detected that the road scene is structured, the image data and the text data corresponding to the structured road scene are input into a preset multi-task network universal model to obtain a detection result of the drivable area.

[0058] Further, in an embodiment, the first processing module is specifically configured to: extracting, based on a text backbone network, a target text feature in a non-structured road scene from text data corresponding to the non-structured road scene; extracting, based on an image encoder, a target image feature in the non-structured road scene from image data corresponding to the non-structured road scene.

[0059] When it is detected that the road scene is non-structured, the target text feature and the target image feature in the non-structured road scene are extracted respectively, the cross-modal feature fusion model is used to perform cross-modal feature fusion processing on the target text feature and the target image feature to obtain a target fusion feature, so that the system can use the relationship between different modal features to effectively infer and predict the scene without the data set covering all obstacle scenes, and the multi-task learning module is used to learn and process the target fusion feature to obtain a detection result of the drivable area, so that the detection of the drivable area in a scene that does not appear in the data set is realized, and the accuracy of the detection of the drivable area in a complex environment is improved.

[0060] The functions of each module in the above drivable area detection system correspond to each step in the above drivable area detection method embodiment, and the functions and implementation processes are not repeated here.

[0061] In a third aspect, the embodiments of the present application provide a drivable area detection device. The drivable area detection device can be a personal computer (PC), a notebook computer, a server, or other devices with data processing functions.

[0062] Reference Figure 5 , Figure 5 FIG. 1 is a schematic diagram of a hardware structure of a drivable area detection device according to an embodiment of the present application. The drivable area detection device can include a processor, a memory, a communication interface, and a communication bus.

[0063] The communication bus can be any type of bus, for example, a system bus, a PCI bus, HyperTransport, InfiniBand, Industry Standard Architecture (ISA), Ethernet, or any other bus that allows processor-based devices to communicate with one another.

[0064] The communication interface includes an input / output (I / O) interface, a physical interface, and a logical interface, and the like, which are used to implement the interconnection of devices inside the drivable area detection device, and the interconnection of the drivable area detection device and other devices (for example, other computing devices or user devices). The physical interface can be an Ethernet interface, a fiber interface, an ATM interface, and the like; the user device can be a display (Display), a keyboard (Keyboard), and the like.

[0065] The memory can be various types of storage media, such as random access memory (RAM), read-only memory (ROM), non-volatile RAM (NVRAM), flash memory, optical storage, hard disk, programmable ROM (PROM), erasable PROM (EPROM), electrically erasable PROM (EEPROM), and the like.

[0066] The processor can be a general-purpose processor, which can call the drivable area detection program stored in the memory and execute the drivable area detection method provided by the embodiments of the present application. For example, the general-purpose processor can be a central processing unit (CPU). The method executed when the drivable area detection program is called can refer to each embodiment of the drivable area detection method of the present application, which will not be described here.

[0067] Those skilled in the art can understand that the hardware structure shown in the above-mentioned embodiments is not a limitation of the present application, and can include more or fewer components than those shown, or combine certain components, or different component arrangements. Figure 5

[0068] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium.

[0069] The computer readable storage medium of the present application stores a drivable area detection program, wherein when the drivable area detection program is executed by the processor, the steps of the drivable area detection method as described above are implemented.

[0070] ​The method implemented when the drivable area detection program is executed can refer to each embodiment of the drivable area detection method of the present application, which will not be repeated here.

[0071] It should be noted that the above-mentioned sequence numbers of the embodiments of the present application are only for description, and do not represent the advantages and disadvantages of the embodiments.

[0072] The terms "comprise" and "have" and any variations thereof in the specification and claims of the present application and the above drawings are intended to cover non-exclusive inclusion. For example, a process, method, system, product or device including a series of steps or units is not limited to the listed steps or units, but can optionally include steps or units not listed, or can optionally include other steps or units inherent to the process, method, product or device. The terms "first", "second" and "third" and the like descriptions are used to distinguish different objects, and do not represent the order or limit the types of "first", "second" and "third".

[0073] In the description of the embodiments of the present application, "exemplary", "for example" or "for instance" is used to represent as an example, illustration or description. Any embodiment or design scheme described as "exemplary", "for example" or "for instance" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. In fact, the words "exemplary", "for example" or "for instance" are intended to present the relevant concept in a specific manner.

[0074] In the description of the embodiments of the present application, unless otherwise specified, " / " represents or, for example, A / B can represent A or B; "and / or" in the text only represents a description of the relationship between the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent three cases of A alone, A and B together, and B alone. In addition, in the description of the embodiments of the present application, "multiple" means two or more than two.

[0075] In some of the processes described in the embodiments of the present application, a plurality of operations or steps are included in a specific order, but it should be understood that these operations or steps can be executed or executed in parallel without the order in which they appear in the embodiments of the present application. The serial number of the operation is only used to distinguish different operations, and the serial number itself does not represent any execution order. In addition, these processes can include more or fewer operations, and these operations or steps can be executed in sequence or in parallel, and these operations or steps can be combined.

[0076] Those skilled in the art can clearly understand the above-mentioned embodiment method can be realized by means of software and the necessary general hardware platform, of course, can also be realized by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) as described above, and includes a plurality of instructions for making a terminal device execute the method described in each embodiment of the present application.

[0077] The above is only the preferred embodiment of the present application, and does not limit the patent scope of the present application, and any equivalent structure or equivalent process transformation using the content of the specification and drawings, or direct or indirect application in other related technical fields, are also included in the patent protection scope of the present application.

Claims

1. A travelable area detection method characterized by comprising: The drivable area detection method comprises the following steps: When it is detected that the road scene is unstructured, target text features and target image features under the unstructured road scene are extracted respectively; The target text features and the target image features are subjected to cross-modal feature fusion processing through a preset cross-modal feature fusion model to obtain target fusion features; The target fusion features are subjected to learning processing through a preset multi-task learning module to obtain a detection result of the drivable area.

2. The travelable area detecting method according to claim 1, characterized by: The cross-modal feature fusion model comprises a self-attention mechanism, a cross-attention mechanism, a cross-fusion enhancement layer and a feedforward neural network; The self-attention mechanism is used for outputting a first target feature vector corresponding to the target text features and a second target feature vector corresponding to the target image features; The cross-attention mechanism is used for fusion processing of the first target feature vector and the second target feature vector to obtain first fusion features and second fusion features respectively; The cross-fusion enhancement layer is used for weighted processing of the first fusion features and the second fusion features to obtain third fusion features; The feedforward neural network is used for feature mapping and non-linear conversion of the third fusion features to obtain the target fusion features.

3. The travelable area detecting method according to claim 2, characterized by, The first target feature vector comprises a first query vector, a first key vector and a first value vector, the second target feature vector comprises a second query vector, a second key vector and a second value vector, and the fusion processing of the first target feature vector and the second target feature vector to obtain the first fusion features and the second fusion features respectively comprises: The first fusion features are determined based on the first query vector, the second key vector and the second value vector; The second fusion features are determined based on the second query vector, the first key vector and the first value vector.

4. The travelable area detecting method according to claim 3, characterized by, The determination of the first fusion features based on the first query vector, the second key vector and the second value vector comprises: Similarity calculation is performed on the first query vector and the second key vector to obtain a weight corresponding to the second key vector; The first fusion features are determined based on the weight and the second value vector.

5. The travelable area detecting method according to claim 1, characterized by, Before the step of learning processing of the target fusion features through the preset multi-task learning module, the method further comprises: Image features in the target fusion features are subjected to decoding processing based on an image decoder to obtain new image features; The new image features and the target text features are input into the preset cross-modal feature fusion model to obtain new target fusion features, and the step of learning processing of the target fusion features through the preset multi-task learning module is continued based on the new target fusion features.

6. The travelable area detecting method according to claim 1, characterized by, The method further comprises: When it is detected that the road scene is structured, image data and text data corresponding to the structured road scene are input into a preset multi-task network universal model to obtain a detection result of the drivable area.

7. The travelable area detecting method according to claim 1, characterized by, The target text features and the target image features under the unstructured road scene are extracted respectively, comprising: Target text features under the unstructured road scene are extracted from text data corresponding to the unstructured road scene based on a text backbone network; Target image features under the unstructured road scene are extracted from image data corresponding to the unstructured road scene based on an image encoder.

8. A travelable area detection system characterized by comprising: The drivable area detection system comprises: The first processing module is configured to extract target text features and target image features in an unstructured road scene respectively when it is detected that the road scene is unstructured; The second processing module is configured to perform cross-modal feature fusion processing on the target text features and the target image features through a preset cross-modal feature fusion model to obtain target fusion features; The third processing module is configured to perform learning processing on the target fusion features through a preset multi-task learning module to obtain a detection result of the drivable area.

9. A travelable area detecting apparatus characterized by comprising: The drivable area detection device comprises a processor, a memory, and a drivable area detection program stored in the memory and executable by the processor, wherein when the drivable area detection program is executed by the processor, the steps of the drivable area detection method according to any one of claims 1 to 7 are implemented.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a drivable area detection program, wherein when the drivable area detection program is executed by the processor, the steps of the drivable area detection method according to any one of claims 1 to 7 are implemented.

Images