Feasible region prediction method, device, system and storage medium

Abstract

The application provides a feasible domain prediction method, device, system and storage medium, which is applied to vehicle automatic driving or auxiliary driving, and the method comprises the following steps: acquiring a surround view image at a current time, and obtaining bird's eye view features according to the surround view image; the surround view image comprises images of multiple perspectives collected by multiple cameras on the vehicle; extracting the bird's eye view features to obtain bird's eye view high-dimensional image features at the current time; generating a future feasible domain prediction map according to a time sequence queue formed by the bird's eye view high-dimensional image features at the current time and bird's eye view high-dimensional image features at multiple historical times, and outputting the feasible domain prediction map. The application realizes analysis and prediction of a future scene, can provide a basis for behavior decision of vehicle automatic driving or auxiliary driving, avoids repeated calculation and information accumulation error caused by dividing the feasible domain segmentation and obstacle prediction into two modules by combining scene perception and behavior prediction, and does not need to independently perform behavior prediction.

Description

Technical Field

[0001] This application relates to the field of assisted / autonomous driving technology, and more specifically to a feasible domain prediction method, apparatus, system, and storage medium. Background Technology

[0002] In applications such as assisted driving and autonomous driving, the segmentation of drivable areas is a crucial technical step. This segmentation is generally based on information fed back from sensors such as cameras or LiDAR. Cameras primarily provide image information, which has the advantages of long viewing distance, high resolution, and clear representation of scene color and texture features. However, it suffers from a lack of distance information and difficulties in converting image coordinates to the world coordinate system. LiDAR primarily provides point cloud information, which has the advantages of accurate distance measurement and no need for coordinate system transformation. However, it suffers from lower resolution and a lack of color and texture information. Currently, the problem of drivable region extraction is generally considered a static scene segmentation problem.

[0003] Cameras have a price advantage over LiDAR. Taking a target detection algorithm based on bird's-eye-view features from surround-view cameras as an example, a Convolutional Neural Network (CNN) encoder encodes the input image of each camera. A Transformer or similar model transforms the encoded features from the image coordinate system to the vehicle coordinate system, forming a Bird's-Eye-View Feature (BEVFeature). Targets are then detected from the BEVFeature, outputting the detection results from a bird's-eye-view perspective.

[0004] However, the aforementioned technologies focus on describing the current environment through BEVFeatures. Both object detection and feasible region segmentation characterize the current environment but lack analysis of future scenarios.

[0005] In view of the above problems, this application proposes a new feasible domain prediction method, apparatus, system and storage medium to at least partially solve the above problems. Summary of the Invention

[0006] The summary section introduces a series of simplified concepts, which will be further explained in detail in the detailed description section. The summary section of this invention is not intended to limit the key features and essential technical features of the claimed technical solution, nor is it intended to determine the scope of protection of the claimed technical solution.

[0007] This application provides a feasible region prediction method, which is applied to autonomous driving or assisted driving of a vehicle. The method includes: acquiring a surround view image at the current moment and obtaining bird's-eye view features based on the surround view image; the surround view image includes images from multiple perspectives captured by multiple cameras on the vehicle; extracting the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment; generating a future feasible region prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and high-dimensional image features of the bird's-eye view at multiple historical moments, and outputting the feasible region prediction map.

[0008] In one example, obtaining the bird's-eye view features based on the panoramic images includes: extracting features from the images from the multiple perspectives to obtain high-dimensional image features from the multiple perspectives; and fusing the high-dimensional image features from the multiple perspectives to obtain the bird's-eye view features.

[0009] In one example, fusing the high-dimensional image features of the multiple viewpoint images to obtain the bird's-eye view features includes: transforming the high-dimensional image features of the multiple viewpoint images from the coordinate system of their respective viewpoints to the vehicle coordinate system, and obtaining the bird's-eye view features based on the result of the transformation.

[0010] In one example, generating a future feasible region prediction map based on a temporal queue composed of high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments includes: generating high-dimensional bird's-eye view image features at multiple future moments based on the temporal queue; and upsampling the high-dimensional bird's-eye view image features at each of the multiple future moments to generate a feasible region prediction map at each moment, thereby obtaining the feasible region prediction map at the multiple future moments.

[0011] In one example, the method is implemented by a trained neural network comprising a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network, wherein: the first sub-network is used to acquire images from the multiple viewpoints at the current time and extract features from the multiple viewpoints to obtain high-dimensional image features from the multiple viewpoints; the second sub-network is used to fuse the high-dimensional image features from the multiple viewpoints to obtain the bird's-eye view features; the third sub-network is used to extract the bird's-eye view features to obtain high-dimensional bird's-eye view features at the current time; and the fourth sub-network is used to generate a future feasible domain prediction map based on a temporal queue composed of the high-dimensional bird's-eye view features at the current time and high-dimensional bird's-eye view features from multiple historical times.

[0012] In one example, fusing the high-dimensional image features of the multiple viewpoint images to obtain the bird's-eye view features includes: using the high-dimensional image features of the viewpoint images as keys and the pixel position coordinates in the bird's-eye view features as query inputs to the second sub-network, and obtaining the bird's-eye view features based on the output of the second sub-network.

[0013] In one example, the third sub-network includes multiple convolutional layers. The step of extracting the bird's-eye view features to obtain high-dimensional bird's-eye view image features includes: extracting features from the bird's-eye view features through one of the convolutional layers, and then extracting features again from the results extracted by the previous convolutional layer through the next convolutional layer to obtain the high-dimensional bird's-eye view image features.

[0014] In one example, the fourth sub-network includes a number of sub-networks equal to the number of high-dimensional bird's-eye view image features in the temporal queue. The step of generating a future feasible region prediction map based on the temporal queue composed of the high-dimensional bird's-eye view image features at the current time and high-dimensional bird's-eye view image features at multiple historical times includes: generating high-dimensional bird's-eye view image features for the corresponding future time based on the temporal queue using the corresponding sub-networks; and performing upsampling processing on the high-dimensional bird's-eye view image features for the corresponding future time to generate a feasible region prediction map for the corresponding future time, thereby obtaining feasible region prediction maps for multiple future times.

[0015] In one example, the method is implemented by multiple trained neural networks, including a first neural network, a second neural network, a third neural network, and a fourth neural network, wherein: the first neural network is used to acquire images from the multiple perspectives at the current time and extract features from the images from the multiple perspectives to obtain high-dimensional image features of the multiple perspective images; the second neural network is used to fuse the high-dimensional image features of the multiple perspective images to obtain the bird's-eye view features; the third neural network is used to extract the bird's-eye view features to obtain the high-dimensional image features of the bird's-eye view at the current time; and the fourth neural network is used to generate a future feasible domain prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current time and the high-dimensional image features of the bird's-eye view at multiple historical times.

[0016] In one example, fusing the high-dimensional image features of the multiple viewpoint images to obtain the bird's-eye view features includes: using the high-dimensional image features of the viewpoint images as keys and the pixel position coordinates in the bird's-eye view features as query inputs to the second neural network, and obtaining the bird's-eye view features based on the output of the second neural network.

[0017] In one example, the third neural network includes multiple convolutional layers. The step of extracting the bird's-eye view features to obtain high-dimensional bird's-eye view image features includes: extracting features from the bird's-eye view features through one of the convolutional layers, and then extracting features again from the results extracted by the previous convolutional layer through the next convolutional layer to obtain the high-dimensional bird's-eye view image features.

[0018] In one example, the fourth neural network includes a number of subnetworks equal to the number of high-dimensional bird's-eye view image features in the temporal queue. The step of generating a future feasible region prediction map based on the temporal queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments includes: generating high-dimensional bird's-eye view image features for corresponding future moments based on the corresponding subnetworks and the temporal queue; and upsampling the high-dimensional bird's-eye view image features for each corresponding future moment to generate a future feasible region prediction map for that corresponding moment, thereby obtaining feasible region prediction maps for multiple future moments.

[0019] In one example, the feasible region prediction map is a probability map presented in a probabilistic manner, which represents the probability that a pixel in the panoramic image at a future time belongs to a drivable region.

[0020] In one example, when the pixel value of a pixel in the probability map is not greater than a set threshold, the pixel in the future panoramic view corresponding to that pixel does not belong to the drivable area; when the pixel value of a pixel in the probability map is greater than the set threshold, the pixel in the future panoramic view corresponding to that pixel belongs to the drivable area.

[0021] This application also provides a feasible region prediction device for use in autonomous or assisted driving of vehicles, comprising: a bird's-eye view feature module for acquiring a surround view image at the current moment and obtaining bird's-eye view features based on the surround view image; the surround view image includes images from multiple perspectives captured by multiple cameras on the vehicle; a high-dimensional bird's-eye view image feature module for extracting the bird's-eye view features to obtain high-dimensional bird's-eye view image features at the current moment; and a feasible region prediction map module for generating a future feasible region prediction map based on a temporal queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features from multiple historical moments, and outputting the feasible region prediction map.

[0022] This application also provides a feasible region prediction device for use in autonomous or assisted driving of vehicles, comprising: multiple cameras mounted on the vehicle for acquiring surround view images at the current moment; the surround view images include images from multiple perspectives; one or more processors for: acquiring the surround view images and obtaining bird's-eye view features based on the surround view images; extracting the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment; generating a future feasible region prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and high-dimensional image features of the bird's-eye view at multiple historical moments, and outputting the feasible region prediction map.

[0023] This application also provides a feasible region prediction apparatus, including a memory and a processor, wherein the memory stores a computer program executed by the processor, and the computer program, when executed by the processor, causes the processor to perform the feasible region prediction method described in any one of the above.

[0024] In another aspect, this application provides a system for autonomous or assisted driving of a vehicle, the system including the feasible domain prediction device described in any one of the foregoing.

[0025] In another aspect, this application provides a computer-readable storage medium storing a computer program that, when run by a processor, causes the processor to perform the feasible region prediction method described in any one of the preceding claims.

[0026] The feasible domain prediction method, apparatus, system, and storage medium according to embodiments of this application obtain a future feasible domain prediction map by using the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments. This enables the analysis and prediction of future scenarios, thereby providing a basis for the behavioral decisions of autonomous or assisted driving vehicles. Furthermore, the feasible domain prediction map is generated based on the acquired surround view image at the current moment. By combining scene perception and behavior prediction, the feasible domain prediction map can directly provide the future movement trajectory of obstacles, thereby dividing drivable and non-drivable areas. This avoids the redundant calculations and information accumulation errors caused by dividing feasible domain segmentation and obstacle prediction into two modules in related technologies, eliminating the need for independent behavior prediction. Attached Figure Description

[0027] To more clearly illustrate the technical solutions in the embodiments of this application, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] In the attached diagram:

[0029] Figure 1 A schematic block diagram of an electronic device according to an embodiment of this application is shown;

[0030] Figure 2 A schematic flowchart of a feasible region prediction method according to an embodiment of this application is shown;

[0031] Figure 3 A flowchart illustrating a process for obtaining bird's-eye view features from a surround view image according to an embodiment of this application is shown.

[0032] Figure 4 This diagram illustrates the transformation of high-dimensional image features from multiple viewpoints to the vehicle coordinate system according to an embodiment of the present application;

[0033] Figure 5 A flowchart illustrating the extraction of bird's-eye view features to obtain high-dimensional image features of a bird's-eye view according to an embodiment of this application is shown.

[0034] Figure 6 A flowchart illustrating the generation of a future feasible domain prediction graph based on a temporal queue according to an embodiment of this application is shown.

[0035] Figure 7 A schematic block diagram of a feasible domain prediction apparatus according to an embodiment of this application is shown;

[0036] Figure 8 A schematic block diagram of another feasible domain prediction device according to an embodiment of this application is shown. Detailed Implementation

[0037] To make the objectives, technical solutions, and advantages of this application more apparent, exemplary embodiments according to this application will be described in detail below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of this application, and not all embodiments of this application. It should be understood that this application is not limited to the exemplary embodiments described herein. Based on the embodiments of this application described herein, all other embodiments obtained by those skilled in the art without inventive effort should fall within the protection scope of this application.

[0038] The following description provides numerous specific details to offer a more thorough understanding of this application. However, it will be apparent to those skilled in the art that this application can be practiced without one or more of these details. In other instances, certain technical features well-known in the art have not been described to avoid confusion with this application.

[0039] It should be understood that this application can be implemented in various forms and should not be construed as being limited to the embodiments set forth herein. Rather, providing these embodiments will make the disclosure thorough and complete, and will fully convey the scope of this application to those skilled in the art.

[0040] The terminology used herein is for the purpose of describing particular embodiments only and is not intended to limit the scope of this application. When used herein, the singular forms “a,” “an,” and “the” are also intended to include the plural forms unless the context clearly indicates otherwise. It should also be understood that the terms “comprising” and / or “including,” when used in this specification, identify the presence of the stated features, integers, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components, and / or groups. When used herein, the term “and / or” includes any and all combinations of the associated listed items.

[0041] To fully understand this application, a detailed structure will be presented in the following description to illustrate the technical solution proposed in this application. Optional embodiments of this application are described in detail below; however, in addition to these detailed descriptions, this application may have other implementation methods.

[0042] First, refer to Figure 1 This describes an example electronic device 100 for implementing the feasible domain prediction method and apparatus of embodiments of the present invention.

[0043] like Figure 1 As shown, the electronic device 100 includes one or more processors 102, one or more memories 104, input devices 106, and output devices 108, which are interconnected via a bus system 110 and / or other forms of connection mechanisms (not shown). It should be noted that... Figure 1 The components and structure of the electronic device 100 shown are merely exemplary and not limiting; the electronic device may also have other components and structures as needed.

[0044] The processor 102 may be a central processing unit (CPU) or other form of processing unit with data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device 100 to perform desired functions.

[0045] The memory 104 may include one or more computer program products, which may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 102 may execute the program instructions to implement the client functions (implemented by the processor) in the embodiments of the present invention described below, and / or other desired functions. Various applications and various data may also be stored in the computer-readable storage medium, such as various data used and / or generated by the applications.

[0046] The input device 106 may be a device used by a user to input commands, and may include one or more of the following: keyboard, mouse, microphone, and touch screen.

[0047] The output device 108 can output various information (such as images or sounds) to the outside (e.g., a user) and may include one or more of a display, a speaker, etc.

[0048] For example, an example electronic device for implementing the feasible domain prediction method and apparatus according to embodiments of the present invention can be implemented as a terminal such as a smartphone or tablet computer.

[0049] Below, we will refer to Figure 2 A feasible region prediction method according to an embodiment of the present invention is described. Figure 2 This is a schematic flowchart of a feasible region prediction method 200 according to an embodiment of this application. The feasible region prediction method of this application is used in a feasible region prediction device, which includes a processor, memory, input device, and output device, etc., and can be implemented as the electronic device 100 described above. Specifically, the feasible region prediction method 200 of this application can be applied to autonomous driving or assisted driving of vehicles, and includes the following steps:

[0050] In step S210, the current surround view image is acquired, and the bird's-eye view features are obtained based on the surround view image; the surround view image includes images from multiple perspectives captured by multiple cameras on the vehicle;

[0051] In step S220, the bird's-eye view features are extracted to obtain the high-dimensional image features of the bird's-eye view at the current moment;

[0052] In step S230, a future feasible region prediction map is generated based on the temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments, and the feasible region prediction map is output.

[0053] According to the feasible region prediction method 200 of the present invention, a future feasible region prediction map is obtained by using the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments. This enables the analysis and prediction of future scenarios, thereby providing a basis for the behavioral decisions of autonomous or assisted driving vehicles. Moreover, the feasible region prediction map is generated based on the acquired surround view image at the current moment. By combining scene perception and behavior prediction, the feasible region prediction map can directly provide the future trajectory of obstacles, thereby dividing drivable and non-drivable areas. This avoids the redundant calculations and information accumulation errors caused by dividing feasible region segmentation and obstacle prediction into two modules in related technologies, and eliminates the need for independent behavior prediction.

[0054] In an embodiment of the present invention, when acquiring the surround view image at the current moment in step S210, multiple cameras installed on the vehicle can be exposed simultaneously to collect images from multiple perspectives, and the images from multiple perspectives together constitute the surround view image.

[0055] Then, image processing techniques are used to process the panoramic images to obtain bird's-eye view features. In one example, features can first be extracted from images acquired from multiple viewpoints to obtain high-dimensional image features from multiple viewpoints. Here, high-dimensional image features refer to the multi-dimensional image features obtained after extracting from the viewpoint images, typically with more than three dimensions. For example, the viewpoint images can be processed using ResNet50 (Residual Network 50), and the output result is the high-dimensional image features of the viewpoint images. Then, the high-dimensional image features of multiple viewpoint images can be fused to obtain the bird's-eye view features. Of course, in addition to the above method, other image processing methods can also be used to obtain bird's-eye view features, and this is not limited. For example, see attached... Figure 3 As shown, feature extraction can be performed on the image captured by camera 1 by feature extractor 1, feature extraction can be performed on the image captured by camera 2 by feature extractor 2, feature extraction can be performed on the image captured by camera 3 by feature extractor 3, and so on, to obtain high-dimensional image features of n viewpoints. Then, the high-dimensional image features of n viewpoints are fused by a converter network to obtain a bird's-eye view feature with feature dimensions of batchsize*H*W*C (where batchsize represents the batch size, H represents the height of the feature, W represents the width of the feature, and C represents the channels of the feature).

[0056] It is worth noting that the process of fusing high-dimensional image features from multiple viewpoints to obtain the bird's-eye view features essentially involves coordinate transformation. This transformation converts the high-dimensional image features from their respective viewpoint coordinate systems to the vehicle coordinate system, thus obtaining the bird's-eye view features based on the transformation result. The resulting bird's-eye view features can be viewed as a bird's-eye view image in the vehicle coordinate system. For example, as shown in the attached figure... Figure 4 As shown, on the right Figure 4 (a) Figure 4 (b) Figure 4 (c) Figure 4 (d) Figure 4 (e) and Figure 4 (f) shows images captured simultaneously by cameras from different perspectives. Each image is in its own image coordinate system, with the left side showing... Figure 4 (g) is an example image of a BEVFeature, which integrates image information from all views on the right and projects it onto the vehicle coordinate system.

[0057] In the embodiments of the present invention, the process of obtaining high-dimensional image features of the bird's-eye view from the bird's-eye view features in step S220 can refer to the process of obtaining high-dimensional image features of the view image from the view image in step S210, or it can be implemented by other image processing methods.

[0058] In an embodiment of the present invention, step S230 involves temporally encoding the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments to obtain a temporal queue, and then generating a future feasible region prediction map based on the temporal queue. For example, taking F... t As a high-dimensional image feature of the bird's-eye view at the current moment, F t-n F t-(n-1) ...F t-1 As high-dimensional image features of aerial views from multiple historical moments, the generated time-series queue can be expressed as {F}. t-n F t-(n-1) ...F t}

[0059] In another example, generating a future feasible region prediction map based on a temporal queue composed of high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments may include: generating high-dimensional bird's-eye view image features for multiple future moments based on the temporal queue; upsampling the high-dimensional bird's-eye view image features at each of the multiple future moments to generate a feasible region prediction map for each moment, thereby obtaining the feasible region prediction maps for the multiple future moments. In this example, the number of feasible region prediction maps for future moments is the same as the number of current and historical moments, thereby predicting drivable and indestructible areas in the panoramic image over a future period.

[0060] In embodiments of the present invention, the aforementioned calculations of extracting features from viewpoint images to obtain high-dimensional image features, fusing high-dimensional image features from viewpoint images to obtain bird's-eye view features, extracting bird's-eye view features to obtain high-dimensional image features of the bird's-eye view, and generating a feasible region prediction map can be implemented by different parts of the same trained neural network. For example, a neural network includes a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network, wherein: for example, the first sub-network acquires images from multiple viewpoints at the current moment and extracts features from the images from multiple viewpoints to obtain high-dimensional image features from multiple viewpoints; the output of the first sub-network is input to the second sub-network, which fuses the high-dimensional image features from multiple viewpoints to obtain bird's-eye view features; the output of the second sub-network is input to the third sub-network, which extracts the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment; the output of the third sub-network is input to the fourth sub-network, which generates a future feasible region prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments. Neural networks can be convolutional neural networks, deep neural networks, etc. For example, they can be CNN-based networks such as MobileNet (mobile network) and ResNet (residual network), or Transformer-based networks such as VisionTransformer (visual transformer). There is no limitation on the specific type of neural network.

[0061] Furthermore, the second sub-network can fuse high-dimensional image features from multiple viewpoints to obtain bird's-eye view features in the following way: the high-dimensional image features from the viewpoints are used as the key, and the pixel position coordinates in the bird's-eye view features are used as the query input to the second sub-network, and the bird's-eye view features are obtained based on the output of the second sub-network.

[0062] Furthermore, the third sub-network may include multiple convolutional layers. The third sub-network can extract high-dimensional bird's-eye view features by using one convolutional layer to extract features from the bird's-eye view, and then using subsequent convolutional layers to extract features again from the results of the previous convolutional layer, thus obtaining the high-dimensional bird's-eye view features. For example, see attached... Figure 5 As shown, the bird's-eye view features are first input into the first convolutional layer (CONV1) for feature extraction. Then, the results extracted by the first convolutional layer (CONV1) are input into the next convolutional layer, and so on, until the nth convolutional layer (CONV1) is reached. n After n feature extractions, the high-dimensional image features F of the bird's-eye view are obtained.

[0063] Furthermore, the fourth sub-network may include the same number of sub-networks as the number of high-dimensional bird's-eye view image features in the temporal queue. The fourth sub-network can generate future feasible region prediction maps based on the temporal queue composed of the current high-dimensional bird's-eye view image features and high-dimensional bird's-eye view image features from multiple historical times, as follows: First, it generates high-dimensional bird's-eye view image features for the corresponding future times based on the temporal queue using the corresponding sub-networks. Second, it upsamples the high-dimensional bird's-eye view image features for the corresponding future times to generate future feasible region prediction maps for those times, thus obtaining feasible region prediction maps for multiple future times. For example, see attached... Figure 6 As shown, taking a sub-network consisting of multiple Transformers in the fourth sub-network as an example, sub-network Transformer1 is based on F t-n F t-(…) F t The time-series queue generates a high-dimensional image feature F of the bird's-eye view at the future time t+1. t+1 By analyzing F t+1 Upsampling is performed to obtain the feasible region prediction map at the future time t+1; other sub-networks implement a similar process, and finally the feasible region prediction maps at multiple future times such as t+1, t+(...), t+q can be obtained.

[0064] In another embodiment of the present invention, the aforementioned calculations of extracting features from viewpoint images to obtain high-dimensional image features, fusing high-dimensional image features from viewpoint images to obtain bird's-eye view features, extracting bird's-eye view features to obtain high-dimensional image features of the bird's-eye view, and generating a feasible region prediction map can also be implemented by multiple different neural networks. For example, the multiple neural networks include a first neural network, a second neural network, a third neural network, and a fourth neural network, wherein: for example, the first neural network acquires images from multiple viewpoints at the current moment and extracts features from the images from multiple viewpoints to obtain high-dimensional image features from multiple viewpoints; the output of the first neural network is input to the second neural network, which fuses the high-dimensional image features from multiple viewpoints to obtain bird's-eye view features; the output of the second neural network is input to the third neural network, which extracts the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment; the output of the third neural network is input to the fourth neural network, which generates a future feasible region prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments. Among them, the neural network can be a convolutional neural network, a deep neural network, etc. For example, the first neural network can be a CNN-based network such as MobileNet or ResNet, or a Transformer-based network such as VisionTransformer. The second neural network can be a Transformer network, the third neural network can be a convolutional neural network, a Transformer network, etc., and the fourth neural network can be a Transformer network. The specific type of neural network is not limited.

[0065] Furthermore, the second neural network can fuse high-dimensional image features from multiple viewpoints to obtain bird's-eye view features in the following manner: the high-dimensional image features from the viewpoints are used as the key, and the pixel position coordinates in the bird's-eye view features are used as the query input to the second neural network, and the bird's-eye view features are obtained based on the output of the second neural network.

[0066] Furthermore, the third neural network may include multiple convolutional layers. The third neural network can extract high-dimensional bird's-eye view features by using one convolutional layer to extract features from the bird's-eye view, and then using subsequent convolutional layers to extract features again from the results of the previous convolutional layer, thus obtaining the high-dimensional bird's-eye view features. For example, see attached... Figure 5 As shown, the bird's-eye view features are first input into the first convolutional layer for feature extraction, and then the result extracted by the first convolutional layer is input into the next convolutional layer, and so on, until the nth convolutional layer is used to achieve n feature extractions, thereby obtaining the high-dimensional image features F of the bird's-eye view.

[0067] Furthermore, the fourth neural network may include the same number of subnetworks as the number of high-dimensional bird's-eye view image features in the time-series queue. The fourth neural network can generate future feasible region prediction maps based on the time-series queue composed of the current time-series high-dimensional bird's-eye view image features and multiple historical time-series high-dimensional bird's-eye view image features: generating corresponding future time-series high-dimensional bird's-eye view image features based on the corresponding subnetworks; upsampling the corresponding future time-series high-dimensional bird's-eye view image features to generate future feasible region prediction maps for those times, thus obtaining feasible region prediction maps for multiple future times. For example, see attached... Figure 6 As shown, taking the fourth neural network, which includes multiple Transformer subnetworks, as an example, subnetwork Transformer1 is based on F... t-n F t-(…) F t The time-series queue generates a high-dimensional image feature F of the bird's-eye view at the future time t+1. t+1 By analyzing F t+1 Upsampling is performed to obtain the feasible region prediction map at the future time t+1; other sub-networks implement a similar process, and finally the feasible region prediction maps at multiple future times such as t+1, t+(...), t+q can be obtained.

[0068] In an embodiment of the present invention, the obtained feasible region prediction map can be a probability map presented in a probabilistic manner. The probability map can characterize the probability that a pixel in the panoramic image at a future time belongs to a drivable area, thereby probabilistically expressing the drivability of the road in the future time period.

[0069] Furthermore, the feasible region prediction map can be probabilistically represented as follows: when the pixel value of a pixel in the probabilistic map is not greater than a set threshold, the pixel in the future panoramic view corresponding to that pixel does not belong to the drivable region; when the pixel value of a pixel in the probabilistic map is greater than the set threshold, the pixel in the future panoramic view corresponding to that pixel belongs to the drivable region. For example, the pixel value of each pixel in the probabilistic map is a floating-point number, and the value of the floating-point number is in the interval [0, 1]. Assuming that the pixel value corresponding to the pixel with coordinates (x, y) is k, then if the pixel is in the non-drivable region at time Q in the future, then k→0 (k tends to 0); if the pixel is in the drivable region at time Q in the future, then k→1 (k tends to 1).

[0070] Based on the above description, the feasible region prediction method according to embodiments of the present invention obtains a future feasible region prediction map by using the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments. This enables the analysis and prediction of future scenarios, thereby providing a basis for the behavioral decisions of autonomous or assisted driving vehicles. Furthermore, the feasible region prediction map is generated based on the acquired surround view image at the current moment. By combining scene perception and behavior prediction, the feasible region prediction map can directly provide the future trajectory of obstacles, thereby dividing drivable and non-drivable areas. This avoids the redundant calculations and information accumulation errors caused by dividing feasible region segmentation and obstacle prediction into two modules in related technologies, eliminating the need for independent behavior prediction.

[0071] The feasible region prediction method according to embodiments of the present invention has been described above by way of example. Exemplarily, the feasible region prediction method according to embodiments of the present invention can be implemented in a device, apparatus, or system having a memory and a processor.

[0072] Furthermore, the feasible domain prediction method according to embodiments of the present invention can be easily deployed on local terminals such as smartphones and tablets. Alternatively, the feasible domain prediction method according to embodiments of the present invention can also be deployed on a server (or in the cloud). Alternatively, the feasible domain prediction method according to embodiments of the present invention can also be distributed and deployed on a server (or in the cloud) and at local terminals.

[0073] Figure 7 A schematic block diagram of a feasible region prediction apparatus according to an embodiment of the present invention is shown. Figure 7 As shown, the feasible region prediction device 700 according to an embodiment of the present invention can be applied to autonomous driving or assisted driving of vehicles, including a bird's-eye view feature module 710, a bird's-eye view high-dimensional image feature module 720, and a feasible region prediction map module 730. The bird's-eye view feature module 710 is used to acquire a surround view image at the current moment and obtain bird's-eye view features based on the surround view image; the surround view image includes images from multiple perspectives captured by multiple cameras on the vehicle; the bird's-eye view high-dimensional image feature module 720 is used to extract the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment; the feasible region prediction map module 730 is used to generate a future feasible region prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and high-dimensional image features of the bird's-eye view at multiple historical moments, and output the feasible region prediction map.

[0074] Among them, the bird's-eye view feature module 710, the bird's-eye view high-dimensional image feature module 720, and the feasible region prediction map module 730 can be derived from... Figure 1The processor 102 in the illustrated electronic device 100 executes program instructions stored in the memory 104 to implement and perform corresponding steps in the feasible region prediction method 200 according to an embodiment of the present invention. The following description focuses on the main functions of each module of the feasible region prediction device, omitting the details already described above.

[0075] In an embodiment of the present invention, when the bird's-eye view feature module 710 acquires the surround view image at the current moment, it can simultaneously expose multiple cameras installed on the vehicle to collect images from multiple perspectives, and the images from multiple perspectives together constitute the surround view image.

[0076] Then, image processing techniques are used to process the panoramic images to obtain bird's-eye view features. In one example, features can first be extracted from images acquired from multiple viewpoints to obtain high-dimensional image features from multiple viewpoints. Here, high-dimensional image features refer to the multi-dimensional image features obtained after extracting from the viewpoint images, typically with more than three dimensions. For example, the viewpoint images can be processed using ResNet50, and the output result is the high-dimensional image features of the viewpoint images. Then, the high-dimensional image features of multiple viewpoint images can be fused to obtain the bird's-eye view features. Of course, in addition to the above method, other image processing methods can also be used to obtain bird's-eye view features, and this is not limited. For example, see attached... Figure 3 As shown, feature extraction can be performed on the image captured by camera 1 by feature extractor 1, feature extraction can be performed on the image captured by camera 2 by feature extractor 2, feature extraction can be performed on the image captured by camera 3 by feature extractor 3, and so on, to obtain high-dimensional image features of n viewpoints. Then, the high-dimensional image features of n viewpoints are fused by the Transformer network to obtain the bird's-eye view features with feature dimensions of batchsize*H*W*C.

[0077] It is worth noting that the process of fusing high-dimensional image features from multiple viewpoints to obtain the bird's-eye view features essentially involves coordinate transformation. This transformation converts the high-dimensional image features from their respective viewpoint coordinate systems to the vehicle coordinate system, thus obtaining the bird's-eye view features based on the transformation result. The resulting bird's-eye view features can be viewed as a bird's-eye view image in the vehicle coordinate system. For example, as shown in the attached figure... Figure 4 As shown, on the right Figure 4 (a) Figure 4 (b) Figure 4 (c) Figure 4 (d) Figure 4 (e) and Figure 4 (f) shows images captured simultaneously by cameras from different perspectives. Each image is in its own image coordinate system, with the left side showing... Figure 4(g) is an example image of a BEVFeature, which integrates image information from all views on the right and projects it onto the vehicle coordinate system.

[0078] In the embodiments of the present invention, the process by which the high-dimensional image feature module 720 obtains high-dimensional image features from bird's-view features can refer to the process by which the bird's-view feature module 710 obtains high-dimensional image features from a viewpoint image, or it can be implemented using other image processing methods.

[0079] In an embodiment of the present invention, the feasible region prediction map module 730 obtains a temporal queue by temporally encoding the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments, and then generates a future feasible region prediction map based on the temporal queue. For example, taking F... t As a high-dimensional image feature of the bird's-eye view at the current moment, F t-n F t-(n-1) ...F t-1 As high-dimensional image features of aerial views from multiple historical moments, the generated time-series queue can be expressed as {F}. t-n F t-(n-1) ...F t}

[0080] In another example, generating a future feasible region prediction map based on a temporal queue composed of high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments may include: generating high-dimensional bird's-eye view image features for multiple future moments based on the temporal queue; upsampling the high-dimensional bird's-eye view image features at each of the multiple future moments to generate a feasible region prediction map for each moment, thereby obtaining the feasible region prediction maps for the multiple future moments. In this example, the number of feasible region prediction maps for future moments is the same as the number of current and historical moments, thereby predicting drivable and indestructible areas in the panoramic image over a future period.

[0081] In embodiments of the present invention, the bird's-eye view feature module 710, the high-dimensional bird's-eye view image feature module 720, and the feasible region prediction map module 730 can be implemented through different parts of the same trained neural network. For example, a neural network includes a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network, wherein: for example, the first sub-network acquires images from multiple perspectives at the current moment and extracts features from the images from multiple perspectives to obtain high-dimensional image features from multiple perspectives; the output of the first sub-network is input to the second sub-network, which fuses the high-dimensional image features from multiple perspectives to obtain bird's-eye view features; the output of the second sub-network is input to the third sub-network, which extracts the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment; the output of the third sub-network is input to the fourth sub-network, which generates a future feasible region prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments. Neural networks can be convolutional neural networks, deep neural networks, etc. For example, they can be CNN-based networks such as MobileNet and ResNet, or Transformer-based networks such as VisionTransformer. There is no limitation on the specific type of neural network.

[0082] Furthermore, the second sub-network can fuse high-dimensional image features from multiple viewpoints to obtain bird's-eye view features in the following way: the high-dimensional image features from the viewpoints are used as the key, and the pixel position coordinates in the bird's-eye view features are used as the query input to the second sub-network, and the bird's-eye view features are obtained based on the output of the second sub-network.

[0083] Furthermore, the third sub-network may include multiple convolutional layers. The third sub-network can extract high-dimensional bird's-eye view features by using one convolutional layer to extract features from the bird's-eye view, and then using subsequent convolutional layers to extract features again from the results of the previous convolutional layer, thus obtaining the high-dimensional bird's-eye view features. For example, see attached... Figure 5 As shown, the bird's-eye view features are first input into the first convolutional layer for feature extraction, and then the result extracted by the first convolutional layer is input into the next convolutional layer, and so on, until the nth convolutional layer is used to achieve n feature extractions, thereby obtaining the high-dimensional image features F of the bird's-eye view.

[0084] Furthermore, the fourth sub-network may include the same number of sub-networks as the number of high-dimensional bird's-eye view image features in the temporal queue. The fourth sub-network can generate future feasible region prediction maps based on the temporal queue composed of the current high-dimensional bird's-eye view image features and high-dimensional bird's-eye view image features from multiple historical times, as follows: First, it generates high-dimensional bird's-eye view image features for the corresponding future times based on the temporal queue using the corresponding sub-networks. Second, it upsamples the high-dimensional bird's-eye view image features for the corresponding future times to generate future feasible region prediction maps for those times, thus obtaining feasible region prediction maps for multiple future times. For example, see attached... Figure 6 As shown, taking a sub-network consisting of multiple Transformers in the fourth sub-network as an example, sub-network Transformer1 is based on F t-n F t-(…) F t The time-series queue generates a high-dimensional image feature F of the bird's-eye view at the future time t+1. t+1 By analyzing F t+1 Upsampling is performed to obtain the feasible region prediction map at the future time t+1; other sub-networks implement a similar process, and finally the feasible region prediction maps at multiple future times such as t+1, t+(...), t+q can be obtained.

[0085] In another embodiment of the present invention, the bird's-eye view feature module 710, the high-dimensional bird's-eye view image feature module 720, and the feasible region prediction map module 730 can also be implemented using multiple different neural networks. For example, the multiple neural networks include a first neural network, a second neural network, a third neural network, and a fourth neural network, wherein: for example, the first neural network acquires images from multiple perspectives at the current moment and extracts features from the images from multiple perspectives to obtain high-dimensional image features from multiple perspectives; the output of the first neural network is input to the second neural network, which fuses the high-dimensional image features from multiple perspectives to obtain bird's-eye view features; the output of the second neural network is input to the third neural network, which extracts the bird's-eye view features to obtain high-dimensional image features from the bird's-eye view at the current moment; the output of the third neural network is input to the fourth neural network, which generates a future feasible region prediction map based on a temporal queue composed of the high-dimensional image features from the bird's-eye view at the current moment and the high-dimensional image features from the bird's-eye view at multiple historical moments. Among them, the neural network can be a convolutional neural network, a deep neural network, etc. For example, the first neural network can be a CNN-based network such as MobileNet or ResNet, or a Transformer-based network such as VisionTransformer. The second neural network can be a Transformer network, the third neural network can be a convolutional neural network, a Transformer network, etc., and the fourth neural network can be a Transformer network. The specific type of neural network is not limited.

[0086] Furthermore, the second neural network can fuse high-dimensional image features from multiple viewpoints to obtain bird's-eye view features in the following manner: the high-dimensional image features from the viewpoints are used as the key, and the pixel position coordinates in the bird's-eye view features are used as the query input to the second neural network, and the bird's-eye view features are obtained based on the output of the second neural network.

[0087] Furthermore, the third neural network may include multiple convolutional layers. The third neural network can extract high-dimensional bird's-eye view features by using one convolutional layer to extract features from the bird's-eye view, and then using subsequent convolutional layers to extract features again from the results of the previous convolutional layer, thus obtaining the high-dimensional bird's-eye view features. For example, see attached... Figure 5 As shown, the bird's-eye view features are first input into the first convolutional layer for feature extraction, and then the result extracted by the first convolutional layer is input into the next convolutional layer, and so on, until the nth convolutional layer is used to achieve n feature extractions, thereby obtaining the high-dimensional image features F of the bird's-eye view.

[0088] Furthermore, the fourth neural network may include the same number of subnetworks as the number of high-dimensional bird's-eye view image features in the time-series queue. The fourth neural network can generate future feasible region prediction maps based on the time-series queue composed of the current time-series high-dimensional bird's-eye view image features and multiple historical time-series high-dimensional bird's-eye view image features: generating corresponding future time-series high-dimensional bird's-eye view image features based on the corresponding subnetworks; upsampling the corresponding future time-series high-dimensional bird's-eye view image features to generate future feasible region prediction maps for those times, thus obtaining feasible region prediction maps for multiple future times. For example, see attached... Figure 6 As shown, taking the fourth neural network, which includes multiple Transformer subnetworks, as an example, subnetwork Transformer1 is based on F... t-n F t-(…) F t The time-series queue generates a high-dimensional image feature F of the bird's-eye view at the future time t+1. t+1 By analyzing F t+1 Upsampling is performed to obtain the feasible region prediction map at the future time t+1; other sub-networks implement a similar process, and finally the feasible region prediction maps at multiple future times such as t+1, t+(...), t+q can be obtained.

[0089] In an embodiment of the present invention, the obtained feasible region prediction map can be a probability map presented in a probabilistic manner. The probability map can characterize the probability that a pixel in the panoramic image at a future time belongs to a drivable area, thereby probabilistically expressing the drivability of the road in the future time period.

[0090] Furthermore, the feasible region prediction map can be probabilistically represented as follows: when the pixel value of a pixel in the probabilistic map is not greater than a set threshold, the pixel in the future panoramic view corresponding to that pixel does not belong to the drivable region; when the pixel value of a pixel in the probabilistic map is greater than the set threshold, the pixel in the future panoramic view corresponding to that pixel belongs to the drivable region. For example, the pixel value of each pixel in the probabilistic map is a floating-point number, and the value of the floating-point number is in the interval [0, 1]. Assuming that the pixel value corresponding to the pixel with coordinates (x, y) is k, then if the pixel is in the non-drivable region at time Q in the future, then k→0 (k tends to 0); if the pixel is in the drivable region at time Q in the future, then k→1 (k tends to 1).

[0091] Based on the above description, the feasible region prediction device 700 according to an embodiment of the present invention obtains a future feasible region prediction map through the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments, thereby realizing the analysis and prediction of future scenarios and providing a basis for the behavioral decisions of autonomous driving or assisted driving of vehicles; and the feasible region prediction map is generated based on the acquired surround view image at the current moment. By combining scene perception and behavior prediction, the feasible region prediction map can directly give the future movement trajectory of obstacles, thereby dividing the drivable area and the non-drivable area, avoiding the repeated calculation and information accumulation error caused by dividing feasible region segmentation and obstacle prediction into two modules in related technologies, and eliminating the need to perform behavior prediction independently.

[0092] Furthermore, those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementations should not be considered beyond the scope of this invention.

[0093] According to an embodiment of the present invention, a feasible region prediction device is also provided. The device is applied to autonomous driving or assisted driving of a vehicle and includes: a plurality of cameras installed on the vehicle for acquiring surround view images at the current moment; the surround view images include images from multiple perspectives; one or more processors for: acquiring the surround view images and obtaining bird's-eye view features based on the surround view images; extracting the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment; generating a future feasible region prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and high-dimensional image features of the bird's-eye view at multiple historical moments, and outputting the feasible region prediction map.

[0094] In one example, the bird's-eye view features are obtained from the panoramic images, including: extracting features from images from multiple perspectives to obtain high-dimensional image features from multiple perspectives; and fusing the high-dimensional image features from multiple perspectives to obtain the bird's-eye view features.

[0095] In one embodiment, fusing high-dimensional image features from multiple viewpoints to obtain bird's-eye view features includes: transforming the high-dimensional image features from multiple viewpoints from the coordinate system of their respective viewpoints to the vehicle coordinate system, and obtaining bird's-eye view features based on the transformation result.

[0096] In one embodiment, generating a future feasible region prediction map based on a temporal queue consisting of high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments includes: generating high-dimensional bird's-eye view image features at multiple future moments based on the temporal queue; and upsampling the high-dimensional bird's-eye view image features at each moment in the high-dimensional bird's-eye view image features at the multiple future moments to generate a feasible region prediction map at each moment, thereby obtaining a feasible region prediction map at multiple future moments.

[0097] In one embodiment, the steps executed by the processor are implemented by a trained neural network, which includes a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network, wherein: the first sub-network is used to acquire images from multiple perspectives at the current time and extract features from the images from multiple perspectives to obtain high-dimensional image features from multiple perspectives; the second sub-network is used to fuse the high-dimensional image features from multiple perspectives to obtain bird's-eye view features; the third sub-network is used to extract bird's-eye view features to obtain high-dimensional bird's-eye view features at the current time; and the fourth sub-network is used to generate a future feasible domain prediction map based on a temporal queue composed of the high-dimensional bird's-eye view features at the current time and the high-dimensional bird's-eye view features from multiple historical time points.

[0098] In one embodiment, fusing high-dimensional image features from multiple viewpoints to obtain bird's-eye view features includes: using the high-dimensional image features of the viewpoints as keys and the pixel position coordinates in the bird's-eye view features as query inputs to a second sub-network, and obtaining the bird's-eye view features based on the output of the second sub-network.

[0099] In one embodiment, the third sub-network includes multiple convolutional layers to extract bird's-eye view features to obtain high-dimensional bird's-eye view image features. This includes: extracting bird's-eye view features through one of the convolutional layers, and then extracting features again from the results extracted by the previous convolutional layer through the next convolutional layer to obtain high-dimensional bird's-eye view image features.

[0100] In one embodiment, the fourth sub-network includes a number of sub-networks equal to the number of high-dimensional bird's-eye view image features in the temporal queue. It generates a future feasible region prediction map based on the temporal queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments. This includes: generating high-dimensional bird's-eye view image features for the corresponding future moments based on the temporal queue using the respective sub-networks; and upsampling the high-dimensional bird's-eye view image features for the corresponding future moments to generate a future feasible region prediction map for that moment, thus obtaining feasible region prediction maps for multiple future moments.

[0101] In one embodiment, the steps executed by the processor are implemented by multiple trained neural networks, including a first neural network, a second neural network, a third neural network, and a fourth neural network, wherein: the first neural network is used to acquire images from multiple perspectives at the current moment and extract features from the images from multiple perspectives to obtain high-dimensional image features of the multiple perspective images; the second neural network is used to fuse the high-dimensional image features of the multiple perspective images to obtain bird's-eye view features; the third neural network is used to extract bird's-eye view features to obtain high-dimensional bird's-eye view features at the current moment; and the fourth neural network is used to generate a future feasible domain prediction map based on a temporal queue composed of the high-dimensional bird's-eye view features at the current moment and the high-dimensional bird's-eye view features from multiple historical moments.

[0102] In one embodiment, fusing high-dimensional image features from multiple viewpoints to obtain bird's-eye view features includes: using the high-dimensional image features of the viewpoints as keys and the pixel position coordinates in the bird's-eye view features as query inputs to a second neural network, and obtaining the bird's-eye view features based on the output of the second neural network.

[0103] In one embodiment, the third neural network includes multiple convolutional layers to extract bird's-eye view features and obtain high-dimensional bird's-eye view image features. This includes: extracting bird's-eye view features through one of the convolutional layers, and then extracting features again from the results extracted by the previous convolutional layer through the next convolutional layer to obtain high-dimensional bird's-eye view image features.

[0104] In one embodiment, the fourth neural network includes a number of subnetworks equal to the number of high-dimensional bird's-eye view image features in the time-series queue. It generates a future feasible region prediction map based on the time-series queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments. This includes: generating high-dimensional bird's-eye view image features for the corresponding future moments based on the time-series queue using the respective subnetworks; and upsampling the high-dimensional bird's-eye view image features for the corresponding future moments to generate a future feasible region prediction map for that moment, thus obtaining feasible region prediction maps for multiple future moments.

[0105] In one embodiment, the feasible region prediction map is a probability map presented in a probabilistic manner, which represents the probability that a pixel in the panoramic image at a future time belongs to a drivable region.

[0106] In one embodiment, when the pixel value of a pixel in the probability map is not greater than a set threshold, the pixel in the future panoramic view corresponding to the pixel does not belong to the drivable area; when the pixel value of a pixel in the probability map is greater than the set threshold, the pixel in the future panoramic view corresponding to the pixel belongs to the drivable area.

[0107] Figure 8 A schematic block diagram of a feasible region prediction apparatus according to an embodiment of the present invention is shown. The feasible region prediction apparatus 800 includes a memory 810 and a processor 820.

[0108] The memory 810 stores computer programs for implementing corresponding steps in the feasible region prediction method according to embodiments of the present invention. The processor 820 is used to run the computer programs stored in the memory 810 to execute corresponding steps in the feasible region prediction method according to embodiments of the present invention, and to implement corresponding modules in the feasible region prediction apparatus according to embodiments of the present invention.

[0109] In one embodiment, when the computer program is run by the processor 820, the feasible region prediction device 800 performs the following steps: acquiring a surround view image at the current moment and obtaining bird's-eye view features based on the surround view image; the surround view image includes images from multiple perspectives captured by multiple cameras on the vehicle; extracting the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment; generating a future feasible region prediction map based on a time-series queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments, and outputting the feasible region prediction map.

[0110] In one embodiment, obtaining bird's-eye view features from panoramic images includes: extracting features from images from multiple perspectives to obtain high-dimensional image features from multiple perspectives; and fusing the high-dimensional image features from multiple perspectives to obtain bird's-eye view features.

[0111] In one embodiment, fusing high-dimensional image features from multiple viewpoints to obtain bird's-eye view features includes: transforming the high-dimensional image features from multiple viewpoints from the coordinate system of their respective viewpoints to the vehicle coordinate system, and obtaining bird's-eye view features based on the transformation result.

[0112] In one embodiment, generating a future feasible region prediction map based on a temporal queue consisting of high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments includes: generating high-dimensional bird's-eye view image features at multiple future moments based on the temporal queue; and upsampling the high-dimensional bird's-eye view image features at each moment in the high-dimensional bird's-eye view image features at the multiple future moments to generate a feasible region prediction map at each moment, thereby obtaining a feasible region prediction map at multiple future moments.

[0113] In one embodiment, the steps executed by the feasible region prediction device 800 when the computer program is run by the processor 820 are implemented by a trained neural network, which includes a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network, wherein: the first sub-network is used to acquire images from multiple perspectives at the current time and extract features from the images from multiple perspectives to obtain high-dimensional image features from multiple perspectives; the second sub-network is used to fuse the high-dimensional image features from multiple perspectives to obtain bird's-eye view features; the third sub-network is used to extract bird's-eye view features to obtain high-dimensional bird's-eye view features at the current time; and the fourth sub-network is used to generate a future feasible region prediction map based on a temporal queue composed of the high-dimensional bird's-eye view features at the current time and the high-dimensional bird's-eye view features from multiple historical times.

[0114] In one embodiment, fusing high-dimensional image features from multiple viewpoints to obtain bird's-eye view features includes: using the high-dimensional image features of the viewpoints as keys and the pixel position coordinates in the bird's-eye view features as query inputs to a second sub-network, and obtaining the bird's-eye view features based on the output of the second sub-network.

[0115] In one embodiment, the third sub-network includes multiple convolutional layers to extract bird's-eye view features to obtain high-dimensional bird's-eye view image features. This includes: extracting bird's-eye view features through one of the convolutional layers, and then extracting features again from the results extracted by the previous convolutional layer through the next convolutional layer to obtain high-dimensional bird's-eye view image features.

[0116] In one embodiment, the fourth sub-network includes a number of sub-networks equal to the number of high-dimensional bird's-eye view image features in the temporal queue. It generates a future feasible region prediction map based on the temporal queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments. This includes: generating high-dimensional bird's-eye view image features for the corresponding future moments based on the temporal queue using the respective sub-networks; and upsampling the high-dimensional bird's-eye view image features for the corresponding future moments to generate a future feasible region prediction map for that moment, thus obtaining feasible region prediction maps for multiple future moments.

[0117] In one embodiment, the steps executed by the feasible region prediction device 800 when the computer program is run by the processor 820 are implemented by a plurality of trained neural networks, including a first neural network, a second neural network, a third neural network, and a fourth neural network, wherein: the first neural network is used to acquire images from multiple perspectives at the current time and extract features from the images from multiple perspectives to obtain high-dimensional image features of the multiple perspective images; the second neural network is used to fuse the high-dimensional image features of the multiple perspective images to obtain bird's-eye view features; the third neural network is used to extract the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current time; and the fourth neural network is used to generate a future feasible region prediction map based on a time-series queue composed of the high-dimensional image features of the bird's-eye view at the current time and the high-dimensional image features of the bird's-eye view at multiple historical times.

[0118] In one embodiment, fusing high-dimensional image features from multiple viewpoints to obtain bird's-eye view features includes: using the high-dimensional image features of the viewpoints as keys and the pixel position coordinates in the bird's-eye view features as query inputs to a second neural network, and obtaining the bird's-eye view features based on the output of the second neural network.

[0119] In one embodiment, the third neural network includes multiple convolutional layers to extract bird's-eye view features and obtain high-dimensional bird's-eye view image features. This includes: extracting bird's-eye view features through one of the convolutional layers, and then extracting features again from the results extracted by the previous convolutional layer through the next convolutional layer to obtain high-dimensional bird's-eye view image features.

[0120] In one embodiment, the fourth neural network includes a number of subnetworks equal to the number of high-dimensional bird's-eye view image features in the time-series queue. It generates a future feasible region prediction map based on the time-series queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments. This includes: generating high-dimensional bird's-eye view image features for the corresponding future moments based on the time-series queue using the respective subnetworks; and upsampling the high-dimensional bird's-eye view image features for the corresponding future moments to generate a future feasible region prediction map for that moment, thus obtaining feasible region prediction maps for multiple future moments.

[0121] In one embodiment, the feasible region prediction map is a probability map presented in a probabilistic manner, which represents the probability that a pixel in the panoramic image at a future time belongs to a drivable region.

[0122] In one embodiment, when the pixel value of a pixel in the probability map is not greater than a set threshold, the pixel in the future panoramic view corresponding to the pixel does not belong to the drivable area; when the pixel value of a pixel in the probability map is greater than the set threshold, the pixel in the future panoramic view corresponding to the pixel belongs to the drivable area.

[0123] According to embodiments of the present invention, a system for autonomous or assisted driving of a vehicle is also provided, the system including the feasible region prediction device described in any one of the foregoing descriptions. The description of the feasible region prediction device can be found above and will not be repeated here.

[0124] Furthermore, according to embodiments of the present invention, a storage medium is provided on which a computer program is stored. When the computer program is run by a computer or processor, it is used to execute the feasible region prediction method of the present invention and to implement corresponding modules in the feasible region prediction apparatus according to embodiments of the present invention. The storage medium may, for example, include a memory card of a smartphone, a storage component of a tablet computer, a hard disk of a personal computer, a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a portable compact disc read-only memory (CD-ROM), a USB memory, or any combination of the above storage media. The computer-readable storage medium may be any combination of one or more computer-readable storage media. For example, one computer-readable storage medium may contain computer-readable program code for obtaining bird's-eye view features from a panoramic image, and another computer-readable storage medium may contain computer-readable program code for extracting bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment.

[0125] In one embodiment, the computer program, when run by a computer, can implement the various functional modules of the feasible region prediction apparatus according to the embodiments of the present invention, and / or can execute the feasible region prediction method according to the embodiments of the present invention.

[0126] In one embodiment, when the computer program is run by a computer or processor, it causes the computer or processor to perform the following steps: acquiring a surround view image at the current moment and obtaining bird's-eye view features based on the surround view image; the surround view image includes images from multiple perspectives captured by multiple cameras on the vehicle; extracting the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view at the current moment; generating a future feasible region prediction map based on a time-series queue composed of the high-dimensional image features of the bird's-eye view at the current moment and high-dimensional image features of the bird's-eye view at multiple historical moments, and outputting the feasible region prediction map.

[0127] In one embodiment, obtaining bird's-eye view features from panoramic images includes: extracting features from images from multiple perspectives to obtain high-dimensional image features from multiple perspectives; and fusing the high-dimensional image features from multiple perspectives to obtain bird's-eye view features.

[0128] In one embodiment, fusing high-dimensional image features from multiple viewpoints to obtain bird's-eye view features includes: transforming the high-dimensional image features from multiple viewpoints from the coordinate system of their respective viewpoints to the vehicle coordinate system, and obtaining bird's-eye view features based on the transformation result.

[0129] In one embodiment, generating a future feasible region prediction map based on a temporal queue consisting of high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments includes: generating high-dimensional bird's-eye view image features at multiple future moments based on the temporal queue; and upsampling the high-dimensional bird's-eye view image features at each moment in the high-dimensional bird's-eye view image features at the multiple future moments to generate a feasible region prediction map at each moment, thereby obtaining a feasible region prediction map at multiple future moments.

[0130] In one embodiment, the steps that the computer program executes when run by a computer or processor are implemented by a trained neural network, which includes a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network, wherein: the first sub-network is used to acquire images from multiple perspectives at the current moment and extract features from the images from multiple perspectives to obtain high-dimensional image features from multiple perspectives; the second sub-network is used to fuse the high-dimensional image features from multiple perspectives to obtain bird's-eye view features; the third sub-network is used to extract bird's-eye view features to obtain high-dimensional bird's-eye view features at the current moment; and the fourth sub-network is used to generate a future feasible domain prediction map based on a temporal queue composed of the high-dimensional bird's-eye view features at the current moment and the high-dimensional bird's-eye view features from multiple historical moments.

[0131] In one embodiment, fusing high-dimensional image features from multiple viewpoints to obtain bird's-eye view features includes: using the high-dimensional image features of the viewpoints as keys and the pixel position coordinates in the bird's-eye view features as query inputs to a second sub-network, and obtaining the bird's-eye view features based on the output of the second sub-network.

[0132] In one embodiment, the third sub-network includes multiple convolutional layers to extract bird's-eye view features to obtain high-dimensional bird's-eye view image features. This includes: extracting bird's-eye view features through one of the convolutional layers, and then extracting features again from the results extracted by the previous convolutional layer through the next convolutional layer to obtain high-dimensional bird's-eye view image features.

[0133] In one embodiment, the fourth sub-network includes a number of sub-networks equal to the number of high-dimensional bird's-eye view image features in the temporal queue. It generates a future feasible region prediction map based on the temporal queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments. This includes: generating high-dimensional bird's-eye view image features for the corresponding future moments based on the temporal queue using the respective sub-networks; and upsampling the high-dimensional bird's-eye view image features for the corresponding future moments to generate a future feasible region prediction map for that moment, thus obtaining feasible region prediction maps for multiple future moments.

[0134] In one embodiment, the steps that the computer program executes when run by a computer or processor are implemented by multiple trained neural networks, including a first neural network, a second neural network, a third neural network, and a fourth neural network, wherein: the first neural network is used to acquire images from multiple perspectives at the current moment and extract features from the images from multiple perspectives to obtain high-dimensional image features of the multiple perspective images; the second neural network is used to fuse the high-dimensional image features of the multiple perspective images to obtain bird's-eye view features; the third neural network is used to extract bird's-eye view features to obtain high-dimensional bird's-eye view features at the current moment; and the fourth neural network is used to generate a future feasible domain prediction map based on a time-series queue composed of the high-dimensional bird's-eye view features at the current moment and the high-dimensional bird's-eye view features at multiple historical moments.

[0135] In one embodiment, fusing high-dimensional image features from multiple viewpoints to obtain bird's-eye view features includes: using the high-dimensional image features of the viewpoints as keys and the pixel position coordinates in the bird's-eye view features as query inputs to a second neural network, and obtaining the bird's-eye view features based on the output of the second neural network.

[0136] In one embodiment, the third neural network includes multiple convolutional layers to extract bird's-eye view features and obtain high-dimensional bird's-eye view image features. This includes: extracting bird's-eye view features through one of the convolutional layers, and then extracting features again from the results extracted by the previous convolutional layer through the next convolutional layer to obtain high-dimensional bird's-eye view image features.

[0137] In one embodiment, the fourth neural network includes a number of subnetworks equal to the number of high-dimensional bird's-eye view image features in the time-series queue. It generates a future feasible region prediction map based on the time-series queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features at multiple historical moments. This includes: generating high-dimensional bird's-eye view image features for the corresponding future moments based on the time-series queue using the respective subnetworks; and upsampling the high-dimensional bird's-eye view image features for the corresponding future moments to generate a future feasible region prediction map for that moment, thus obtaining feasible region prediction maps for multiple future moments.

[0138] In one embodiment, the feasible region prediction map is a probability map presented in a probabilistic manner, which represents the probability that a pixel in the panoramic image at a future time belongs to a drivable region.

[0139] In one embodiment, when the pixel value of a pixel in the probability map is not greater than a set threshold, the pixel in the future panoramic view corresponding to the pixel does not belong to the drivable area; when the pixel value of a pixel in the probability map is greater than the set threshold, the pixel in the future panoramic view corresponding to the pixel belongs to the drivable area.

[0140] Each module in the feasible region prediction device according to the present invention can be implemented by a computer program stored in a memory running by a processor of an electronic device according to the present invention, or by a computer program stored in a computer-readable storage medium of a computer program product according to the present invention being implemented by a computer running.

[0141] Furthermore, according to embodiments of the present invention, a computer program is also provided, which can be stored on a cloud or local storage medium. When this computer program is run by a computer or processor, it is used to perform corresponding steps of the feasible region prediction method of the embodiments of the present invention, and to implement corresponding modules in the feasible region prediction apparatus according to embodiments of the present invention.

[0142] Based on the above description, the feasible domain prediction method, apparatus, system, and storage medium according to embodiments of the present invention obtain a future feasible domain prediction map by using the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments. This enables the analysis and prediction of future scenarios, thereby providing a basis for the behavioral decisions of autonomous or assisted driving vehicles. Furthermore, the feasible domain prediction map is generated based on the acquired surround view image at the current moment. By combining scene perception and behavior prediction, the feasible domain prediction map can directly provide the future trajectory of obstacles, thereby dividing drivable and non-drivable areas. This avoids the redundant calculations and information accumulation errors caused by dividing feasible domain segmentation and obstacle prediction into two modules in related technologies, eliminating the need for independent behavior prediction.

[0143] Although exemplary embodiments have been described herein with reference to the accompanying drawings, it should be understood that the above exemplary embodiments are merely illustrative and are not intended to limit the scope of this application. Various changes and modifications can be made therein by those skilled in the art without departing from the scope and spirit of this application. All such changes and modifications are intended to be included within the scope of this application as claimed in the appended claims.

[0144] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0145] In the several embodiments provided in this application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are merely illustrative. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another device, or some features may be ignored or not executed.

[0146] Numerous specific details are set forth in the specification provided herein. However, it will be understood that embodiments of this application may be practiced without these specific details. In some instances, well-known methods, structures, and techniques have not been shown in detail so as not to obscure the understanding of this specification.

[0147] Similarly, it should be understood that, in order to streamline this application and aid in understanding one or more of the various inventive aspects, features of this application may sometimes be grouped together in a single embodiment, figure, or description thereof in the description of exemplary embodiments of this application. However, this approach should not be construed as reflecting an intention that the claimed application requires more features than are expressly recited in each claim. Rather, as reflected in the corresponding claims, its inventive point lies in solving the corresponding technical problem with features fewer than all features of a single disclosed embodiment. Therefore, the claims following the detailed description are hereby expressly incorporated into that detailed description, wherein each claim itself is a separate embodiment of this application.

[0148] Those skilled in the art will understand that, apart from the mutual exclusion of features, all features disclosed in this specification (including the accompanying claims, abstract, and drawings) and all processes or units of any method or apparatus so disclosed can be combined in any combination. Unless otherwise expressly stated, each feature disclosed in this specification (including the accompanying claims, abstract, and drawings) may be replaced by an alternative feature that serves the same, equivalent, or similar purpose.

[0149] Furthermore, those skilled in the art will understand that although some embodiments described herein include certain features but not others included in other embodiments, combinations of features from different embodiments are intended to be within the scope of this application and form different embodiments. For example, in the claims, any one of the claimed embodiments can be used in any combination.

[0150] The various component embodiments of this application can be implemented in hardware, or as software modules running on one or more processors, or a combination thereof. Those skilled in the art will understand that microprocessors or digital signal processors (DSPs) can be used in practice to implement some or all of the functions of some modules according to the embodiments of this application. This application can also be implemented as an apparatus program (e.g., a computer program and computer program product) for performing part or all of the methods described herein. Such an implementation of this application can be stored on a computer-readable medium, or can be in the form of one or more signals. Such signals can be downloaded from an Internet website, provided on a carrier signal, or provided in any other form.

[0151] It should be noted that the above embodiments are illustrative of this application and not limiting of it, and that those skilled in the art can devise alternative embodiments without departing from the scope of the appended claims. In the claims, any reference signs placed between parentheses should not be construed as limiting the claims. This application can be implemented by means of hardware comprising several different elements and by means of a suitably programmed computer. In the unit claims enumerating several means, several of these means may be embodied by the same item of hardware. The use of the words first, second, and third, etc., does not indicate any order. These words can be interpreted as names.

[0152] The above description is merely a specific embodiment or illustration of the embodiments of this application. The scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. The scope of protection of this application shall be determined by the scope of the claims.

Claims

1. A feasible region prediction method, the method being applied to autonomous or assisted driving of vehicles, characterized in that, include: The system acquires a surround view image at the current moment and obtains bird's-eye view features based on the surround view image; the surround view image includes images from multiple perspectives captured by multiple cameras on the vehicle. The bird's-eye view features are extracted to obtain the high-dimensional image features of the bird's-eye view at the current moment; A future feasible region prediction map is generated based on the time-series queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments, and the feasible region prediction map is output. The step of generating a future feasible domain prediction map based on the temporal queue composed of the high-dimensional image features of the current bird's-eye view and the high-dimensional image features of multiple historical bird's-eye views includes: Generate high-dimensional image features of bird's-eye views for multiple future moments based on the time-series queue; Upsampling is performed on the high-dimensional bird's-eye view features at each of the multiple future time points to generate a feasible region prediction map for each time point, thereby obtaining the feasible region prediction maps for the multiple future time points.

2. The feasible region prediction method as described in claim 1, characterized in that, The process of obtaining bird's-eye view features based on the panoramic image includes: Feature extraction is performed on the images from the multiple viewpoints to obtain high-dimensional image features of the multiple viewpoints; The high-dimensional image features of the multiple viewpoint images are fused to obtain the bird's-eye view features.

3. The feasible region prediction method as described in claim 2, characterized in that, The process of fusing the high-dimensional image features of the multiple viewpoint images to obtain the bird's-eye view features includes: The high-dimensional image features of the multiple viewpoint images are transformed from the coordinate system of the image at each viewpoint to the vehicle coordinate system, and the bird's-eye view features are obtained based on the transformation result.

4. The feasible region prediction method as described in claim 2, characterized in that, The method is implemented by a trained neural network, which includes a first sub-network, a second sub-network, a third sub-network, and a fourth sub-network, wherein: The first sub-network is used to acquire images from the multiple perspectives at the current time and to extract features from the multiple perspective images to obtain high-dimensional image features of the multiple perspective images; The second sub-network is used to fuse the high-dimensional image features of the multiple viewpoint images to obtain the bird's-eye view features; The third sub-network is used to extract the bird's-eye view features to obtain the high-dimensional image features of the bird's-eye view at the current moment; The fourth sub-network is used to generate a future feasible domain prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments.

5. The feasible region prediction method as described in claim 4, characterized in that, The process of fusing the high-dimensional image features of the multiple viewpoint images to obtain the bird's-eye view features includes: The high-dimensional image features of the viewpoint image are used as keys, and the pixel position coordinates in the bird's-eye view features are used as query inputs to the second sub-network. The bird's-eye view features are obtained based on the output of the second sub-network.

6. The feasible region prediction method as described in claim 4, characterized in that, The third sub-network includes multiple convolutional layers, and the extraction of the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view includes: The bird's-eye view features are extracted by one of the convolutional layers, and the results extracted by the previous convolutional layer are extracted again by the next convolutional layer to obtain the high-dimensional image features of the bird's-eye view.

7. The feasible region prediction method as described in claim 4, characterized in that, The fourth sub-network includes a number of sub-networks equal to the number of high-dimensional bird's-eye view image features in the temporal queue. The step of generating a future feasible domain prediction map based on the temporal queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features from multiple historical moments includes: Based on the respective sub-networks, high-dimensional image features of the bird's-eye view at the corresponding future time are generated according to the temporal queue; Upsampling is performed on the high-dimensional image features of the bird's-eye view at the corresponding future time to generate a feasible region prediction map for the corresponding future time, so as to obtain feasible region prediction maps for multiple future times.

8. The feasible region prediction method as described in claim 2, characterized in that, The method is implemented by multiple trained neural networks, including a first neural network, a second neural network, a third neural network, and a fourth neural network, wherein: The first neural network is used to acquire images from the multiple perspectives at the current time and to extract features from the images from the multiple perspectives to obtain high-dimensional image features of the multiple perspective images; The second neural network is used to fuse the high-dimensional image features of the multiple viewpoint images to obtain the bird's-eye view features; The third neural network is used to extract the features of the bird's-eye view to obtain the high-dimensional image features of the bird's-eye view at the current moment. The fourth neural network is used to generate a future feasible domain prediction map based on a temporal queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments.

9. The feasible region prediction method as described in claim 8, characterized in that, The process of fusing the high-dimensional image features of the multiple viewpoint images to obtain the bird's-eye view features includes: The high-dimensional image features of the viewpoint image are used as keys, and the pixel position coordinates in the bird's-eye view features are used as query inputs to the second neural network. The bird's-eye view features are obtained based on the output of the second neural network.

10. The feasible region prediction method as described in claim 8, characterized in that, The third neural network includes multiple convolutional layers, and the extraction of the bird's-eye view features to obtain high-dimensional image features of the bird's-eye view includes: The bird's-eye view features are extracted by one of the convolutional layers, and the results extracted by the previous convolutional layer are extracted again by the next convolutional layer to obtain the high-dimensional image features of the bird's-eye view.

11. The feasible region prediction method as described in claim 8, characterized in that, The fourth neural network includes a number of subnetworks equal to the number of high-dimensional bird's-eye view image features in the temporal queue. The step of generating a future feasible domain prediction map based on the temporal queue composed of the high-dimensional bird's-eye view image features at the current moment and high-dimensional bird's-eye view image features from multiple historical moments includes: Based on the respective sub-networks, high-dimensional image features of the bird's-eye view at the corresponding future time are generated according to the temporal queue; Upsampling is performed on the high-dimensional image features of the bird's-eye view at the corresponding future time to generate a feasible region prediction map for the corresponding future time, so as to obtain feasible region prediction maps for multiple future times.

12. The feasible region prediction method as described in claim 1, characterized in that, The feasible region prediction map is a probability map presented in a probabilistic manner, which is used to represent the probability that a pixel in the panoramic image at a future time belongs to the drivable region.

13. The feasible region prediction method as described in claim 12, characterized in that, When the pixel value of a pixel in the probability graph is not greater than a set threshold, the pixel in the future panoramic view corresponding to the pixel does not belong to the drivable area. When the pixel value of a pixel in the probability graph is greater than the set threshold, the pixel in the future panoramic view corresponding to the pixel belongs to the drivable area.

14. A feasible region prediction device, applied to autonomous or assisted driving of vehicles, characterized in that, include: The bird's-eye view feature module is used to acquire the surround view image at the current moment and obtain bird's-eye view features based on the surround view image; the surround view image includes images from multiple perspectives captured by multiple cameras on the vehicle; The high-dimensional image feature module for bird's-eye view is used to extract the features of the bird's-eye view to obtain the high-dimensional image features of the bird's-eye view at the current moment. The feasible region prediction graph module is used to generate a future feasible region prediction graph based on the time-series queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments, and output the feasible region prediction graph. The step of generating a future feasible domain prediction map based on the temporal queue composed of the high-dimensional image features of the current bird's-eye view and the high-dimensional image features of multiple historical bird's-eye views includes: Generate high-dimensional image features of bird's-eye views for multiple future moments based on the time-series queue; Upsampling is performed on the high-dimensional bird's-eye view features at each of the multiple future time points to generate a feasible region prediction map for each time point, thereby obtaining the feasible region prediction maps for the multiple future time points.

15. A feasible region prediction device, applied to autonomous or assisted driving of vehicles, characterized in that, include: Multiple cameras mounted on the vehicle are used to capture surround view images at the current moment; the surround view images include images from multiple perspectives; One or more processors, for: Obtain the panoramic image and derive bird's-eye view features based on the panoramic image; The bird's-eye view features are extracted to obtain the high-dimensional image features of the bird's-eye view at the current moment; A future feasible region prediction map is generated based on the time-series queue composed of the high-dimensional image features of the bird's-eye view at the current moment and the high-dimensional image features of the bird's-eye view at multiple historical moments, and the feasible region prediction map is output. The step of generating a future feasible domain prediction map based on the temporal queue composed of the high-dimensional image features of the current bird's-eye view and the high-dimensional image features of multiple historical bird's-eye views includes: Generate high-dimensional image features of bird's-eye views for multiple future moments based on the time-series queue; Upsampling is performed on the high-dimensional bird's-eye view features at each of the multiple future time points to generate a feasible region prediction map for each time point, thereby obtaining the feasible region prediction maps for the multiple future time points.

16. A feasible region prediction device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program that is executed by the processor, and the computer program, when executed by the processor, causes the processor to perform the feasible region prediction method according to any one of claims 1 to 14.

17. A system for autonomous or assisted driving of a vehicle, characterized in that, The system includes the feasible region prediction apparatus according to any one of claims 14 to 16.

18. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run by a processor, causes the processor to perform the feasible region prediction method according to any one of claims 1 to 13.

Images