Method, apparatus, medium, and electronic device for performing an environmental prediction task

Abstract

A method, device, medium and electronic device for performing an environment prediction task are disclosed. The method comprises: obtaining environment images collected by a plurality of cameras arranged at different viewing angles of a movable device at the same collection time, to obtain an environment image set; performing feature extraction on the environment image set to obtain a feature extraction result; performing spatial position coding on the feature extraction result to obtain a spatial position coding result; determining an initial environment feature tensor corresponding to a predetermined environment prediction task; determining a target environment feature tensor based on the feature extraction result, the spatial position coding result and the initial environment feature tensor; and determining a prediction result corresponding to the predetermined environment prediction task based on the target environment feature tensor. The embodiments of the present disclosure can realize prediction of future environment information, and can save the parameterization and sampling of the probability distribution and the tedious algorithm process of recursive prediction, thereby reducing the calculation amount and memory consumption, improving the prediction efficiency, and facilitating the landing use.

Description

TECHNICAL FIELD

[0001] The present disclosure relates to driving technology, and in particular, to a method, device, medium and electronic device for performing an environment prediction task. BACKGROUND

[0002] The application of automatic driving technology on mobile devices such as vehicles is becoming more and more widespread. When using automatic driving technology, information collected by sensor devices arranged on the mobile device can be referred to for predicting future environment information related to automatic driving, thereby providing prior information for planning and control of the mobile device. SUMMARY

[0003] At present, the scheme for predicting future environment information related to automatic driving has a large amount of calculation and low efficiency. In order to solve the above technical problems, the present disclosure provides a method, device, medium and electronic device for performing an environment prediction task to reduce the amount of calculation required for predicting future environment information and improve prediction efficiency.

[0004] According to one aspect of the present disclosure, a method for performing an environment prediction task is provided, comprising:

[0005] obtaining environment images respectively collected by a plurality of cameras arranged at different viewing angles of a mobile device at the same collection time to obtain an environment image set;

[0006] performing feature extraction on the environment image set to obtain a feature extraction result;

[0007] performing spatial position encoding on the feature extraction result to obtain a spatial position encoding result;

[0008] determining an initial environment feature tensor corresponding to a predetermined environment prediction task;

[0009] determining a target environment feature tensor based on the feature extraction result, the spatial position encoding result and the initial environment feature tensor;

[0010] determining a prediction result corresponding to the predetermined environment prediction task based on the target environment feature tensor.

[0011] According to another aspect of the present disclosure, a device for performing an environment prediction task is provided, comprising:

[0012] a first obtaining module configured to obtain environment images respectively collected by a plurality of cameras arranged at different viewing angles of a mobile device at the same collection time to obtain an environment image set;

[0013] a feature extraction module configured to perform feature extraction on the environment image set obtained by the first obtaining module to obtain a feature extraction result;

[0014] a spatial position encoding module, configured to perform spatial position encoding on the feature extraction result obtained by the feature extraction module to obtain a spatial position encoding result;

[0015] a first determination module, configured to determine an initial environment feature tensor corresponding to the predetermined environment prediction task;

[0016] a second determination module, configured to determine a target environment feature tensor based on the feature extraction result obtained by the feature extraction module, the spatial position encoding result obtained by the spatial position encoding module, and the initial environment feature tensor determined by the first determination module;

[0017] a third determination module, configured to determine a prediction result corresponding to the predetermined environment prediction task based on the target environment feature tensor determined by the second determination module.

[0018] According to still another aspect of the embodiments of the present disclosure, a computer readable storage medium is provided, the storage medium storing a computer program, the computer program being used to execute the method for performing an environment prediction task.

[0019] According to yet another aspect of the embodiments of the present disclosure, an electronic device is provided, the electronic device comprising:

[0020] a processor;

[0021] a memory for storing executable instructions of the processor;

[0022] the processor, configured to read the executable instructions from the memory and execute the instructions to implement the method for performing an environment prediction task.

[0023] According to yet another aspect of the embodiments of the present disclosure, a computer program product is provided, when instructions in the computer program product are executed by a processor, the method for performing an environment prediction task is executed.

[0024] The method, device, medium, electronic device and product for performing an environment prediction task provided by the embodiments of the present disclosure can realize the prediction of future environment information through the implementation of the operations such as the environment image set acquisition operation, the feature extraction operation, the spatial position encoding operation, and the initialization operation, and can reduce the calculation amount and memory consumption, improve the prediction efficiency, and facilitate the landing use. BRIEF DESCRIPTION OF DRAWINGS

[0025] Figure 1 is a flowchart of the method for performing an environment prediction task provided by an exemplary embodiment of the present disclosure.

[0026] Figure 2is a flowchart of a manner of determining a spatial position encoding result in an example embodiment of the present disclosure.

[0027] Figure 3 is a flowchart of a manner of determining a spatial position encoding result in another example embodiment of the present disclosure.

[0028] Figure 4 is a flowchart of a manner of obtaining a fusion feature map corresponding to each extraction feature map in an example embodiment of the present disclosure.

[0029] Figure 5 is a flowchart of a manner of obtaining a fusion feature map corresponding to a target extraction feature map in an example embodiment of the present disclosure.

[0030] Figure 6 is a schematic diagram of a principle of an attention mechanism.

[0031] Figure 7 is a flowchart of a manner of obtaining an initial environment feature tensor in an example embodiment of the present disclosure.

[0032] Figure 8 is a flowchart of a manner of generating a prediction result corresponding to a predetermined environment prediction task in an example embodiment of the present disclosure.

[0033] Figure 9 is a schematic diagram of a motion flow prediction map corresponding to a backward motion flow prediction task in an example embodiment of the present disclosure.

[0034] Figure 10 is a principle diagram of implementing prediction of future environment information in an example embodiment of the present disclosure.

[0035] Figure 11 is a schematic diagram of obtaining an initial environment feature tensor through an initialization operation in an example embodiment of the present disclosure.

[0036] Figure 12 is a principle diagram of generating spatial position information corresponding to an extraction feature map by using a 3D coordinate generator in an example embodiment of the present disclosure.

[0037] Figure 13 is a working principle diagram of a 3D position encoder in an example embodiment of the present disclosure.

[0038] Figure 14 is a structural schematic diagram of an apparatus for performing an environment prediction task provided in an example embodiment of the present disclosure.

[0039] Figure 15-1 is a structural schematic diagram of a spatial position encoding module in an example embodiment of the present disclosure.

[0040] Figure 15-2is a structural diagram of a spatial position encoding module in another example embodiment of the present disclosure.

[0041] Figure 16 is a structural diagram of a first determining module in an example embodiment of the present disclosure.

[0042] Figure 17 is a structural diagram of a third determining module in an example embodiment of the present disclosure.

[0043] Figure 18 is a structural diagram of an electronic device provided by an example embodiment of the present disclosure. DETAILED DESCRIPTION

[0044] In order to explain the present disclosure, the example embodiments of the present disclosure will be described in detail below with reference to the accompanying drawings. It is obvious that the described embodiments are only a part of the embodiments of the present disclosure, but not all the embodiments. It should be understood that the present disclosure is not limited by the example embodiments.

[0045] It should be noted that: unless otherwise specified, the relative arrangement, numerical expression and numerical value of the components and steps set forth in these embodiments do not limit the scope of the present disclosure.

[0046] SUMMARY

[0047] When using automatic driving technology, information collected by sensor devices arranged on the movable device can be referred to for predicting future environment information related to automatic driving, thereby providing prior information for planning control of the movable device.

[0048] In the related art, in order to realize the prediction of the future environment information, the following steps generally need to be performed: converting the environment image into 3D form data; projecting the 3D form data into a Bird's Eye View (BEV) space to obtain bird's eye view data; time encoding the bird's eye view data to obtain spatio-temporal state information; parameterizing two probability distributions, which are a current distribution and a future distribution, the current distribution is only associated with the current spatio-temporal state information, and the future distribution is associated with the current spatio-temporal state information and observable future label information; selecting a sampling point from the current distribution and the future distribution; based on the sampling point and the current spatio-temporal state information, performing recursive prediction via a future prediction model to obtain predicted future state information; and predicting the required future environment information based on the future state information.

[0049] In the process of implementing the present disclosure, the inventors found that in the above-mentioned scheme for predicting the future environment information, since parameterization and sampling of the probability distribution are required, and recursive prediction is required, the implementation has a large amount of calculation and low efficiency, which is not conducive to landing use.

[0050] Exemplary methods

[0051] Figure 1 is a flowchart of a method for performing an environment prediction task provided by an example embodiment of the present disclosure. Figure 1 The method shown can be applied on an electronic device, for example, on a vehicle-mounted computing platform. Figure 1 The method shown can include steps 110, 120, 130, 140, 150, and 160, which are described below respectively.

[0052] Step 110, obtaining environment images respectively collected by a plurality of cameras arranged at different viewing angles of a movable device at a same collection time, to obtain an environment image set.

[0053] Optionally, the movable device includes but is not limited to a vehicle, an airplane, a ship, etc., which are not listed one by one here.

[0054] Optionally, the plurality of cameras arranged at different viewing angles of the movable device can be represented as N cameras. The N cameras can be 6 cameras, which can be respectively located at the front, left front, right front, back, left back, and right back of the movable device. Of course, the value of N is not limited to 6, for example, it can also be 4, 8, etc., which are not listed one by one here. At the same collection time, N environment images can be collected by the N cameras to obtain an environment image set including the N environment images.

[0055] Step 120, performing feature extraction on the environment image set to obtain a feature extraction result.

[0056] In step 120, the N environment images in the environment image set can be respectively subjected to feature extraction by a feature extractor to obtain N extracted feature maps corresponding to the N environment images one by one. The feature extraction result can include the N extracted feature maps obtained. The feature extractor can be a convolutional neural network or other types of neural networks. Each extracted feature map can be considered as a 2D image feature (also referred to as 2D features). The size of each extracted feature map can be represented as [H_f, W_f, C], where H_f represents the height, W_f represents the width, and C represents the number of channels. Optionally, C can be greater than 3, for example, C can be 10, 12, 15, etc., which are not listed one by one here.

[0057] Step 130, performing spatial position encoding on the feature extraction result to obtain a spatial position encoding result.

[0058] In step 130, the feature extraction result can be spatial position coded by a spatial position encoder (also referred to as a 3D position encoder), that is, 2D image features are added with spatial position information (also referred to as 3D position information) to obtain a spatial position coding result that carries both 2D image features and 3D position information.

[0059] In step 140, an initialization environment feature tensor corresponding to the predetermined environment prediction task is determined.

[0060] Optionally, the predetermined environment prediction task can refer to a task for predicting future environment information of a predetermined dimension to provide prior information for planning control of the movable device. The predetermined dimension can be a dimension related to the position, motion condition, etc. of an object around the movable device. The object around the movable device includes but is not limited to vehicles, pedestrians, cyclists, etc. Cyclists include but are not limited to people riding bicycles, people riding electric bikes, people riding motorcycles, etc.

[0061] In step 140, the initialization environment feature tensor corresponding to the predetermined environment prediction task can be generated through an initialization operation. The initialization environment feature tensor can be a high-dimensional tensor for carrying environment features around the movable device. The initialization environment feature tensor can be in the form of a feature map. The size of the initialization environment feature tensor can be represented as [H_q, W_q, C], where H_q represents the height, W_q represents the width, and C represents the number of channels.

[0062] In step 150, a target environment feature tensor is determined based on the feature extraction result, the spatial position coding result, and the initialization environment feature tensor.

[0063] In step 150, the initialization environment feature tensor can be updated based on the feature extraction result and the spatial position coding result to obtain the target environment feature tensor by using a certain updating method. The attention mechanism can be introduced in the above updating method, including but not limited to the self-attention (Self Attention) mechanism, the cross-attention (Cross Attention) mechanism, etc.

[0064] In step 160, a prediction result corresponding to the predetermined environment prediction task is determined based on the target environment feature tensor.

[0065] In step 160, the target environment feature tensor can be decoded to obtain the prediction result corresponding to the predetermined environment prediction task. The prediction result can include future environment information predicted for the predetermined dimension.

[0066] Of course, the implementation of step 160 is not limited to this, for example, the target environment feature tensor can be optimized first to obtain an optimized environment feature tensor more accurate and effective than the target environment feature tensor, and then the optimized environment feature tensor is decoded to obtain the prediction result corresponding to the predetermined environment prediction task.

[0067] In the embodiments of the present disclosure, the multi-view image data (equivalent to the environment image set in the foregoing) can be obtained through the multiple cameras arranged at different viewing angles of the movable device. The feature extraction result carrying the 2D image features can be obtained by performing feature extraction on the multi-view image data. The spatial position encoding result carrying both the 2D image features and the 3D position information can be obtained by performing spatial position encoding on the feature extraction result. Since the 2D image features and the 3D position information are closely related to the actual scene in which the movable device is located, the target environment feature tensor is determined based on the feature extraction result, the spatial position encoding result and the initialized environment feature tensor, which is conducive to ensuring that the determined target environment feature tensor is adapted to the actual scene in which the movable device is located, thereby providing an effective reference for determining the prediction result corresponding to the predetermined environment prediction task, so as to realize the prediction of future environment information. It can be seen that in the embodiments of the present disclosure, the prediction of future environment information is realized through the implementation of the operations of obtaining the environment image set, feature extraction, spatial position encoding, initialization and the like, without the need for parameterization and sampling of the probability distribution, and without the need for recursive prediction, so as to eliminate the cumbersome algorithm processes such as parameterization and sampling of the probability distribution, and recursive prediction, thereby reducing the computational amount and memory consumption, improving the prediction efficiency, and facilitating the landing use.

[0068] In some optional examples, as shown in FIG. 1, step 130 includes steps 1301, 1302, 1303 and 1304. Figure 2

[0069] Step 1301 determines the imaging beam direction information corresponding to each pixel point in each extraction feature map in the feature extraction result.

[0070] As introduced above, the size of each extraction feature map in the feature extraction result can be represented as [H_f, W_f, C], and assuming that H_f*W_f=M, each extraction feature map can include M pixel points. For each extraction feature map, the imaging beam direction information corresponding to each of the M pixel points included in the extraction feature map can be determined.

[0071] ​Optionally, the imaging light beam corresponding to any pixel point can refer to a light beam passing through a camera optical center corresponding to the extraction feature map where the pixel point is located and the pixel point. The camera optical center corresponding to the extraction feature map where any pixel point is located can be understood as the optical center of the camera used to collect the environment image corresponding to the extraction feature map when collecting the environment image. In this way, the imaging light beam corresponding to any pixel point can be represented by a straight line equation, which can be used as the imaging light beam direction information corresponding to the pixel point. Of course, the imaging light beam direction information corresponding to any pixel point is not limited to the form of a straight line equation, for example, it can also be in the form of a vector.

[0072] Step 1302, sampling along the preset depth range on the imaging light beam represented by the imaging light beam direction information corresponding to each pixel point to obtain a set of spatial sampling points corresponding to each pixel point.

[0073] Optionally, for any pixel point, a plurality of spatial sampling points can be obtained by sampling along the preset depth range on the imaging light beam corresponding to the pixel point according to a predetermined sampling rule. The set of spatial sampling points corresponding to the pixel point can include the obtained plurality of spatial sampling points. The predetermined sampling rule includes but is not limited to equidistant sampling, random sampling, etc. The preset depth range can be set according to actual conditions, and the present disclosure does not limit it.

[0074] Step 1303, determining the spatial position information corresponding to each extraction feature map based on the set of spatial sampling points corresponding to each pixel point.

[0075] Optionally, the set of spatial sampling points corresponding to each pixel point can include P spatial sampling points, and then the 3D coordinates of the P spatial sampling points in the set of spatial sampling points corresponding to the pixel point in the world coordinate system can be determined to obtain P 3D coordinates, and the set of P 3D coordinates can be referred to as a 3D coordinate set. The spatial position information corresponding to each extraction feature map can include M 3D coordinate sets corresponding one-to-one to the M pixel points in the extraction feature map.

[0076] Step 1304, determining the spatial position encoding result based on each extraction feature map and the corresponding spatial position information.

[0077] Optionally, for each extraction feature map, a fusion algorithm can be used to fuse the extraction feature map and the spatial position information corresponding to the extraction feature map to obtain a fusion feature map corresponding to the extraction feature map. The spatial position encoding structure can include the fusion feature map corresponding to each extraction feature map.

[0078] Generally, any pixel point in the image can be regarded as a bundle of light rays in the real world projected onto the imaging plane of the camera, and then the spatial point corresponding to the pixel point in the real world should be located on the light ray. In view of this, in the embodiments of the present disclosure, for each pixel point in each extracted feature map, the imaging light beam corresponding to the pixel point can be sampled along the preset depth range to obtain a set of spatial sampling points. Each spatial sampling point in the obtained set of spatial sampling points can be regarded as a spatial point that may correspond to the pixel point. Therefore, based on the obtained set of spatial sampling points, the spatial position information corresponding to the extracted feature map is determined, which is beneficial to guarantee the rationality and reliability of the determined spatial position information, thereby being beneficial to guarantee the rationality and reliability of the subsequently obtained spatial position encoding result.

[0079] In some optional examples, as shown in Figure 3 Step 130 includes step 1305, step 1306 and step 1307.

[0080] Step 1305 determines the spatial position information corresponding to each pixel point in each extracted feature map in the feature extraction result, to obtain the spatial position information corresponding to each extracted feature map.

[0081] Optionally, for each pixel point in each extracted feature map, the 3D coordinate set corresponding to the pixel point can be determined according to Figure 2 According to the related description in the embodiment shown in

[0082] Step 1306 fuses each extracted feature map and the corresponding spatial position information to obtain a fused feature map corresponding to each extracted feature map.

[0083] In some optional embodiments of the present disclosure, as shown in Figure 4 Step 1306 includes step 13061, step 13062 and step 13063.

[0084] Step 13061 determines, based on the spatial position information corresponding to each extracted feature map, a reference feature map for spatial position encoding corresponding to each extracted feature map.

[0085] Optionally, the size of each extracted feature map can be represented as [H_f, W_f, C], and each set of spatial sampling points can include P spatial sampling points. Then, an initialization feature map with a size of [H_f, W_f, P*4] can be generated through an initialization operation, H_f represents height, W_f represents width, and P*4 represents the number of channels.

[0086] For any extracted feature map, since the height and width of the initial feature map are the same as those of the extracted feature map, the pixels in the initial feature map can be mapped one-to-one with the pixels in the extracted feature map. For example, the pixel with coordinates (u1, v1) in the initial feature map can be mapped to the pixel with coordinates (u1, v1) in the extracted feature map. Assume that the P 3D coordinates in the 3D coordinate set corresponding to the pixel with coordinates (u1, v1) in the extracted feature map are represented as (x1, y1, z1), (x2, y2, z2), ..., (x... P y P , z P Then, x1, y1, z1, 1, x2, y2, z2, 1, ..., x P y P z P The eigenvalue 1 is assigned as the feature value to the pixel with coordinates (u1, v1) in the initial feature map, and x1, y1, z1, 1, x2, y2, z2, 1, ..., x P y P z P The M pixels can be arranged sequentially in the P*4 channels of the initial feature map. Following this method, feature values ​​can be assigned to all M pixels in the initial feature map to update the initial feature map, thus obtaining an updated feature map. This updated feature map can then be used as the reference feature map corresponding to the extracted feature map.

[0087] In some embodiments, the size of the initial feature map can also be [H_f, W_f, P*3], so that x1, y1, z1, x2, y2, z2, ..., x P y P z P Assigned as feature values ​​to the pixel with coordinates (u1, v1) in the initialized feature map, where x1, y1, z1, x2, y2, z2, ..., x P y P z P The P*3 channels of the initial feature map can be arranged sequentially, and the update of the initial feature map can also be achieved in this way.

[0088] Step 13062: Align each extracted feature map with its corresponding reference feature map to obtain an aligned feature map group for each extracted feature map.

[0089] Optionally, the size of each extracted feature map can be represented as [H_f, W_f, C], and the size of the reference feature map corresponding to the extracted feature map can be represented as [H_f, W_f, P*4]. Assuming that P*4 is greater than C, the size of the reference feature map corresponding to the extracted feature map can be reduced from [H_f, W_f, P*4] to [H_f, W_f, C] using a multilayer perceptron (MLP). The aligned feature map group corresponding to the extracted feature map can include the extracted feature map and the reference feature map with the size reduced to [H_f, W_f, C].

[0090] In some embodiments, the size of the extracted feature map can be increased from [H_f, W_f, C] to [H_f, W_f, P*4]. The aligned feature map group corresponding to the extracted feature map may include the extracted feature map with the size increased to [H_f, W_f, P*4] and the reference feature map corresponding to the extracted feature map.

[0091] Step 13063: Fuse the two feature maps in the aligned feature map group corresponding to each extracted feature map to obtain the fused feature map corresponding to each extracted feature map.

[0092] Optionally, for each extracted feature map, the two feature maps in the aligned feature map group corresponding to the extracted feature map can be added element-wise, multiplied element-wise, etc., and the feature map obtained after the operation can be used as the fused feature map corresponding to the extracted feature map.

[0093] The previous paragraph introduced the fusion of two feature maps in an aligned feature map group through addition or multiplication operations. In practice, the fusion of two feature maps in an aligned feature map group can also be achieved through weighted summation, mean operation, or other operation logic.

[0094] use Figure 4 In the embodiment shown, for each extracted feature map, the spatial location information corresponding to the extracted feature map can first be converted into a feature map form. Then, the extracted feature map and the converted reference feature map are aligned in size to ensure the size consistency of the two aligned feature maps. In this way, the two aligned feature maps can be fused normally, thereby effectively realizing the fusion of the extracted feature map and the spatial location information corresponding to the extracted feature map.

[0095] In other alternative embodiments of this disclosure, such as Figure 5 As shown, step 1306 includes steps 13064, 13065, 13066, 13067, 13068 and 13069.

[0096] Step 13064, determining a reference time from the plurality of acquisition times corresponding to the plurality of environment image sets respectively.

[0097] Optionally, the number of the plurality of environment image sets can be multiple, and the plurality of environment image sets can be represented as R environment image sets. The acquisition times corresponding to the R environment image sets can be different. R can be 2, 3, or an integer greater than 3, which will not be listed one by one here.

[0098] In step 13064, the acquisition time closest to the current time can be selected from the R acquisition times corresponding to the R environment image sets as the reference time. Of course, the reference time is not limited to this, for example, the second closest acquisition time to the current time can be selected from the R acquisition times as the reference time.

[0099] Step 13065, determining a target environment image set from the plurality of environment image sets, the acquisition time corresponding to the target environment image set being different from the reference time.

[0100] Since one of the R acquisition times is used as the reference time, the remaining R-1 acquisition times of the R acquisition times can each correspond to an environment image set as a target environment image set, that is, the number of target environment image sets can be R-1.

[0101] Step 13066, determining a target extraction feature map corresponding to each environment image in the target environment image set from the feature extraction result.

[0102] In step 120 above, for each of the R environment image sets, the N environment images in the environment image set can be subjected to feature extraction to obtain an extraction result corresponding to the environment image set. The extraction result can include N extraction feature maps corresponding to the N environment images in the environment image set one by one. In this way, the feature extraction result obtained in step 120 can include R extraction results corresponding to the R environment image sets.

[0103] In step 13066, the target extraction result corresponding to the target environment image set can be first found from the feature extraction result, and then the N extraction feature maps can be obtained from the target extraction result. Each of the N extraction feature maps obtained can be a target extraction feature map.

[0104] Step 13067, determining the pose change information of the movable device between the acquisition time corresponding to the target environment image set and the reference time.

[0105] Optionally, the movable device can be provided with an inertial measurement unit (IMU). It can be understood that the inertial measurement unit generally includes an accelerometer and a gyroscope, the accelerometer can be used to measure acceleration, and the gyroscope can be used to measure angular velocity and angular acceleration.

[0106] In step 13067, data collected by the inertial measurement unit can be calculated to determine the pose change information of the movable device between the collection time corresponding to the target environment image set and the reference time. The pose change information can be in the form of a matrix, which is used to represent the translation and rotation of the movable device between the collection time corresponding to the target environment image set and the reference time.

[0107] In step 13068, the spatial position information corresponding to the target extraction feature map is converted from the collection time corresponding to the target environment image set to the reference time by using the pose change information, and a conversion result is obtained.

[0108] As introduced above, the spatial position information corresponding to each extraction feature map can include M 3D coordinate sets corresponding to M pixel points in the extraction feature map. In step 13068, for each 3D coordinate set in the spatial position information corresponding to the target extraction feature map, each 3D coordinate in the 3D coordinate set can be multiplied by the matrix used to represent the pose change information, and the 3D coordinate is updated to the multiplication result obtained. In this way, the update of the spatial position information corresponding to the target extraction feature map can be realized, and the updated spatial position information can be used as the conversion result in step 13068.

[0109] In step 13069, the target extraction feature map is fused with the conversion result to obtain a fusion feature map corresponding to the target extraction feature map.

[0110] It should be noted that the implementation of step 13069 can refer to the introduction of the fusion manner of the extraction feature map and the corresponding spatial position information in the embodiment shown in Figure 4 The description of the embodiment is omitted here.

[0111] In the implementation shown in Figure 5 By using the pose change information of the movable device, the target extraction feature map can be converted from the non-reference time to the reference time, which is equivalent to aligning the spatial position information corresponding to different extraction feature maps to the reference time, realizing the time alignment between different extraction feature maps. Using the time-aligned extraction feature maps in the subsequent steps is conducive to avoiding the error introduced by the time misalignment, thereby being conducive to ensuring the accuracy and reliability of the future environment information finally obtained.

[0112] At step 1307, the spatial position encoding result is determined based on the fusion feature map corresponding to each extracted feature map.

[0113] Optionally, the spatial position encoding result can include the fusion feature map corresponding to each extracted feature map, and the fusion feature maps corresponding to different extracted feature maps can be independent of each other. Alternatively, the fusion feature map corresponding to each extracted feature map can be superimposed along the channel direction to obtain a superimposed feature map, which can be taken as the spatial position encoding result.

[0114] In the embodiments of the present disclosure, for each pixel point in each extracted feature map in the feature extraction result, the corresponding spatial position information can be determined respectively. By integrating these spatial position information, the spatial position information corresponding to each extracted feature map can be efficiently and reliably obtained. By fusing each extracted feature map with the corresponding spatial position information, the fusion feature map carrying 2D image features and 3D position information at the same time can be efficiently and reliably obtained. In this way, the spatial position encoding result determined based on the fusion feature map can also carry 2D image features and 3D position information at the same time.

[0115] In some optional examples, step 150 includes:

[0116] The spatial position encoding result is taken as the key tensor of the cross-attention network, the feature extraction result is taken as the value tensor of the cross-attention network, and the initialized environment feature tensor is taken as the query tensor of the cross-attention network to obtain the target environment feature tensor.

[0117] It should be noted that the cross-attention network is a network based on the cross-attention mechanism. The cross-attention network can be a Transformer network, and the Transformer network can adopt an encoding-decoding structure.

[0118] Generally speaking, the principle of the attention mechanism is to calculate the attention distribution on the given information (i.e., to judge what information is important and what information is not important), and to perform weighted average operation according to the attention distribution to obtain the attention calculation result. Optionally, the principle of the attention mechanism can refer to Figure 6 , which is: Source includes a series of <Key, Value> data pairs (also known as key-value pairs), and for element Query, the similarity between Query and each Key is calculated to obtain the weight coefficient corresponding to each Key. The weight coefficients are used to perform weighted summation on each Value to obtain the final Attention calculation result (i.e., Attention Value).

[0119] In an embodiment of the present disclosure, the spatial position encoding result can be taken as a key tensor of the cross-attention network, the feature extraction result can be taken as a value tensor of the cross-attention network, and the initialized environment feature tensor can be taken as a query tensor of the cross-attention network. Then, the spatial position encoding result and the feature extraction result can form a query tensor. Figure 6 The initialized environment feature tensor can be taken as a Query in the Source. Figure 6 The AttentionValue in the Source can be taken as a target environment feature tensor obtained by querying. Figure 6 The AttentionValue in the Source can be taken as a target environment feature tensor obtained by querying. By introducing the cross-attention mechanism, the feature interaction between the initialized environment feature tensor and the spatial position encoding result and the feature extraction result can be realized. In the interaction process, important features can be focused on as much as possible to learn useful information from the spatial position encoding result and the feature extraction result for updating the initialized environment feature tensor, so that a more accurate and effective target environment feature tensor is obtained compared with the initialized environment feature tensor. This is conducive to ensuring the accuracy and reliability of the future environment information finally obtained.

[0120] In some optional examples, as shown in Figure 7 Step 140 includes step 1401, step 1402, step 1403, and step 1404.

[0121] Step 1401 determines the spatial coordinates corresponding to each of the plurality of grids in the bird's eye view based on a preset bird's eye view size and a preset size ratio relationship between the bird's eye view and the real scene perception range.

[0122] Optionally, the preset bird's eye view size can be represented as [H_q, W_q, 3], and the size ratio relationship between the preset bird's eye view and the real scene perception range can be in the form of a ratio, for example, represented as K.

[0123] Since the bird's eye view size is [H_q, W_q, 3], the bird's eye view size corresponds to a bird's eye view feature map that can include H_q*W_q grids. Thus, the plurality of grids in the bird's eye view in step 1401 can be H_q*W_q grids. For any grid in the plurality of grids, the pixel coordinates of the grid can be determined, for example, the value of the column where the grid is located can be taken as the horizontal coordinate in the pixel coordinates of the grid, and the value of the row where the grid is located can be taken as the vertical coordinate in the pixel coordinates of the grid. Based on K and the pixel coordinates of the grid, the spatial coordinates (x, y, z) corresponding to the grid can be determined. Assuming that the pixel coordinates of the grid are (u2, v2), then x=u2*K, y=v2*K, and z=0.

[0124] The size ratio relationship described above represents the ratio of the size of the real scene perception range to the size of the bird's-eye view. In specific implementation, the size ratio relationship can also represent the ratio of the size of the bird's-eye view to the size of the real scene perception range, so that x = u2 / K, y = v2 / K, z = 0.

[0125] Step 1402: Based on the bird's-eye view size and the spatial coordinates corresponding to each of the multiple grids, determine the first reference tensor for feature tensor initialization under the bird's-eye view.

[0126] Optionally, an initial bird's-eye view feature map of size [H_q, W_q, 3], comprising multiple grids, can be generated. For each grid in the initial bird's-eye view feature map, the x, y, and 0 coordinates corresponding to that grid can be assigned as feature values, and x, y, and 0 can be sequentially arranged in the three channels of the initial bird's-eye view feature map. Following this method, feature values ​​can be assigned to each pixel in the initial bird's-eye view feature map to update it, thus obtaining an updated bird's-eye view feature map, which can then be used as the first reference tensor.

[0127] Step 1403: Upgrade the first reference tensor to obtain the second reference tensor.

[0128] Optionally, the size of the first reference tensor can be represented as [H_q, W_q, 3]. In step 1403, the size of the first reference tensor can be increased from [H_q, W_q, 3] to [H_q, W_q, C] by a multilayer perceptron to obtain the second reference tensor.

[0129] Step 1404: Based on the second reference tensor, determine the initial environmental feature tensor corresponding to the predetermined environment prediction task.

[0130] Optionally, the size of the second reference tensor can be [H_q, W_q, C]. In this case, the second reference tensor can be determined as the initialization environment feature tensor.

[0131] In some embodiments, the size of the second reference tensor may not be [H_q, W_q, C], but may be [H_q, W_q, 2C]. In such cases, the size of the second reference tensor may be adjusted to obtain a tensor with a size of [H_q, W_q, C] as the initialization environment feature tensor.

[0132] In the embodiments of the present disclosure, based on the aerial view size and the size ratio relationship, the spatial coordinates corresponding to each of the plurality of grids in the aerial view perspective can be reasonably determined, which can be used together with the aerial view size to determine the first reference tensor. The first reference tensor can be used to perform basic characterization on the relevant position information of the environment around the movable device in the world coordinate system. Through dimensionality increasing processing of the first reference tensor, a higher-dimensional second reference tensor can be efficiently and reliably obtained to initialize the determination of the environment feature tensor. In this way, through the initialization operation, the initialized environment feature tensor carrying the environment features around the movable device is obtained, and the initialization process is simple and easy to implement.

[0133] In some optional examples, as shown in FIG. 16, step 160 includes steps 1601, 1602, 1603 and 1604. Figure 8

[0134] Step 1601 obtains the point cloud data collected by the radar arranged on the movable device at the collection time corresponding to the environment image set.

[0135] Optionally, the radar arranged on the movable device includes but is not limited to a laser radar, a millimeter wave radar, etc.

[0136] Suppose that the radar arranged on the movable device actually collects data at the collection time corresponding to the environment image set, the point cloud data actually collected by the radar at the collection time can be used as the point cloud data in step 1601. Suppose that the radar arranged on the movable device does not actually collect data at the collection time corresponding to the environment image set, but actually collects data at a first time earlier than and adjacent to the collection time and a second time later than and adjacent to the collection time, the point cloud data corresponding to the collection time can be obtained by interpolating the point cloud data collected at the first time and the point cloud data collected at the second time, and used as the point cloud data in step 1601.

[0137] Step 1602 generates an aerial feature map using the point cloud data.

[0138] Optionally, the point cloud data can be converted into the aerial feature map in any implementable manner, and the present disclosure does not limit the specific conversion manner. The target environment feature tensor can be in the form of a feature map, and the height and width of the aerial feature map can be the same as the height and width of the target environment feature tensor, respectively.

[0139] Step 1603 fuses the target environment feature tensor and the aerial feature map to obtain a fused tensor.

[0140] ​In step 1603, the target environment feature tensor can be superimposed with the bird's eye view feature map along the channel direction, and the obtained superimposed feature map can be taken as the fusion tensor. Alternatively, the obtained superimposed feature map can be resized so that the resized superimposed feature map meets the input requirements of subsequent processing, and the resized superimposed feature map can be taken as the fusion tensor.

[0141] In step 1604, based on the fusion tensor, a prediction result corresponding to a predetermined environment prediction task is generated via a prediction network corresponding to the predetermined environment prediction task.

[0142] Optionally, the prediction network corresponding to the predetermined environment prediction task can be obtained by pre-training. The prediction network can also be referred to as a prediction head. The prediction network includes an up-sampling layer and a multi-layer perceptron. The fusion tensor can be equivalent to the optimized environment feature tensor in the foregoing, and the prediction network can decode the fusion tensor to obtain the prediction result corresponding to the predetermined environment prediction task.

[0143] In the embodiments of the present disclosure, the future environment information can be predicted based on the point cloud data and the multi-view image data, which can improve the richness of the information on which the prediction of the future environment information is based. In addition, the point cloud data can be used to obtain accurate depth information, and thus the accuracy and reliability of the finally obtained future environment information can be improved.

[0144] In some optional examples, the predetermined environment prediction task includes an environment occupancy map prediction task.

[0145] Optionally, the environment occupancy map prediction task can be used to predict the occupancy of each grid in the plurality of grids at each future time in the plurality of future times, and the occupancy can include a probability value. The prediction result corresponding to the environment occupancy map prediction task can be an environment occupancy probability map. The size of the environment occupancy probability map can be represented as [T_p, H_p, W_p, 1], where T_p represents the number of predicted future frames, H_p represents the height, W_p represents the width, and 1 represents a one-dimensional probability value.

[0146] In the embodiments of the present disclosure, through the execution of the environment occupancy map prediction task, it can be determined which positions around the movable device at the future time are occupied, for example, it can be determined that the positions with a probability value greater than a preset probability value (for example, 0.6, 0.7, etc.) in the environment occupancy probability map are occupied. This is beneficial for reasonably planning the trajectory of the movable device, ensuring smooth movement of the movable device on the road surface, and avoiding collision of the movable device.

[0147] In some optional examples, the predetermined environment prediction task includes a motion flow prediction task.

[0148] Optionally, the motion flow prediction task can be used to predict a motion condition of a dynamic object (also referred to as a dynamic object) around the movable device at each of the plurality of future time instants, the motion condition including but not limited to a motion direction, a motion speed, and the like.

[0149] In some optional embodiments of the present disclosure,

[0150] The motion flow prediction task includes a backward motion flow prediction task.

[0151] The prediction result includes a motion flow prediction map in a bird's eye view corresponding to the backward motion flow prediction task.

[0152] The motion flow prediction map includes a motion direction of each object corresponding to each grid, and / or a motion speed of each object corresponding to each grid.

[0153] Any object motion direction is a direction from a grid occupied by an object at a future time instant to a grid occupied by the object at a previous time instant of the future time instant.

[0154] Optionally, a size of the motion flow prediction map corresponding to the backward motion flow prediction task can be represented as [T_p, H_p, W_p, 2], T_p representing a predicted number of future frames, H_p representing a height, W_p representing a width, and 2 representing a two-dimensional vector, a direction of the two-dimensional vector representing an object motion direction, and a length of the two-dimensional vector representing an object motion speed.

[0155] In an optional example, the motion flow prediction map corresponding to the backward motion flow prediction task can be as shown in FIG. 9. Figure 9 Assuming that a grid occupied by an object at a future time instant is grid 91, a grid occupied by the object at a previous time instant of the future time instant can be grid 92, grid 93, or grid 94, and the like, which is conducive to realizing multi-modal trajectory prediction for dynamic objects. In addition, any grid will only be occupied by one dynamic object at a future time instant, and will not be occupied by multiple dynamic objects, which is conducive to avoiding collision problems in motion flow prediction.

[0156] Of course, the motion flow prediction task package is not limited to the backward motion flow prediction task, and can also include a forward motion flow prediction task. In a motion flow prediction map corresponding to the forward motion flow prediction task, any object motion direction can be a direction from a grid occupied by an object at a previous time instant of a future time instant to a grid occupied by the object at the future time instant.

[0157] In the embodiments of the present disclosure, through execution of the motion flow prediction task, a motion condition of a dynamic object around the movable device at a future time instant can be determined, which is conducive to reasonably planning a trajectory of the movable device, ensuring smooth movement of the movable device on a road surface, and avoiding collision of the movable device.

[0158] In some optional examples, in order to realize the prediction of future environment information, as shown in Figure 10 The environment image set corresponding to the t-th moment and the environment image set corresponding to the (t-1)-th moment can be obtained through the vehicle-mounted surround view camera system (equivalent to the camera system composed of N cameras in the foregoing description). The t-th moment can be the current moment, and the (t-1)-th moment can be the previous moment of the current moment.

[0159] Through the initialization operation, the occupancy motion flow occflow query (equivalent to the initialization environment feature tensor in the case that the predetermined environment prediction task includes both the environment occupancy map prediction task and the motion flow prediction task) can be obtained. Optionally, the process of the initialization operation can refer to the related description in the embodiment shown in Figure 11 and the related description in the embodiment shown in 7.

[0160] The convolutional neural network (equivalent to the feature extractor in the foregoing description) can be used to perform feature extraction on both the environment image set corresponding to the t-th moment and the environment image set corresponding to the (t-1)-th moment, to obtain a feature extraction result. Assuming that the size of each extracted feature map in the feature extraction result is represented as [H_f, W_f, C], the size of the feature extraction result can be represented as [2N, H_f, W_f, C].

[0161] The 3D coordinate generator can be used to generate spatial position information corresponding to each extracted feature map in the feature extraction result. The determination principle of the spatial position information corresponding to the extracted feature map can refer to the related description in the foregoing description. In brief, as shown in Figure 12 for each pixel point on the imaging plane, a spatial point in the real space that can correspond to the pixel point is determined to obtain a set of spatial sampling points, and based on the obtained set of spatial sampling points, corresponding spatial position information is determined.

[0162] The pose change information of the movable device between the (t-1)-th moment and the t-th moment can be used to time-align (i.e., align to the t-th moment) the spatial position information corresponding to each extracted feature map in the feature extraction result.

[0163] The 3D position encoder shown in Figure 13 can be used to fuse each extracted feature map and the spatial position information corresponding to the extracted feature map, to obtain a fused feature map corresponding to the extracted feature map, and accordingly obtain a spatial position encoding result. Figure 13 1x1 in the foregoing description represents a convolutional layer with a 1x1 convolution kernel, relu represents a rectified linear unit (ReLU) layer, and sigmoid represents a kind of activation function.

[0164] The spatial position encoding result is taken as a key tensor of the Transformer network (equivalent to the cross-attention network in the foregoing description), the feature extraction result is taken as a value tensor of the Transformer network, and the occflow query obtained through the initialization operation is taken as a query tensor of the Transformer network for querying, so that the occflow query can learn image features in the Transformer network to obtain an updated occflow query (equivalent to the target environment feature tensor in the foregoing description).

[0165] The updated occflow query can be decoded by using the occflow prediction head (equivalent to the prediction network in the foregoing description) to obtain an environment occupancy probability map and a motion flow prediction map, so that the prediction of future environment information (which can include the environment occupancy probability map and the motion flow prediction map) is realized.

[0166] In summary, the embodiments of the present disclosure can use the images of different perspectives obtained by the vehicle-mounted surround view camera system to realize the prediction from the sensor input to the scene-level environment occupancy probability map and the motion flow prediction map in an end-to-end manner, provide prior information for reasonable planning and control of the autonomous vehicle, and the prediction process eliminates the steps of feature projection to the bird's eye view space, parameterization and sampling of the probability distribution, and recursive prediction, greatly simplifies the algorithm process, reduces the algorithm calculation amount and memory consumption, and improves the prediction efficiency. Since the spatial position encoding is performed on the 2D image features to assign spatial position information to the 2D image features, the features do not need to be converted from two dimensions to three dimensions, which is beneficial to avoid errors caused by the conversion process and reduce the algorithm calculation amount. In addition, through the execution of the backward motion flow prediction task, multi-modal motion prediction can be realized, and the collision problem in the motion flow prediction can be avoided.

[0167] Exemplary apparatus

[0168] Figure 14 FIG. 1 is a structural schematic diagram of an apparatus for performing an environment prediction task provided by an example embodiment of the present disclosure. Figure 14 The apparatus shown includes a first acquisition module 1410, a feature extraction module 1420, a spatial position encoding module 1430, a first determination module 1440, a second determination module 1450, and a third determination module 1460.

[0169] The first acquisition module 1410 is configured to acquire environment images respectively collected by a plurality of cameras arranged at different perspectives of a movable device at the same collection time, to obtain an environment image set.

[0170] The feature extraction module 1420 is configured to perform feature extraction on the environment image set obtained by the first acquisition module 1410, to obtain a feature extraction result.

[0171] The spatial location encoding module 1430 is used to perform spatial location encoding on the feature extraction results obtained by the feature extraction module 1420 to obtain the spatial location encoding result.

[0172] The first determining module 1440 is used to determine the initial environmental feature tensor corresponding to the predetermined environment prediction task;

[0173] The second determining module 1450 is used to determine the target environment feature tensor based on the feature extraction result obtained by the feature extraction module 1420, the spatial position encoding result obtained by the spatial position encoding module 1430, and the initial environment feature tensor determined by the first determining module 1440.

[0174] The third determining module 1460 is used to determine the prediction result corresponding to the predetermined environment prediction task based on the target environment feature tensor determined by the second determining module 1450.

[0175] In some optional examples, such as Figure 15-1 As shown, the spatial location encoding module 1430 includes:

[0176] The first determining submodule 14301 is used to determine the spatial location information corresponding to each pixel in each extracted feature map in the feature extraction result obtained by the feature extraction module 1420, and obtain the spatial location information corresponding to each extracted feature map.

[0177] The first fusion submodule 14302 is used to fuse each extracted feature map with the corresponding spatial location information determined by the first determining submodule 14301 to obtain a fused feature map corresponding to each extracted feature map.

[0178] The second determining submodule 14303 is used to determine the spatial location encoding result based on the fused feature map corresponding to each extracted feature map obtained by the first fusion submodule 14302.

[0179] In some optional examples, the first fusion submodule 14302 includes:

[0180] The first determining unit is used to determine the reference time from the acquisition times corresponding to the multiple environmental image sets;

[0181] The second determining unit is used to determine, from multiple environmental image sets, a target environmental image set whose acquisition time is different from the reference time determined by the first determining unit;

[0182] The third determining unit is used to determine the target extracted feature map corresponding to each environment image in the target environment image set determined by the second determining unit from the feature extraction results obtained by the feature extraction module 1420.

[0183] The fourth determining unit is used to determine the pose change information of the mobile device between the acquisition time corresponding to the target environment image set determined by the second determining unit and the reference time determined by the first determining unit.

[0184] The conversion unit is used to convert the spatial position information corresponding to the target extracted feature map determined by the third determination unit from the acquisition time corresponding to the target environment image set determined by the second determination unit to the reference time determined by the first determination unit using the pose change information determined by the fourth determination unit, so as to obtain the conversion result.

[0185] The first fusion unit is used to fuse the target extraction feature map determined by the third determining unit with the transformation result obtained by the transformation unit to obtain the fused feature map corresponding to the target extraction feature map determined by the third determining unit.

[0186] In some optional examples, the first fusion submodule 14302 includes:

[0187] The fifth determining unit is used to determine a reference feature map for spatial location encoding corresponding to each extracted feature map based on the spatial location information corresponding to each extracted feature map determined by the first determining submodule 14301.

[0188] The size alignment unit is used to align the size of each extracted feature map with the corresponding reference feature map determined by the first determining submodule 14301 to obtain an aligned feature map group corresponding to each extracted feature map.

[0189] The second fusion unit is used to fuse the two feature maps in the aligned feature map group corresponding to each extracted feature map obtained by the size alignment unit to obtain the fused feature map corresponding to each extracted feature map.

[0190] In some optional examples, such as Figure 15-2 As shown, the spatial location encoding module 1430 includes:

[0191] The third determining submodule 14304 is used to determine the imaging beam direction information corresponding to each pixel in each extracted feature map in the feature extraction result obtained by the feature extraction module 1420;

[0192] The sampling submodule 14305 is used to sample along a preset depth range on the imaging beam characterized by the imaging beam direction information corresponding to each pixel point determined by the third determining submodule 14304, so as to obtain a set of spatial sampling points corresponding to each pixel point.

[0193] The fourth determining submodule 14306 is used to determine the spatial location information corresponding to each extracted feature map based on the spatial sampling point set corresponding to each pixel point obtained by the sampling submodule 14305.

[0194] The fifth determining sub-module 14307 is configured to determine a spatial position encoding result based on each extracted feature map and the corresponding spatial position information determined by the fourth determining sub-module 14306.

[0195] In some optional examples, the second determining module 1450 includes:

[0196] The query sub-module is configured to query the spatial position encoding result obtained by the spatial position encoding module 1430 as a key tensor of the cross-attention network, the feature extraction result obtained by the feature extraction module 1420 as a value tensor of the cross-attention network, and the initialization environment feature tensor determined by the first determining module 1440 as a query tensor of the cross-attention network, to obtain a target environment feature tensor.

[0197] In some optional examples, as shown in Figure 16 The first determining module 1440 includes:

[0198] The sixth determining sub-module 14401 is configured to determine, based on a preset aerial view size and a preset size ratio relationship between the aerial view and a real scene perception range, spatial coordinates corresponding to each of the plurality of grids in the aerial view perspective;

[0199] The seventh determining sub-module 14402 is configured to determine, based on the aerial view size and the spatial coordinates corresponding to each of the plurality of grids determined by the sixth determining sub-module 14401, a first reference tensor for feature tensor initialization in the aerial view perspective;

[0200] The dimension increasing sub-module 14403 is configured to increase the dimension of the first reference tensor determined by the seventh determining sub-module 14402 to obtain a second reference tensor;

[0201] The eighth determining sub-module 14404 is configured to determine, based on the second reference tensor obtained by the dimension increasing sub-module 14403, an initialization environment feature tensor corresponding to a predetermined environment prediction task.

[0202] In some optional examples, as shown in Figure 17 The third determining module 1460 includes:

[0203] The ninth determining sub-module 14601 is configured to obtain, from the radar arranged on the movable device, point cloud data collected at a collection time corresponding to the set of environment images obtained by the first obtaining module 1410;

[0204] The first generating sub-module 14602 is configured to generate an aerial view feature map using the point cloud data obtained by the ninth determining sub-module 14601.

[0205] The second fusion submodule 14603 is configured to fuse the target environment feature tensor determined by the second determination submodule 1450 and the bird's eye view feature map generated by the first generation submodule 14602, to obtain a fusion tensor.

[0206] The second generation submodule 14604 is configured to generate a prediction result corresponding to a predetermined environment prediction task based on the fusion tensor obtained by the second fusion submodule 14603 via a prediction network corresponding to the predetermined environment prediction task.

[0207] In some optional examples, the predetermined environment prediction task includes:

[0208] an environment occupancy map prediction task;

[0209] and / or,

[0210] a motion flow prediction task.

[0211] In some optional examples,

[0212] the motion flow prediction task includes a backward motion flow prediction task;

[0213] the prediction result includes a motion flow prediction map in a bird's eye view corresponding to the backward motion flow prediction task;

[0214] the motion flow prediction map includes a plurality of grids each corresponding to an object motion direction; and / or, a plurality of grids each corresponding to an object motion speed;

[0215] any object motion direction is a direction from a grid occupied by an object at a future moment to a grid occupied at a moment before the future moment.

[0216] In the apparatus of the present disclosure, the various optional embodiments, optional implementation manners and optional examples disclosed above can be flexibly selected and combined as needed to achieve corresponding functions and effects, and the present disclosure does not make one-by-one enumeration.

[0217] Exemplary electronic device

[0218] Figure 18 FIG. 1 illustrates a block diagram of an electronic device according to an embodiment of the present disclosure. The electronic device 100 can include one or more processors 110 and memory 120.

[0219] The processor 110 can be a central processing unit (CPU) or other form of processing unit having data processing and / or instruction execution capabilities. The processor 110 can control other components in the electronic device 100 to perform desired functions.

[0220] The memory 1820 can include one or more computer program products that can include various forms of computer-readable storage media having stored thereon one or more computer program instructions. The computer program instructions can be executed by the processor 1810 to implement the methods described above and / or other desired function. The computer-readable storage media can include, for example, volatile memory, such as SRAM and / or cache memory; non-volatile memory, such as ROM, hard disks, and / or flash memory; or any combination of the above. The computer-readable storage media can be removable without limitation, such as disks, memory cards, and / or other storage media.

[0221] In one example, the electronic device 1800 can further include an input device 1830 and an output device 1840, which are interconnected through a bus system and / or other forms of connection mechanisms (not shown).

[0222] The input device 1830 can include, for example, a keyboard, a mouse, and / or the like.

[0223] The output device 1840 can output various information to the outside, which can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto, and / or the like.

[0224] Of course, for simplicity, Figure 18 Only some of the components of the electronic device 1800 related to the present disclosure are shown in FIG. 18, and components such as buses, input / output interfaces, and / or the like are omitted. In addition, the electronic device 1800 can further include any other appropriate components, depending on the specific application.

[0225] Exemplary computer program product and computer readable storage medium

[0226] In addition to the above-mentioned methods and devices, embodiments of the present disclosure can be a computer program product including computer program instructions that, when executed by a processor, cause the processor to perform steps of the methods according to various embodiments of the present disclosure described in the above “Exemplary Methods” section of the specification.

[0227] The computer program product can be written in any combination of one or more programming languages, including an object-oriented programming language such as Java, C++, and / or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computing device, partly on the user's device, as a stand-alone software package, partly on the user's computing device and partly on a remote computing device or entirely on the remote cloud device or server.

[0228] Furthermore, an embodiment of the present disclosure can also be a computer readable storage medium having stored thereon computer program instructions which, when executed by a processor, cause the processor to perform steps of a method according to the embodiments of the present disclosure as described in the above “Exemplary Methods” section of the specification.

[0229] The computer readable storage medium can be any combination of one or more computer readable medium. The computer readable storage medium can be any combination of one or more computer readable medium.

[0230] The above describes the basic principles of the present disclosure in conjunction with specific embodiments, however, the advantages, benefits, effects and the like mentioned in the present disclosure are only examples and are not limiting, and it cannot be considered that these advantages, benefits, effects and the like are necessarily possessed by each embodiment of the present disclosure. The above specific details of the disclosure are only for the purpose of example and understanding, and are not limiting, and the above details do not limit the present disclosure to necessarily adopt the above specific details to realize.

[0231] Those skilled in the art can make various modifications and variations to the present disclosure without departing from the spirit and scope of the present disclosure. Thus, if these modifications and variations of the present disclosure fall within the scope of the claims of the present disclosure and their equivalent technologies, the present disclosure also intends to include these modifications and variations.

Claims

1. A method for performing an environment prediction task, comprising: obtaining environment images collected by a plurality of cameras arranged at different perspectives of a movable device at a same collection time, to obtain an environment image set; performing feature extraction on the environment image set to obtain a feature extraction result; performing spatial position encoding on the feature extraction result to obtain a spatial position encoding result; determining an initial environment feature tensor corresponding to a predetermined environment prediction task, the initial environment feature tensor being a high-dimensional tensor for carrying environment features around the movable device; based on the feature extraction result, the spatial position encoding result and the initial environment feature tensor, determining a target environment feature tensor, comprising: updating the initial environment feature tensor based on the feature extraction result and the spatial position encoding result, to obtain the target environment feature tensor; based on the target environment feature tensor, determining a prediction result corresponding to the predetermined environment prediction task.

2. The method of claim 1, wherein, The spatial position encoding result is obtained by performing spatial position encoding on the feature extraction result, comprising: determining spatial position information corresponding to each pixel point in each extracted feature map in the feature extraction result, to obtain spatial position information corresponding to each extracted feature map; fusing each extracted feature map and the corresponding spatial position information to obtain a fused feature map corresponding to each extracted feature map; based on the fused feature map corresponding to each extracted feature map, determining the spatial position encoding result.

3. The method of claim 1, wherein, The spatial position encoding result is obtained by performing spatial position encoding on the feature extraction result, comprising: determining imaging beam direction information corresponding to each pixel point in each extracted feature map in the feature extraction result; sampling along a preset depth range on an imaging beam represented by the imaging beam direction information corresponding to each pixel point, to obtain a set of spatial sampling points corresponding to each pixel point; based on the set of spatial sampling points corresponding to each pixel point, determining spatial position information corresponding to each extracted feature map; based on each extracted feature map and the corresponding spatial position information, determining the spatial position encoding result.

4. The method of claim 2, wherein, The fused feature map corresponding to each extracted feature map is obtained by fusing each extracted feature map and the corresponding spatial position information, comprising: determining a reference time from a plurality of collection times corresponding to the environment image set; from a plurality of environment image sets, determining a target environment image set whose corresponding collection time is different from the reference time; from the feature extraction result, determining a target extracted feature map corresponding to each environment image in the target environment image set; determining pose change information of the movable device between the collection time corresponding to the target environment image set and the reference time; using the pose change information, converting the spatial position information corresponding to the target extracted feature map from the collection time corresponding to the target environment image set to the reference time, to obtain a conversion result; The target extraction feature map is fused with the conversion result to obtain a fused feature map corresponding to the target extraction feature map.

5. The method of claim 2, wherein, The fusion of the each extraction feature map and the corresponding spatial position information to obtain the fused feature map corresponding to the each extraction feature map comprises: Based on the spatial position information corresponding to the each extraction feature map, a reference feature map for spatial position encoding corresponding to the each extraction feature map is determined; The each extraction feature map and the corresponding reference feature map are size-aligned to obtain an aligned feature map group corresponding to the each extraction feature map; The two feature maps in the aligned feature map group corresponding to the each extraction feature map are fused to obtain the fused feature map corresponding to the each extraction feature map.

6. The method of claim 1, wherein, The determination of the target environment feature tensor based on the feature extraction result, the spatial position encoding result and the initialized environment feature tensor comprises: The spatial position encoding result is taken as a key tensor of a cross-attention network, the feature extraction result is taken as a value tensor of the cross-attention network, and the initialized environment feature tensor is taken as a query tensor of the cross-attention network to obtain a target environment feature tensor.

7. The method of claim 1, wherein, The determination of the initialized environment feature tensor corresponding to the predetermined environment prediction task comprises: Based on a preset aerial view size and a preset size proportion relationship between the aerial view and a real scene perception range, spatial coordinates corresponding to each of a plurality of grids in an aerial view perspective are determined; Based on the aerial view size and the spatial coordinates corresponding to each of the plurality of grids in the aerial view perspective, a first reference tensor for feature tensor initialization in the aerial view perspective is determined; The first reference tensor is dimensionally upgraded to obtain a second reference tensor; Based on the second reference tensor, an initialized environment feature tensor corresponding to the predetermined environment prediction task is determined.

8. The method of claim 1, wherein, The determination of the prediction result corresponding to the predetermined environment prediction task based on the target environment feature tensor comprises: Point cloud data collected by a radar arranged on the movable device at a collection time corresponding to the environment image set is acquired; An aerial view feature map is generated by using the point cloud data; The target environment feature tensor is fused with the aerial view feature map to obtain a fused tensor; Based on the fused tensor, a prediction result corresponding to the predetermined environment prediction task is generated via a prediction network corresponding to the predetermined environment prediction task.

9. The method of any one of claims 1-8, wherein, The predetermined environment prediction task comprises: an environment occupancy map prediction task; and / or, a motion flow prediction task.

10. The method of claim 9, wherein the motion flow prediction task comprises a backward motion flow prediction task; the prediction result comprises a motion flow prediction map in an aerial view perspective corresponding to the backward motion flow prediction task; the motion flow prediction map comprises an object motion direction corresponding to each of a plurality of grids and / or an object motion speed corresponding to each of the plurality of grids; any object motion direction is a direction from a grid occupied by an object at a future time to a grid occupied at a time preceding the future time.

11. An apparatus for performing an environment prediction task, comprising: The first obtaining module is configured to obtain environment images collected by a plurality of cameras arranged at different view angles of the movable device at a same collection time, to obtain an environment image set; The feature extraction module is configured to perform feature extraction on the environment image set obtained by the first obtaining module, to obtain a feature extraction result; The spatial position encoding module is configured to perform spatial position encoding on the feature extraction result obtained by the feature extraction module, to obtain a spatial position encoding result; The first determining module is configured to determine an initial environment feature tensor corresponding to a predetermined environment prediction task, the initial environment feature tensor being a high-dimensional tensor for carrying environment features around the movable device; The second determining module is configured to determine a target environment feature tensor based on the feature extraction result obtained by the feature extraction module, the spatial position encoding result obtained by the spatial position encoding module, and the initial environment feature tensor determined by the first determining module, including: updating the initial environment feature tensor based on the feature extraction result and the spatial position encoding result, to obtain the target environment feature tensor; The third determining module is configured to determine a prediction result corresponding to the predetermined environment prediction task based on the target environment feature tensor determined by the second determining module. 12.A computer readable storage medium, the storage medium storing a computer program, the computer program being configured to execute the method for performing an environment prediction task according to any one of claims 1-10. 13.An electronic device, the electronic device comprising: a processor; a memory for storing executable instructions of the processor; the processor is configured to read the executable instructions from the memory and execute the instructions to implement the method for performing an environment prediction task according to any one of claims 1-10.

Images