Three-dimensional (3D) grid occupation method and device

Abstract

The 3D occupancy grid prediction method comprises the following steps: acquiring multi-modal data of a surrounding environment of a terminal device at a first moment; and on the basis of the multi-modal data, a 3D occupation grid of the terminal device at a second moment is predicted, and the second moment is later than the first moment. Therefore, through a mode of guiding the 3D occupied grids, prediction of future 3D occupied grids through historical multi-modal data is realized, generalization of prediction of the future 3D occupied grids is remarkably improved, the limitation that a scene evolution process cannot be fully described due to the fact that only a single occupied semantic mode is depended on is avoided, and the scene evolution efficiency is improved. And meanwhile, the problem that the prediction performance of the future 3D occupied grids is greatly reduced due to the limitation of historical 3D occupied grid data and a single mode is solved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence (AI) technology, and in particular to a 3D occupancy grid method and apparatus. Background Technology

[0002] In the development of autonomous driving technology, complex, voxel-level semantic understanding of the three-dimensional space surrounding the vehicle is becoming increasingly important. This understanding is achieved through a high-level perception task called "occupancy prediction." Occupancy prediction aims to accurately identify the semantic category of each voxel in three-dimensional space, that is, to determine what type of object occupies each voxel, in order to predict the evolution of future scenes. Through this high-level semantic understanding, autonomous driving systems can perceive their surroundings more accurately, thereby significantly improving the safety and efficiency of navigation and decision-making.

[0003] In related technologies, efforts are made to directly predict future occupancy. While these methods take into account occupancy changes in three-dimensional space to some extent, they typically rely on historical occupancy data. However, historical occupancy data is often difficult to obtain in practical applications, especially when the environment changes rapidly or sensor data is discontinuous, making this reliance on historical data very limiting. Summary of the Invention

[0004] This application provides a 3D occupancy grid prediction method, apparatus, electronic device, vehicle, computer storage medium, and computer product that can improve the generalization of future 3D occupancy grid prediction.

[0005] In a first aspect, this application provides a 3D occupancy grid prediction method, comprising: acquiring multimodal data of the surrounding environment of a terminal device at a first moment; and predicting the 3D occupancy grid of the terminal device at a second moment based on the multimodal data, wherein the second moment is later than the first moment.

[0006] In this way, by guiding the use of 3D occupancy grids, the prediction of future 3D occupancy grids is achieved using historical multimodal data. This significantly improves the generalization of future 3D occupancy grid prediction, avoids the limitation of relying solely on a single occupancy semantic modality, which cannot fully characterize the scene evolution process, and overcomes the problem of a significant drop in the performance of future 3D occupancy grid prediction due to limitations in historical 3D occupancy grid data and a single modality.

[0007] In one possible implementation, predicting the 3D occupancy grid of a terminal device at a second time step based on multimodal data includes: encoding each modality of the multimodal data to obtain multiple encoded features, wherein one encoded feature is related to one modality of the multimodal data; concatenating the multiple encoded features to obtain a concatenated feature; obtaining a predicted feature based on the concatenated feature; and decoding the predicted feature to obtain the 3D occupancy grid. Thus, by first encoding the data for each modality separately, then concatenating the encoded features, and processing them using a neural network model, it is possible to predict the future 3D occupancy grid using multimodal data.

[0008] In one possible implementation, predictive features are obtained based on concatenated features, including: mapping the concatenated features to an index space using a codebook to obtain historical index features, where each historical index feature contains at least one index value from the codebook, and each index value is related to a feature vector obtained by flattening the concatenated features; processing the historical index features using an autoregressive model to obtain future index features; and inversely mapping the future index features from the index space to the space containing the concatenated features using the codebook to obtain the predictive features. In this way, prediction accuracy can be improved by aligning multimodal features to a unified feature space.

[0009] In one possible implementation, the multimodal data includes images and radar point clouds.

[0010] In one possible implementation, the terminal device is a vehicle or a robot.

[0011] Secondly, this application provides a 3D occupancy grid prediction device, comprising: an acquisition module for acquiring multimodal data of the surrounding environment of a terminal device at a first moment; and a processing module for predicting the 3D occupancy grid of the terminal device at a second moment based on the multimodal data, wherein the second moment is later than the first moment.

[0012] In one possible implementation, when the processing module predicts the 3D occupancy grid of the terminal device at the second time based on the multimodal data, it specifically performs the following: encodes each modality of the multimodal data to obtain multiple encoded features, wherein one encoded feature is related to one modality of the multimodal data; concatenates the multiple encoded features to obtain a concatenated feature; obtains a predicted feature based on the concatenated feature; and decodes the predicted feature to obtain the 3D occupancy grid.

[0013] In one possible implementation, when the processing module obtains the predicted features based on the concatenated features, it specifically performs the following steps: combining the codebook, mapping the concatenated features to the index space to obtain historical index features, wherein the historical index features contain at least one index value in the codebook, and an index value is related to a feature vector obtained by flattening the concatenated features; processing the historical index features through an autoregressive model to obtain future index features; and combining the codebook, inversely mapping the future index features from the index space to the space where the concatenated features are located to obtain the predicted features.

[0014] In one possible implementation, the multimodal data includes images and radar point clouds.

[0015] In one possible implementation, the terminal device is a vehicle or a robot.

[0016] Thirdly, this application provides a 3D occupancy grid prediction device, comprising: at least one processor and an interface; the at least one processor acquiring program instructions through the interface; and the at least one processor executing the program instructions to implement the method described in the first aspect. Exemplarily, the device may be, but is not limited to, a chip.

[0017] Fourthly, this application provides an electronic device, comprising: at least one memory for storing a program; and at least one processor for executing the program stored in the memory; wherein, when the program stored in the memory is executed, the processor is configured to perform the method as described in the first aspect.

[0018] Fifthly, this application provides a vehicle including electronic equipment as described in the fourth aspect.

[0019] In a sixth aspect, this application provides a computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method described in the first aspect.

[0020] In a seventh aspect, this application provides a computer program product, characterized in that, when the computer program product is run on a processor, it causes the processor to execute the method described in the first aspect.

[0021] It is understood that the beneficial effects of the second to seventh aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0022] Figure 1 This is a schematic diagram of the architecture of a 3D occupancy grid prediction system provided in an embodiment of this application;

[0023] Figure 2This is a schematic diagram of the architecture of another 3D occupancy grid prediction system provided in an embodiment of this application;

[0024] Figure 3 yes Figure 2 The diagram shown illustrates the working process of the system.

[0025] Figure 4 yes Figure 2 The diagram shows the training process of the system.

[0026] Figure 5 yes Figure 2 A schematic diagram illustrating the training and inference process of the neural network model used in the system shown.

[0027] Figure 6 This is a flowchart illustrating a 3D occupancy grid method provided in an embodiment of this application;

[0028] Figure 7 This is a schematic diagram illustrating the steps of predicting the 3D occupancy grid of a terminal device at a second moment based on multimodal data, as provided in an embodiment of this application.

[0029] Figure 8 This is a schematic diagram illustrating a step for obtaining predicted features based on spliced ​​features, provided in an embodiment of this application.

[0030] Figure 9 This is a schematic diagram comparing the performance of the 3D occupancy grid prediction scheme provided in this application embodiment with existing traditional schemes;

[0031] Figure 10 This is a schematic diagram illustrating the accuracy changes in predicting more future frame moments using the 3D occupancy grid prediction scheme provided in the embodiments of this application;

[0032] Figure 11 The occupancy prediction result at time T is obtained by the 3D occupancy grid prediction scheme provided in the embodiments of this application.

[0033] Figure 12 The occupancy prediction result at time T+1 is obtained by the 3D occupancy grid prediction scheme provided in the embodiments of this application.

[0034] Figure 13 The occupancy prediction result at time T+2 is obtained by the 3D occupancy grid prediction scheme provided in the embodiments of this application.

[0035] Figure 14 This is a schematic diagram of the structure of a 3D occupancy grid device provided in an embodiment of this application;

[0036] Figure 15This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application;

[0037] Figure 16 This is a schematic diagram of another 3D occupancy grid device provided in an embodiment of this application;

[0038] Figure 17 This is a schematic diagram of the architecture of an autonomous driving system provided in an embodiment of this application. Detailed Implementation

[0039] In this article, the term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone. The symbol " / " in this article indicates that the related objects are in an "or" relationship; for example, A / B means A or B.

[0040] The terms "first" and "second," etc., used in the specification and claims herein are used to distinguish different objects, not to describe a specific order of objects. For example, "first response message" and "second response message," etc., are used to distinguish different response messages, not to describe a specific order of response messages.

[0041] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0042] In the description of the embodiments of this application, unless otherwise stated, "multiple" means two or more, for example, multiple processing units means two or more processing units, multiple elements means two or more elements, etc.

[0043] In view of the problems described above, this application provides a three-dimensional (3D) occupancy grid prediction scheme. The concept is to predict future 3D occupancy grids using historical multimodal data, guided by a 3D occupancy grid. This significantly improves the generalization of future 3D occupancy grid prediction, avoids the limitation of relying solely on a single occupancy semantic modality to fully characterize the scene evolution process, and overcomes the problem of a significant performance drop in future 3D occupancy grid prediction due to limitations imposed by historical 3D occupancy grid data and a single modality.

[0044] The following is an introduction to this 3D occupancy raster prediction method.

[0045] For example, Figure 1 A schematic diagram of the architecture of a 3D occupancy grid prediction system provided in an embodiment of this application is shown. Figure 1 As shown, the 3D occupancy grid prediction system 100 includes: multiple encoders (e.g., encoder 111, encoder 112, etc.), a feature stitching module 120, a neural network model 130, and a decoder 140. An encoder can be used to encode data of a certain modality at the current time to obtain the encoded features of the data under the corresponding modality. For example, an image modality encoder can be used to encode image data, point cloud modality data can be used to encode radar point clouds, and so on. The feature stitching module 120 is mainly used to stitch together the encoded features of data from different modalities. For example, the feature stitching module 120 can add the encoded features along the channel dimension to stitch these encoded features together. For example, when the dimension of the encoded features under the image modality is (B*H*W*C1), and the dimension of the encoded features under the radar point cloud modality is (B*H*W*C2), the dimension of the stitched feature can be (B*H*W*(C1+C2)).

[0046] The neural network model 130 is mainly used to process the spliced ​​features output by the feature splicing module 120 to obtain predicted features. The decoder 140 is mainly used to decode the predicted features output by the neural network model 130 to obtain the 3D occupancy grid for the next time step. In this embodiment, multimodal data can be used as samples and the 3D occupancy grid as labels to train the 3D occupancy grid system 100. In this way, the system can predict the future 3D occupancy grid using multimodal data.

[0047] exist Figure 1 The system shown achieves effective unification of multimodal features through the guidance of 3D occupancy grids, significantly improving the generalization of future 3D occupancy grid predictions. This method avoids the limitation of relying solely on a single occupancy semantic modality, which fails to fully characterize the scene evolution process. It also overcomes the problem of significant performance degradation in future 3D occupancy grid predictions caused by traditional methods being limited by historical 3D occupancy grid data and single modalities.

[0048] exist Figure 1 In the system shown, to improve prediction accuracy, the stitched features output by the feature stitching module 120 can be aligned to unify the features of different modalities into the same feature space. Based on this concept, this embodiment provides another 3D occupancy grid prediction system. For example... Figure 2As shown, the 3D occupancy grid prediction system 200 includes: multiple encoders (e.g., encoder 111, encoder 112, etc.), a feature stitching module 120, a feature transformation module 210, a neural network model 130, a feature inverse transformation module 220, and a decoder 140. The functions of the encoders, feature stitching module 120, neural network model 130, and decoder 140 can be found in the aforementioned... Figure 1 The relevant descriptions will not be repeated here.

[0049] The feature transformation module 210 is mainly used to unify the features of different modal data into the same feature space. For example, the feature transformation module 210 can first flatten the concatenated features output by the feature concatenation module 120 into multiple feature vectors. For instance, if the dimension of the concatenated features output by the feature concatenation module 120 is B*H*W*D, then the number of flattened feature vectors can be (B*H*W), and the length of each feature vector can be D. Then, it queries the codebook 230 for feature vectors similar to each vector, and represents the flattened feature vectors using the index corresponding to the queried feature vectors. The inverse feature transformation module 220 is mainly used to transform the predicted features output by the neural network model 130 into the space where the corresponding modal data resides. The processing procedure of the inverse feature transformation module 220 can be the reverse of the processing procedure of the feature transformation module 210, which will not be elaborated here. Figure 2 In this context, neural network model 130 can be, but is not limited to, an autoregressive model.

[0050] for Figure 2 The usage process of the 3D occupancy grid prediction system 200 shown can be as follows: Figure 3 As shown. Figure 3As shown, the system comprises the following key processes during its usage phase: a) encoding input data into an index space (Occupancy-guided Encoder), b) using an index space autoregressive model, and c) decoding the predicted future frame index into the final Occupancy using a decoder. Processes a) and c) involve an Occupancy-guided Encoder and Decoder trained using an Occupancy-guided autoencoder, successfully establishing a bridge between the input data (including multi-view images, LiDAR point clouds, multi-view fisheye images, and radar point clouds) and the output data in the occupancy feature space. Based on this, the autoregressive model can more efficiently predict future frames (i.e., a future 3D occupancy grid) within the Occupancy index space. This two-stage approach makes it easier to achieve higher future frame prediction accuracy. In particular, the Occupancy-guided multimodal unified representation integrates multimodal data, laying a solid foundation for occupancy semantic prediction, which is a crucial component of the entire system. The specific implementations of these two stages will be described below.

[0051] 1) Occupancy-guided unified multimodal representation:

[0052] like Figure 4 As shown, for data inputs from different modalities such as images, LiDAR, and Radar, the system uses ocupancy guidance to align and tokenize data from different modalities within a unified latent code space. This guides the data from each modality to be represented in the occupancy feature space, achieving effective alignment within the feature space. Subsequently, the system searches for the dictionary feature in the codebook that is closest to the fused feature, and uses this dictionary feature to represent the fused feature. Then, the decoder reconstructs the current occupancy. This approach not only overcomes the limitations of existing technologies that are confined to a single modality or two-dimensional (2D) level, but also fully leverages the advantages of occupancy modality in 3D spatial characterization and fine-grained representation. Through occupancy guidance, multimodal data can be uniformly represented within the semantic feature space, laying a solid foundation for the next stage of autoregressive model training. Figure 4 In this process, after concatenating the features of the encoded modalities, a shared fusion encoder can be used to process the concatenated features so that features from different modalities can observe each other, thereby achieving better feature fusion. In some embodiments, from Figure 4 It can be seen from the above that... Figure 2 The training process of the system is shown. Figure 4 In the process, after obtaining the predicted future 3D occupancy grid, loss calculation can be performed between it and the 3D occupancy grid in the sample labels, and the parameters of each part of the system can be adjusted along the direction of loss minimization.

[0053] 2) Occupancy index: index space autoregression

[0054] like Figure 5 As shown, to achieve accurate prediction of spatial occupancy semantics, this embodiment, based on multimodal data alignment and tokenization (i.e., encoding features into index values), overcomes the limitations of existing shallow fusion of multimodal features using Concat and directly adopts an Occupancy index space autoregressive model. Specifically, for each index, previous indices are treated as historical information, thus achieving rolling prediction of the next index value. This method not only fully utilizes the powerful representation and prediction capabilities of existing pre-trained large model architectures but can also be easily extended to index encoding of other modal data, such as LiDAR, radar, text, and actions. Furthermore, this architecture lays a solid foundation for the subsequent construction of cross-modal basic large models.

[0055] The specific implementation process of the above system is described below.

[0056] For example, Figure 6 This diagram illustrates a flowchart of a 3D occupancy grid prediction method provided in an embodiment of this application. It is understood that this method can be executed by any device, equipment, platform, or device cluster with computing and processing capabilities. For example, the method can be executed by a 3D occupancy grid prediction device, which can be implemented by software and / or hardware, and can be configured in, but is not limited to, a terminal device or server; typically, it can be configured in a vehicle. For example, the terminal device can be, but is not limited to, a vehicle or a robot. For ease of description, the following description will focus on a terminal device as the execution subject. Figure 6 As shown, the 3D occupancy grid prediction method includes:

[0057] S601. Acquire multimodal data of the surrounding environment of the terminal device at the first moment.

[0058] In this embodiment, the terminal device can be configured with data acquisition devices for different modalities. For example, it can be configured with an image acquisition device (such as a camera) and a point cloud acquisition device (such as radar) to acquire image modal data and point cloud modal data. The data acquisition devices on the terminal device can acquire multimodal data of the terminal device's surrounding environment at various times. For example, the image acquisition device can acquire a pure visual image of the terminal device's surrounding environment at a first moment. The acquired pure visual image can include an image from at least one viewpoint. This pure visual image refers to an image that conveys information or emotion solely through visual elements such as color and shape, without relying on textual descriptions or other information. The point cloud acquisition device can acquire point cloud data of the terminal device's surrounding environment at a first moment. The data acquisition devices configured on the terminal device can acquire data in real time or periodically (e.g., every 1 second or every 2 seconds), depending on the actual situation, and are not limited here. For example, the first moment can be the latest moment or a historical moment, and is not limited here. In some embodiments, when the method is executed by the cloud side (such as a cloud server), after the terminal device collects multimodal data of its surrounding environment at a first moment, it can upload the multimodal data to the cloud side, so that the cloud side can obtain the multimodal data.

[0059] S602. Based on multimodal data, predict the 3D occupancy grid of the terminal device at the second moment, where the second moment is later than the first moment.

[0060] In this embodiment, after acquiring multimodal data, the terminal device can make predictions based on the multimodal data to predict the 3D occupancy grid of the terminal device at a second time point. The second time point is later than the first time point. For example, the terminal device can input the multimodal data into a pre-trained neural network model to process the multimodal data and predict the 3D occupancy grid of the terminal device at the second time point. In other words, the terminal device can predict future 3D occupancy grids using historical multimodal data. For example, the predicted 3D occupancy grid can take the form of... Figure 3 The occupancy shown.

[0061] In this way, by guiding the use of 3D occupancy grids, the prediction of future 3D occupancy grids is achieved using historical multimodal data. This significantly improves the generalization of future 3D occupancy grid prediction, avoids the limitation of relying solely on a single occupancy semantic modality, which cannot fully characterize the scene evolution process, and overcomes the problem of a significant drop in the performance of future 3D occupancy grid prediction due to limitations in historical 3D occupancy grid data and a single modality.

[0062] In some embodiments, the process of predicting the 3D occupancy grid of the terminal device at the second moment in S602 described above can be as follows: Figure 7 As shown. Please refer to [the original text]. Figure 7 In step S701, each modality of the multimodal data is encoded to obtain multiple encoded features. Each encoded feature is associated with one modality of the multimodal data. For example, this step can be performed as described above. Figure 1 The encoder implementation described above is detailed and will not be repeated here. In S702, multiple coded features are concatenated to obtain a concatenated feature. For example, this step can be performed using the aforementioned... Figure 1 The feature concatenation module 120 described above is used for implementation and will not be repeated here. In S703, predicted features are obtained based on the concatenated features. For example, this step can be implemented using the aforementioned... Figure 1 The neural network model 130 described above is implemented in detail and will not be repeated here. In S704, the predicted features are decoded to obtain the 3D occupancy grid of the terminal device at the second time step. For example, this step can be implemented through the aforementioned... Figure 1 The decoder implementation described above is detailed and will not be repeated here. This enables the prediction of future 3D grid occupancy using multimodal data.

[0063] In addition, Figure 7 In S703 shown, to improve prediction accuracy, feature alignment can be performed on the concatenated features to unify the features of different modalities into the same feature space. Specifically, as shown... Figure 8 As shown, in S801, combined with the codebook, the concatenated features are mapped to the index space to obtain historical index features. These historical index features contain at least one index value from the codebook, and each index value is associated with a feature vector obtained by flattening the concatenated features. For example, the concatenated features can first be flattened into at least one feature vector. For instance, if the dimensions of the concatenated features are B*H*W*D, the number of flattened feature vectors can be (B*H*W), and the length of each feature vector can be D. Then, feature vectors similar to each flattened feature vector are queried from the codebook, and the flattened feature vectors are represented using the index corresponding to the queried feature vector (i.e., the index value). For example, if the flattened feature vector is... In the codebook, with The index value corresponding to similar feature vectors is 1, and... If the index value corresponding to a similar feature vector is 2, then the historical index feature can be (1,2). This maps the concatenated features to the index space. In some embodiments, the form of the historical index feature can be similar to... Figure 3 The history_index shown is similar.

[0064] In S802, historical index features are processed using an autoregressive model to obtain future index features. For example, historical index features can be input into an autoregressive model for processing to obtain future index features. In some embodiments, the form of future index features can be similar to... Figure 3 The future_index shown is similar.

[0065] In S803, by combining the codebook, future index features are inversely mapped from the index space to the space containing the concatenated features to obtain the predicted features. Since the future index features are also expressed using index values ​​from the codebook, the codebook can be queried first to convert the index values ​​in the future index features into feature vectors; then, through inverse flattening, the obtained feature vectors are inversely mapped to the space containing the concatenated features to obtain the predicted features. In this way, by aligning multimodal features to a unified feature space, prediction accuracy can be improved.

[0066] The above is an introduction to the technical solution provided in this application. As can be seen from the description, the solution mainly includes two points: First, multimodal unified representation guided by occupancy; second, occupancy index autoregression. In the first point, the occupancy-guided approach effectively unifies multimodal features, significantly improving the model's generalization ability in future frame prediction. This method avoids the limitation of relying solely on a single occupancy semantic modality, which fails to fully characterize the scene evolution process. It also overcomes the problem of significantly reduced future frame occupancy prediction performance caused by traditional methods being limited by historical data and a single modality. In the second point, based on the successful encoding of features into the occupancy index space in the first point, the limitations of existing multimodal feature concat shallow fusion are abandoned, and occupancy index autoregression is directly adopted. This fully utilizes the powerful representation and prediction capabilities of existing pre-trained large model architectures, achieving high-fidelity prediction of future frames. In addition, the technical solution provided in this application can achieve accurate prediction of the semantic occupancy of future frames in autonomous driving scenarios with high fidelity by adopting multimodal unified representation guided by occupancy and autoregressive occupancy index space. This technology enables vehicles to make safe and effective decisions in advance, reduce accident risks, and thus improve road safety and traffic efficiency.

[0067] In addition, such as Figure 9As shown, the technical solution provided in this application, compared with existing traditional methods, not only eliminates the dependence on historical occupancy but also successfully achieves high-fidelity future frame prediction. Specifically, the Ours-13B model achieves a fidelity of up to 99%, while traditional methods only achieve 38.8%. Furthermore, as the size of the pre-trained model increases, the system can obtain further positive benefits, providing possibilities for future scalability exploration. In addition, Figure 10 This further demonstrates the accuracy of the technical solution provided in this application when predicting more future frame times. It can be seen that even when predicting the next 5 frames, the system still maintains a high intersection-over-union (IoU) accuracy, while the accuracy of traditional methods decreases significantly with the increase in the number of predicted frames. This result fully demonstrates the superior performance of the technical solution provided in this application. Figure 11 , Figure 12 and Figure 13 The occupancy prediction results at times T, T+1, and T+2 are presented respectively. It can be seen that the drivable area and surrounding vehicles were predicted relatively stably.

[0068] This analysis demonstrates that the technical solution provided in this application, by introducing occupancy-guided multimodal feature alignment and an autoregressive model, not only enhances the model's generalization ability in complex scenarios but also ensures stability in long-term predictions. This lays a solid technical foundation for high-precision environmental perception and decision-making in future autonomous driving systems and provides strong support for further improving the safety and reliability of autonomous driving technology.

[0069] It is understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. In addition, the various embodiments or technical features involved in the embodiments described above can be combined according to the actual situation, and the combined solution is still within the protection scope of this application.

[0070] Based on the methods in the above embodiments, this application also provides a 3D occupancy grid device.

[0071] For example, Figure 14 A schematic diagram of a 3D occupancy grid device provided in an embodiment of this application is shown. Figure 14As shown, the 3D occupancy grid device 1400 includes an acquisition module 1401 and a processing module 1402. The acquisition module 1401 is used to acquire multimodal data of the surrounding environment of the terminal device at a first moment. The processing module 1402 is used to predict the 3D occupancy grid of the terminal device at a second moment based on the multimodal data, wherein the second moment is later than the first moment.

[0072] In some embodiments, when the processing module 1402 predicts the 3D occupancy grid of the terminal device at a second time based on multimodal data, it is specifically used to: encode each modality of the multimodal data to obtain multiple encoded features, wherein one encoded feature is related to one modality of the multimodal data; concatenate the multiple encoded features to obtain a concatenated feature; obtain a predicted feature based on the concatenated feature; and decode the predicted feature to obtain the 3D occupancy grid.

[0073] In some embodiments, when the processing module 1402 obtains the predicted features based on the concatenated features, it is specifically used to: combine the codebook to map the concatenated features to the index space to obtain historical index features, wherein the historical index features contain at least one index value in the codebook, and an index value is related to a feature vector obtained by flattening the concatenated features; process the historical index features through an autoregressive model to obtain future index features; and combine the codebook to inversely map the future index features from the index space to the space where the concatenated features are located to obtain the predicted features.

[0074] In some embodiments, the multimodal data includes images and radar point clouds.

[0075] In some embodiments, the terminal device is a vehicle or a robot.

[0076] It should be understood that the above-described device is used to execute the methods in the above embodiments. The implementation principle and technical effect of the corresponding program modules in the device are similar to those described in the above methods. The working process of the device can be referred to the corresponding process in the above methods, and will not be repeated here.

[0077] Based on the methods described in the above embodiments, this application also provides an electronic device. For example... Figure 15 As shown, the electronic device 1500 includes a bus 1502, a processor 1504, a memory 1506, and a communication interface 1508. The processor 1504, the memory 1506, and the communication interface 1508 communicate with each other via the bus 1502. It should be understood that this application does not limit the number of processors and memories in the electronic device 1500. In some embodiments, the electronic device 1500 may be, but is not limited to, configured in a vehicle or robot.

[0078] Bus 1502 can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be divided into address buses, data buses, control buses, etc. For ease of representation, Figure 15 The bus 1504 may be represented by a single line, but this does not mean that there is only one bus or one type of bus. The bus 1504 may include a path for transmitting information between various components of the electronic device 1500 (e.g., memory 1506, processor 1504, communication interface 1508).

[0079] Processor 1504 may include any one or more processors such as a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0080] The memory 1506 may include volatile memory, such as random access memory (RAM). The processor 1504 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0081] The memory 1506 stores executable program code, and the processor 1504 executes the executable program code to implement the aforementioned functions. Figure 11 The memory 1506 stores the functions of each module to implement all or part of the steps of the method in the above embodiments. That is, the memory 1506 contains instructions for executing all or part of the steps in the method of the above embodiments.

[0082] Alternatively, the memory 1506 may store executable code, which the processor 1504 executes to implement the functions of the aforementioned 3D occupancy grid prediction device 1100, thereby implementing all or part of the steps in the methods of the above embodiments. That is, the memory 1506 may store instructions for performing all or part of the steps in the methods of the above embodiments.

[0083] The communication interface 1503 uses transceiver modules such as, but not limited to, network interface cards and transceivers to enable communication between the electronic device 1500 and other devices or communication networks.

[0084] Based on the methods described in the above embodiments, this application also provides another 3D occupancy grid prediction device. Please refer to... Figure 16 , Figure 16 This is a schematic diagram of another 3D occupancy grid prediction device provided in an embodiment of this application. Figure 16 As shown, the 3D occupancy grid prediction device 1600 includes one or more processors 1601 and interface circuitry 1602. Optionally, the 3D occupancy grid prediction device 1600 may also include a bus 1603. Wherein:

[0085] Processor 1601 can be an integrated circuit chip with signal processing capabilities. In implementation, each step of the above method can be completed through integrated logic circuits in the hardware of processor 1601 or through software instructions. Processor 1601 can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods and steps disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor.

[0086] The interface circuit 1602 can be used to send or receive data, instructions or information. The processor 1601 can use the data, instructions or other information received by the interface circuit 1602 to process the data, instructions or other information, and can send the processed information out through the interface circuit 1602.

[0087] Optionally, the 3D occupancy grid prediction device 1600 also includes a memory, which may include read-only memory and random access memory, and provides operating instructions and data to the processor. A portion of the memory may also include non-volatile random access memory (NVRAM).

[0088] Optionally, the memory stores executable software modules or data structures, and the processor can execute corresponding operations by calling the operation instructions stored in the memory (which may be stored in the operating system).

[0089] Optionally, the interface circuit 1602 can be used to output the execution results of the processor 1601.

[0090] It should be noted that the functions of the processor 1601 and the interface circuit 1602 can be implemented through hardware design, software design, or a combination of hardware and software; no restrictions are imposed here.

[0091] Based on the methods in the above embodiments, this application provides a computer-readable storage medium storing a computer program that, when executed on a processor, causes the processor to perform the methods in the above embodiments. Exemplarily, the computer-readable storage medium can be any available medium that an electronic device can store, or a data storage device such as a data center containing one or more available media. The available media can be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives).

[0092] Based on the methods in the above embodiments, this application provides a computer program product that, when run on a processor, causes the processor to execute the methods in the above embodiments.

[0093] Based on the methods described in the above embodiments, this application also provides an autonomous driving system. For example... Figure 17 As shown, the autonomous driving system may include modules for perception, prediction, planning, and control. The perception module can output information such as the vehicle's location (position, orientation, etc.), the status of other vehicles (position, orientation, speed, length, width, etc.), and road topology based on information received from sensors. The prediction module can output predictions of the future status of other vehicles based on the information obtained from the perception module, including the status of other vehicles and road topology. The planning module can determine the vehicle's lane selection based on environmental and other vehicle constraints obtained from perception and prediction, and plan the future driving path, speed, etc., outputting the planned driving route. The control module can output control information based on the planned driving route to control the vehicle to complete the driving task.

[0094] The 3D occupancy grid prediction scheme provided in this application is mainly applied to the perception and prediction modules. This method can simultaneously output predictions of surrounding objects and scene information for both the current and future moments (this information is contained within the 3D occupancy grid), thereby achieving more refined global prediction. This predictive capability enables the system to possess a more comprehensive and detailed global prediction ability, effectively improving the quality of the vehicle's driving route decision-making in complex road environments, ultimately achieving a safer and more efficient driving experience.

[0095] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0096] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to the processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and the storage medium can reside in an ASIC.

[0097] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted through the computer-readable storage medium. The computer instructions can be transmitted from one website, computer, cloud computing platform, or data center to another website, computer, cloud computing platform, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a cloud computing platform or data center that integrates one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state disk (SSD)).

[0098] It is understood that the various numerical designations used in the embodiments of this application are merely for descriptive convenience and are not intended to limit the scope of the embodiments of this application.

[0099] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit them. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the protection scope of the technical solutions of the embodiments of this application.

Claims

1. A 3D occupancy grid prediction method, characterized in that, The method includes: Acquire multimodal data of the surrounding environment of the terminal device at the first moment; Based on the multimodal data, the 3D occupancy grid of the terminal device at a second time point is predicted, wherein the second time point is later than the first time point.

2. The method according to claim 1, characterized in that, The step of predicting the 3D occupancy grid of the terminal device at the second time moment based on the multimodal data includes: Each modality in the multimodal data is encoded to obtain multiple encoded features, wherein one encoded feature is associated with one modality in the multimodal data; The multiple encoded features are concatenated to obtain the concatenated features; Based on the splicing features, the predicted features are obtained; The predicted features are decoded to obtain the 3D occupancy grid.

3. The method according to claim 2, characterized in that, The process of obtaining predicted features based on the spliced ​​features includes: By combining the codebook, the concatenated features are mapped to the index space to obtain historical index features, wherein the historical index features contain at least one index value in the codebook, and one of the index values ​​is associated with a feature vector obtained by flattening the concatenated features; The historical index features are processed using an autoregressive model to obtain future index features; By combining the codebook, the future index features are inversely mapped from the index space to the space where the concatenated features are located, so as to obtain the predicted features.

4. The method according to any one of claims 1-3, characterized in that, The multimodal data includes images and radar point clouds.

5. The method according to any one of claims 1-4, characterized in that, The terminal device is a vehicle or a robot.

6. A 3D occupancy grid prediction device, characterized in that, include: The acquisition module is used to acquire multimodal data of the surrounding environment of the terminal device at the first moment; The processing module is used to predict the 3D occupancy grid of the terminal device at a second time point based on the multimodal data, wherein the second time point is later than the first time point.

7. The apparatus according to claim 6, characterized in that, When the processing module predicts the 3D occupancy grid of the terminal device at the second time moment based on the multimodal data, it is specifically used for: Each modality in the multimodal data is encoded to obtain multiple encoded features, wherein one encoded feature is associated with one modality in the multimodal data; The multiple encoded features are concatenated to obtain the concatenated features; Based on the splicing features, the predicted features are obtained; The predicted features are decoded to obtain the 3D occupancy grid.

8. The apparatus according to claim 7, characterized in that, When the processing module obtains the predicted features based on the concatenated features, it is specifically used for: By combining the codebook, the concatenated features are mapped to the index space to obtain historical index features, wherein the historical index features contain at least one index value in the codebook, and one of the index values ​​is associated with a feature vector obtained by flattening the concatenated features; The historical index features are processed using an autoregressive model to obtain future index features; By combining the codebook, the future index features are inversely mapped from the index space to the space where the concatenated features are located, so as to obtain the predicted features.

9. The apparatus according to any one of claims 6-8, characterized in that, The multimodal data includes images and radar point clouds.

10. The apparatus according to any one of claims 6-9, characterized in that, The terminal device is a vehicle or a robot.

11. A 3D occupancy grid prediction device, characterized in that, include: At least one processor and interface; The at least one processor obtains program instructions through the interface; The at least one processor is used to execute the program instructions to implement the method as described in any one of claims 1-5.

12. An electronic device, characterized in that, include: At least one memory for storing programs; At least one processor for executing the program stored in the memory; When the program stored in the memory is executed, the processor is used to execute the method as described in any one of claims 1-5.

13. A vehicle, characterized in that, Including the electronic device as described in claim 12.

14. A computer-readable storage medium storing a computer program that, when run on a processor, causes the processor to perform the method as described in any one of claims 1-5.

15. A computer program product, characterized in that, When the computer program product is run on a processor, the processor causes the processor to perform the method as described in any one of claims 1-5.

Images