Occupancy network prediction method, apparatus and device, and storage medium and product

Abstract

Disclosed herein is an occupancy network prediction method, comprising: on the basis of current frame image data of a surrounding environment of a vehicle, generating a first voxel feature and a second voxel feature, wherein the generation mode of the first voxel feature is different from that of the second voxel feature; performing feature fusion on the first voxel feature and the second voxel feature on the basis of a feature distribution structure; and determining a target-scale fused voxel feature on the basis of a plurality of pieces of historical frame image data and a target voxel fusion feature, and on the basis of the target-scale fused voxel feature, performing occupancy network prediction, so as to obtain occupancy information of an object in the surrounding environment of the vehicle, wherein multi-view image data in a single-frame scenario is processed into the first voxel feature and the second voxel feature for scenario sensing.

Description

Occupancy network prediction method, device, equipment, storage medium and product Cross-reference to related applications

[0001] This application claims priority to Chinese Patent Application No. 202410999094.1, filed on July 24, 2024, the entire contents of which are incorporated herein by reference. TECHNICAL FIELD

[0002] The present disclosure relates to an occupancy network prediction method, device, equipment, storage medium and product. BACKGROUND

[0003] A vehicle needs to perceive the surrounding environment in real time during driving, for example, road structure, traffic participants, and obstacles, etc., to ensure the safety and smoothness of driving. At present, the common way for environment perception is point cloud processing based on lidar or stereo matching based on vision, but the lidar is difficult to capture detailed information, and the vision used for stereo matching is greatly affected by factors such as light and occlusion, resulting in low accuracy of perceiving the surrounding environment. For this reason, a single-frame occupancy network prediction method has emerged, but the single-frame occupancy network prediction method has very limited understanding ability of the scene, resulting in low accuracy of occupancy network prediction in complex environments. SUMMARY

[0004] The technical problem of low accuracy of occupancy network prediction in complex environments is solved by using one or more embodiments of the present disclosure.

[0005] The main purpose of the present disclosure is to provide an occupancy network prediction method, device, equipment, storage medium and product, aiming to solve the technical problem of low accuracy of occupancy network prediction in complex environments in the prior art.

[0006] To achieve the above-mentioned purpose, the present disclosure provides an occupancy network prediction method, which comprises:

[0007] generating a first voxel feature and a second voxel feature according to current frame image data of the surrounding environment of the vehicle, wherein the first voxel feature and the second voxel feature are generated in different ways;

[0008] determining a feature distribution structure between the first voxel feature and the second voxel feature;

[0009] performing feature fusion on the first voxel feature and the second voxel feature according to the feature distribution structure to obtain a target voxel fusion feature;

[0010] Obtaining a plurality of historical frame image data of the surrounding environment of the vehicle, determining a target scale fusion voxel feature according to the plurality of historical frame image data and the target voxel fusion feature, and predicting object occupancy rate information in the surrounding environment of the vehicle through an occupancy network according to the target scale fusion voxel feature.

[0011] In an embodiment, the generating the first voxel feature and the second voxel feature according to the current frame image data of the surrounding environment of the vehicle comprises:

[0012] Generating the first voxel feature according to the current frame image data of the surrounding environment of the vehicle;

[0013] Estimating the current frame image data to obtain depth estimation data; and

[0014] Generating the second voxel feature based on the current frame image data and the depth estimation data.

[0015] In an embodiment, the generating the first voxel feature and the second voxel feature according to the current frame image data of the surrounding environment of the vehicle comprises:

[0016] Obtaining intrinsic parameters and extrinsic parameters of an image acquisition device;

[0017] Determining a first mapping relationship from an image acquisition space to a visual perception space according to first conversion data from the image acquisition device to the vehicle, the intrinsic parameters and the extrinsic parameters;

[0018] Determining a second mapping relationship from the visual perception space to the image acquisition space according to the extrinsic parameters, second conversion data from the vehicle to a global coordinate system, third conversion data from the visual perception space to the vehicle coordinate system, and fourth conversion data from the global coordinate system to the vehicle coordinate system;

[0019] Performing feature extraction on the current frame image data to obtain image feature data;

[0020] Obtaining a target sampling point in the visual perception space and mapping the target sampling point to an image space;

[0021] Generating the first voxel feature based on a voxel container, the first mapping relationship, the image feature data, and the image space after the sampling point is mapped;

[0022] Generating the second voxel feature according to the second mapping relationship.

[0023] In an embodiment, the generating the second voxel feature according to the second mapping relationship comprises:

[0024] Performing characteristic detection on the image acquisition device to obtain device feature data;

[0025] determining depth estimation data of the current frame image data;

[0026] generating second voxel features according to the depth estimation data, the device feature data, the image feature data, and the second mapping relationship.

[0027] In an embodiment, the feature fusion of the first voxel features and the second voxel features according to the feature distribution structure to obtain target voxel fusion features comprises:

[0028] determining a voxel feature fusion strategy according to the feature distribution structure, wherein the voxel feature fusion strategy comprises a receptive field fusion strategy and a focus fusion strategy;

[0029] grouping voxel features to obtain first group voxel features and second group voxel features, wherein the voxel features comprise the first voxel features and the second voxel features;

[0030] splicing the first group voxel features based on the receptive field fusion strategy and a target dimension to obtain first spliced voxel features;

[0031] performing overlap region fusion on the first spliced voxel features through target convolution to obtain first fusion features;

[0032] compressing first voxel features in the second group voxel features in a height direction;

[0033] determining reconstructed voxel features according to the compressed second group voxel features;

[0034] splicing the reconstructed voxel features and second voxel features in the second group voxel features to obtain second spliced voxel features;

[0035] performing feature fusion on the second spliced voxel features to obtain second fusion features;

[0036] determining target voxel fusion features according to the first fusion features and the second fusion features.

[0037] In an embodiment, the determination of target scale fusion voxel features according to the plurality of historical frame image data and the target voxel fusion features comprises:

[0038] determining a plurality of historical image features according to a plurality of historical frame image data, and performing feature alignment on the plurality of historical image features, wherein the aligned plurality of historical image features at least comprises first frame historical image features and second frame historical image features;

[0039] determining a coordinate mapping from the first frame historical image features to the second frame historical image features;

[0040] acquire a dimension of the voxel feature under the current frame image data, and determine a sampling point of a voxel space corresponding to the current frame image data according to the dimension;

[0041] perform spatial conversion on the sampling point based on the coordinate mapping;

[0042] obtain a sampling voxel feature according to the converted sampling point;

[0043] determine a target scale fusion voxel feature according to the sampling voxel feature and the target voxel fusion feature.

[0044] In an embodiment, the determining of the target scale fusion voxel feature according to the sampling voxel feature and the target voxel fusion feature comprises:

[0045] determining a first frame offset sampling feature and a second frame offset sampling feature according to the sampling voxel feature;

[0046] splicing the first frame offset sampling feature and the second frame offset sampling feature to obtain a spliced offset sampling feature;

[0047] performing cascade fusion on the spliced offset sampling feature to obtain a cascade fusion feature;

[0048] performing scale fusion on the target voxel fusion feature and the cascade fusion feature to obtain a target scale fusion voxel feature.

[0049] In addition, to achieve the above-mentioned purpose, the disclosure further provides an occupancy network prediction device, which comprises:

[0050] a generation module configured to generate a first voxel feature and a second voxel feature according to current frame image data of a surrounding environment of a vehicle, wherein the first voxel feature and the second voxel feature are generated in different ways;

[0051] a determination module configured to determine a feature distribution structure between the first voxel feature and the second voxel feature;

[0052] a fusion module configured to perform feature fusion on the first voxel feature and the second voxel feature according to the feature distribution structure to obtain a target voxel fusion feature;

[0053] a prediction module configured to acquire a plurality of historical frame image data of the surrounding environment of the vehicle, determine a target scale fusion voxel feature according to a plurality of the historical multi-frame data and the target voxel fusion feature, and predict occupancy rate information of an object in the surrounding environment of the vehicle through an occupancy network according to the target scale fusion voxel feature.

[0054] In an embodiment, the generating module, when generating the first voxel feature and the second voxel feature according to the current frame image data of the surrounding environment of the vehicle, is specifically configured to:

[0055] generate the first voxel feature according to the current frame image data of the surrounding environment of the vehicle;

[0056] estimate the current frame image data to obtain depth estimation data; and

[0057] generate the second voxel feature based on the current frame image data and the depth estimation data.

[0058] In an embodiment, the generating module, when generating the first voxel feature and the second voxel feature according to the current frame image data of the surrounding environment of the vehicle, is specifically configured to:

[0059] obtain intrinsic parameters and extrinsic parameters of an image acquisition device;

[0060] determine a first mapping relationship from an image acquisition space to a visual perception space according to first conversion data from the image acquisition device to the vehicle, the intrinsic parameters and the extrinsic parameters;

[0061] determine a second mapping relationship from the visual perception space to the image acquisition space according to the extrinsic parameters, second conversion data from the vehicle to a global coordinate system, third conversion data from the visual perception space to the vehicle coordinate system, and fourth conversion data from the global coordinate system to the vehicle coordinate system;

[0062] extract features from the current frame image data to obtain image feature data;

[0063] obtain a target sampling point in the visual perception space and map the target sampling point to an image space;

[0064] generate the first voxel feature based on a voxel container, the first mapping relationship, the image feature data, and the image space after the sampling point is mapped;

[0065] generate the second voxel feature according to the second mapping relationship.

[0066] In an embodiment, the generating module, when generating the second voxel feature according to the second mapping relationship, is specifically configured to:

[0067] perform characteristic detection on the image acquisition device to obtain device feature data;

[0068] determine depth estimation data of the current frame image data;

[0069] generate the second voxel feature according to the depth estimation data, the device feature data, the image feature data, and the second mapping relationship.

[0070] In an embodiment, the fusing module is specifically configured to:

[0071] determine a voxel feature fusion strategy according to the feature distribution structure, wherein the voxel feature fusion strategy comprises a receptive field fusion strategy and a focus fusion strategy;

[0072] group the voxel features to obtain first group voxel features and second group voxel features, wherein the voxel features comprise the first voxel features and the second voxel features;

[0073] splicing the first group voxel features according to a target dimension based on the receptive field fusion strategy to obtain first spliced voxel features;

[0074] perform overlap region fusion on the first spliced voxel features through target convolution to obtain first fused features;

[0075] compress first voxel features in the second group voxel features in a height direction;

[0076] determine reconstructed voxel features according to the compressed second group voxel features;

[0077] splicing the reconstructed voxel features and second voxel features in the second group voxel features to obtain second spliced voxel features;

[0078] perform feature fusion on the second spliced voxel features to obtain second fused features;

[0079] determine target voxel fusion features according to the first fused features and the second fused features.

[0080] In an embodiment, the predicting module is specifically configured to:

[0081] determine a plurality of historical image features according to a plurality of historical frame image data, and perform feature alignment on the plurality of historical image features, wherein the aligned plurality of historical image features at least comprises first frame historical image features and second frame historical image features;

[0082] determine a coordinate mapping from the first frame historical image features to the second frame historical image features;

[0083] obtain a dimension of the voxel features under current frame image data, and determine sampling points of a voxel space corresponding to the current frame image data according to the dimension;

[0084] perform spatial conversion on the sampling points based on the coordinate mapping;

[0085] obtain a sampling voxel feature according to the converted sampling points;

[0086] determine a target scale fusion voxel feature according to the sampling voxel feature and the target voxel fusion feature.

[0087] In an embodiment, the prediction module, when determining the target scale fusion voxel feature according to the sampling voxel feature and the target voxel fusion feature, is specifically configured to:

[0088] determine a first frame offset sampling feature and a second frame offset sampling feature according to the sampling voxel feature;

[0089] concatenate the first frame offset sampling feature and the second frame offset sampling feature to obtain a concatenated offset sampling feature;

[0090] perform cascade fusion on the concatenated offset sampling feature to obtain a cascade fusion feature;

[0091] perform scale fusion on the target voxel fusion feature and the cascade fusion feature to obtain a target scale fusion voxel feature.

[0092] In addition, to achieve the above object, the disclosure further proposes an occupancy network prediction device, which comprises a memory, a processor and a computer program stored on the memory and executable on the processor, and the computer program is configured to implement the steps of the occupancy network prediction method as described above.

[0093] In addition, to achieve the above object, the disclosure further proposes a storage medium, which is a computer readable storage medium, and the storage medium stores a computer program, and the computer program is executed by a processor to implement the steps of the occupancy network prediction method as described above.

[0094] In addition, to achieve the above object, the disclosure further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor to implement the steps of the occupancy network prediction method as described above.

[0095] The one or more technical solutions provided by the present disclosure have at least the following technical effects: generating a first voxel feature and a second voxel feature according to current frame image data of a surrounding environment of a vehicle, the first voxel feature and the second voxel feature being generated in different manners; determining a feature distribution structure between the first voxel feature and the second voxel feature; performing feature fusion on the first voxel feature and the second voxel feature according to the feature distribution structure to obtain a target voxel fusion feature; determining a target scale fusion voxel feature according to a plurality of historical frame image data and the target voxel fusion feature, and performing occupancy network prediction according to the target scale fusion voxel feature; in this way, the multi-view image data in a single frame scene is processed into a scene-aware first voxel feature and a second voxel feature, and feature fusion is performed by using the analyzed feature distribution structure, and the limit relationship between multiple frames is fully considered, thereby effectively improving the accuracy of object occupancy information prediction in the surrounding environment of the vehicle, and enhancing the stability and robustness of environmental perception. BRIEF DESCRIPTION OF DRAWINGS

[0096] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure or the prior art, the drawings needed to be used in the embodiments or related description will be briefly introduced. Obviously, for those skilled in the art, other drawings can also be obtained without creative labor based on these drawings.

[0097] FIG. 1 is a flowchart of an occupancy network prediction method according to an embodiment of the present disclosure;

[0098] FIG. 2 is a flowchart of an occupancy network prediction method according to another embodiment of the present disclosure;

[0099] FIG. 3 is a schematic diagram of the overall process of generating voxel features in the occupancy network prediction method according to the second embodiment of the present disclosure;

[0100] FIG. 4 is a schematic diagram of the module structure of an occupancy network prediction device according to an embodiment of the present disclosure;

[0101] FIG. 5 is a schematic diagram of the device structure of a hardware running environment involved in the occupancy network prediction method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0102] It should be noted that the execution subject of the present embodiment can be a computing service device with data processing, network communication and program running functions, such as a tablet computer, a personal computer, a mobile phone, etc., or an electronic device capable of realizing the above functions, an occupancy network prediction controller, etc. The present embodiment and the following embodiments will be described below taking the occupancy network prediction controller as an example.

[0103] Based on this, the embodiment of the disclosure provides an occupancy network prediction method. Referring to FIG. 1, FIG. 1 is a flowchart of a first embodiment of the occupancy network prediction method of the disclosure.

[0104] In this embodiment, the occupancy network prediction method comprises steps S10-S40.

[0105] In step S10, first voxel features and second voxel features are generated according to image data of a surrounding environment of a vehicle.

[0106] The voxel is a short form of "volume pixel", a three-dimensional space containing voxel can be represented by a three-dimensional rendering or a polygonal surface of a given threshold contour, and is the smallest unit for three-dimensional imaging, scientific data and medical imaging, etc. in three-dimensional space segmentation.

[0107] It should be noted that the image data refers to multi-view image data captured by an image acquisition device arranged on a target object, and the target object can be a vehicle. At this time, the surrounding environment of the target environment can be a complex three-dimensional environment. In this complex three-dimensional environment, the accuracy of the single-frame occupancy network prediction is relatively low. In addition, in this embodiment, the first voxel features and the second voxel features are generated in different ways, and the voxel features are divided into first voxel features and second voxel features. The difference between the two lies in whether additional data is needed to assist in the generation process.

[0108] Step S10 can comprise:

[0109] Generating first voxel features according to current frame image data of a surrounding environment of a vehicle;

[0110] Estimating the current frame image data to obtain depth estimation data; and generating second voxel features based on the current frame image data and the depth estimation data.

[0111] In the disclosure, the first voxel features are faster to generate than the second voxel features, and can also be referred to as fast voxel features in the disclosure. The second voxel features are relatively slower to generate, and can also be referred to as slow voxel features in the disclosure.

[0112] In step S20, a feature distribution structure between the first voxel features and the second voxel features is determined.

[0113] It can be understood that the feature distribution structure refers to the distribution structure of the voxel features between the first voxel features and the second voxel features. The feature distribution structure can be a distribution structure with repeated space in geometric position. If the distribution structure is the repeated space in geometric position, it indicates that the fast voxel and the slow voxel will exhibit similar spatial characteristics.

[0114] Step S30, according to the feature distribution structure, the first voxel feature and the second voxel feature are fused to obtain the target voxel fusion feature.

[0115] It should be understood that the target voxel fusion feature refers to the feature after the fusion of the first voxel feature and the second voxel feature, and the first voxel feature and the second voxel feature involve two types of voxel feature fusion strategies, which are receptive field fusion strategy and receptive field fusion strategy.

[0116] Further, step S30 comprises: determining a voxel feature fusion strategy according to the feature distribution structure, wherein the voxel feature fusion strategy comprises a receptive field fusion strategy and a focus fusion strategy; grouping voxel features to obtain first group voxel features and second group voxel features, wherein the voxel features comprise the first voxel features and the second voxel features; based on the receptive field fusion strategy, using a preset merging manner and a target dimension to splice the first group voxel features to obtain first spliced voxel features; performing overlapping region fusion on the first spliced voxel features through target convolution to obtain first fusion features; compressing the first voxel features in the second group voxel features in the height direction; determining a reconstructed voxel feature according to the compressed second group voxel features; splicing the reconstructed voxel feature with the second voxel features in the second group voxel features to obtain second spliced voxel features; performing feature fusion on the second spliced voxel features to obtain second fusion features; and determining a target voxel fusion feature according to the first fusion features and the second fusion features.

[0117] It can be understood that after obtaining the first voxel feature and the second voxel feature, the voxel feature is obtained, the voxel feature is grouped into the first group voxel feature and the second group voxel feature, and the receptive field fusion strategy is used for feature fusion, and the first group voxel feature is spliced using a preset merging manner and a target dimension, the preset merging manner can be splicing, and the target dimension can be C cams dimension, which will remain D=C depth , so as to obtain a new feature, and the above method can ensure that the first voxel feature V fast and the second voxel feature V slow DxHxW in 3D space position, that is, Z, X, Y ensures the overlapping fusion of the spatial position. Finally, the target convolution is used to complete the fusion of the overlapping region, ensure the generation of redundant features, and reduce the calculation amount, and the target convolution can be a single layer 3D convolution.

[0118] It should be noted that, since the object has geometric information such as height, width, etc. in the real space, the first voxel feature V fastOnly simple geometric mapping and feature filling, the description of these geometric information is weaker than the second voxel feature V slow , considering this key problem, first the first voxel feature V fast According to the height direction compression, the compressed feature can be regarded as the feature distribution of voxel feature in the vertical direction, which can describe the location of the object, and then the reconstructed voxel feature is obtained by using the repeat operation, which can be represented as:

[0119]

[0120] Where, repeat(.) represents the copy operation of the data with dimension 1, D, H, W represents the voxel feature dimension.

[0121] It should be noted that through the above calculation, under the premise of not losing the real space description of the object, the rough description of the object in the height direction can be obtained, the voxel feature reconstruction is completed, and then the reconstructed voxel V fast And the second voxel feature V slow In the second group of voxel features, the slow voxel feature is enriched, and the accuracy of occupancy prediction is improved.

[0122] It should be understood that the current frame image data is set as t, the historical frame image data is set as t-1,...,t-k, k represents the number of historical frame image data used, and t and t-1, that is, the first group of voxel features adopts the receptive field fusion strategy, t-2,...,t-k, that is, the second group of voxel features adopts the receptive field fusion strategy, by grouping the feature fusion strategy between the current frame and the historical frame, it is ensured that the historical frame image data can obtain as much perception data as possible, and the current frame image data can have more generalization performance for the scene, and the final target voxel fusion feature can be represented as:

[0123]

[0124] Where, Indicates the target voxel fusion feature, using F r Represents the first fusion feature, F f Indicates the second fusion feature.

[0125] Step S40, obtaining a plurality of historical frame image data of the surrounding environment of the vehicle, determining a target scale fusion voxel feature according to the plurality of historical frame image data and the target voxel fusion feature, and predicting the occupancy rate information of the object in the surrounding environment of the vehicle through the occupancy network.

[0126] The target scale fused voxel feature refers to a feature determined based on a plurality of features corresponding to the historical frame image data and the target voxel fused feature.

[0127] The occupancy network for predicting the object occupancy rate information is a deep learning model for three-dimensional scene reconstruction and three-dimensional object modeling, and its main function is to predict whether each volume element (voxel) in the space is occupied by an object.

[0128] It should be noted that, in order to enrich the voxel feature and fully consider the cascade relationship between multiple frame image data, the target scale fused voxel feature is determined in combination with a plurality of historical frame image data, and then the target scale fused voxel feature is used to predict the object occupancy rate information in the surrounding environment of the vehicle through the occupancy network, thereby effectively improving the accuracy of object occupancy rate information prediction.

[0129] Further, the step of determining the target scale fused voxel feature according to the plurality of historical frame image data and the target voxel fused feature comprises: determining a plurality of historical image features according to the plurality of historical frame image data, and aligning the plurality of historical image features, wherein the aligned plurality of historical image features at least includes a first frame historical image feature and a second frame historical image feature; determining a coordinate mapping from the first frame historical image feature to the second frame historical image feature; obtaining a dimension of a voxel feature under the current frame image data, and determining a sampling point of a voxel space corresponding to the current frame image data according to the dimension; performing spatial conversion on the sampling point based on the coordinate mapping; obtaining a sampling voxel feature based on the converted sampling point according to a grid sampling strategy; and determining the target scale fused voxel feature according to the sampling voxel feature and the target voxel fused feature.

[0130] It should be understood that, in order to ensure the correlation between different frames is enhanced and the perception performance of the voxel feature used for occupancy prediction is continuously improved through cascading, after obtaining a plurality of historical image features, the plurality of historical image features need to be aligned, and then the aligned plurality of historical image features are cascaded and fused, taking the first frame historical image feature and the second frame historical image feature as an example, wherein the first frame historical image feature can also be referred to as a t frame image feature, and the second frame historical image feature can also be referred to as a t-1 frame image feature.

[0131] Specifically, determining the coordinate mapping from the first frame historical image feature to the second frame historical image feature can include: setting a mapping parameter of a sensing device; and calculating the coordinate mapping from the first frame to the second frame according to the mapping parameter, wherein the mapping parameter of the sensing device includes but is not limited to first conversion data of an image acquisition device to a vehicle coordinate system second conversion data of the vehicle coordinate system to a global coordinate system by using the first conversion data and the second conversion data to obtain a coordinate mapping from the t-1 frame image feature to the t frame image feature, specifically as follows:

[0132]

[0133] wherein M t-1→t represents the coordinate mapping from the t-1 frame image feature to the t frame image feature, represents the first conversion data, represents the second conversion data.

[0134] It can be understood that after obtaining the coordinate mapping from the t-1 frame image feature to the t frame image feature, it is further necessary to determine a sampling point of the voxel space corresponding to the current frame image data, and the sampling point can be a 3D (3 Dimensions) point.

[0135] It should be noted that after obtaining the sampling point of the voxel space corresponding to the current frame image data, it is necessary to convert the sampling point to the visual perception space by means of the coordinate mapping, which can be specifically defined as and then sample the t-1 frame image feature to the t frame image feature, specifically as follows:

[0136]

[0137] wherein, represents sampling the t-1 frame image feature to the t frame image feature, represents converting the sampling point to the visual perception space, M t-1→t represents the coordinate mapping from the t-1 frame image feature to the t frame image feature.

[0138] It should be understood that after obtaining the converted sampling point, the sampling voxel feature is obtained by means of the grid sampling strategy, specifically as follows:

[0139]

[0140] wherein V t-1→t represents the sampling voxel feature, represents sampling the t-1 frame image feature to the t frame image feature, f gs represents the grid sampling strategy.

[0141] Further, the step of determining the target scale fusion voxel feature according to the sampling voxel feature and the target voxel fusion feature comprises: determining a first frame offset sampling feature and a second frame offset sampling feature according to the sampling voxel feature; splicing the first frame offset sampling feature and the second frame offset sampling feature to obtain a spliced offset sampling feature; performing cascade fusion on the spliced offset sampling feature based on a target convolution to obtain a cascade fusion feature; and performing scale fusion on the target voxel fusion feature and the cascade fusion feature by using a pyramid model of an improved convolution module to obtain the target scale fusion voxel feature.

[0142] It can be understood that after obtaining the sampling voxel feature of the t-1 frame image feature at the t time, the interference of redundant features is reduced, and the perception feature is ensured to be more focused on the scene object. In order to ensure the efficiency of calculation, the first frame offset sampling feature and the second frame offset sampling feature are spliced and then cascade fusion is realized by using a target convolution. The target convolution can be a single-layer 3D convolution.

[0143] It should be understood that, in order to further improve the perception ability of different scale objects, a multi-scale fusion manner is adopted for scale fusion, and a pyramid model of an improved convolution module is used in the scale fusion process. Specifically, the 2D (2 Dimensions) convolution module of the original pyramid model is improved to a 3D convolution module to adapt to the data change of the model. After the fusion is completed, the target scale fusion voxel feature is obtained, and the target scale fusion voxel feature is used for final occupancy network prediction.

[0144] The embodiment generates a first voxel feature and a second voxel feature according to current frame image data of the surrounding environment of the vehicle; determines a feature distribution structure between the first voxel feature and the second voxel feature; performs feature fusion on the first voxel feature and the second voxel feature according to the feature distribution structure to obtain a target voxel fusion feature; determines a target scale fusion voxel feature according to a plurality of historical frame image data and the target voxel fusion feature, and performs occupancy network prediction according to the target scale fusion voxel feature; and in the above manner, the multi-view image data in a single frame scene is processed into scene perception first voxel features and second voxel features, and the feature fusion is performed by using the analyzed feature distribution structure, and the limit relationship between multiple frames is fully considered, so that the accuracy of object occupancy rate information prediction in the surrounding environment of the vehicle can be effectively improved, and the stability and robustness of environmental perception can be enhanced.

[0145] Based on the first embodiment of the present disclosure, in the second embodiment of the present disclosure, the same or similar contents as the above-mentioned first embodiment can be referred to the above introduction, and will not be described in detail. On this basis, please refer to FIG. 2, the step S10 comprises steps S101-S107:

[0146] Step S101, obtain the intrinsic parameters and extrinsic parameters of the image acquisition device.

[0147] It should be noted that the image acquisition device can be a high-definition camera arranged on the target object, and the image data of multiple views can be unified to the visual perception space through mapping conversion, which is more conducive to the automatic driving perception task in the current scene.

[0148] The intrinsic parameters of the image acquisition device are used to represent the fixed parameters of the image acquisition device, which can also be referred to as first parameters, and the intrinsic parameters of the image acquisition device include but are not limited to focal length, pixel size, coordinate axis inclination and distortion parameters, etc. The extrinsic parameters of the image acquisition device are used to represent the parameters used by the image acquisition device in the process of acquiring image data, which are variable parameters, also referred to as second parameters, including but not limited to rotation matrix, translation vector, etc.

[0149] Step S102, according to the first conversion data of the image acquisition device to the vehicle coordinate system, the intrinsic parameters and the extrinsic parameters, determine the first mapping relationship of the image acquisition to the visual perception space.

[0150] It should be understood that after obtaining the intrinsic parameters and the extrinsic parameters, the coordinate conversion is performed in combination with the first conversion data of the image acquisition device vehicle coordinate system, the mutual mapping of the image acquisition device space and the visual perception space is completed, and the first mapping relationship of the image acquisition space to the visual perception space is determined. Specifically:

[0151]

[0152] Wherein, P BEV represents the visual perception space, P cams represents the image acquisition space, E represents the extrinsic parameters, I -1 represents the inverse matrix of the intrinsic parameters.

[0153] It should be noted that the first mapping relationship of the image acquisition space to the visual perception space is established by the above formula, so as to complete the sampling and alignment between the visual perception features and the image features.

[0154] Step S103, according to the extrinsic parameters, the second conversion data of the vehicle coordinate system to the global coordinate system, the third conversion data of the visual perception space to the vehicle coordinate system and the fourth conversion data of the global coordinate system to the vehicle coordinate system, determine the second mapping relationship of the visual perception space to the image acquisition space.

[0155] It can be understood that the global coordinate data in the global coordinate system includes but is not limited to inertial navigation data, GPS positioning data, and navigation data, etc. After obtaining the external parameter, the second conversion data from the vehicle coordinate system to the global coordinate system, the third conversion data from the visual perception space to the vehicle coordinate system, and the fourth conversion data from the global coordinate system to the vehicle coordinate system are combined to realize the conversion from the visual perception space to the image acquisition space, so as to determine the mapping relationship from the visual perception space to the image acquisition space, specifically:

[0156]

[0157] wherein, P cams represents the image acquisition space, P BEV represents the visual perception space, and E represents the external parameter, represents the second conversion data, represents the third conversion data, represents the fourth conversion data.

[0158] It should be noted that the information transmission of different representation spaces is completed by the intermediate values of the vehicle coordinate system and the global coordinate system through the above formula. Through the above, the geometric information transmission under the calibration data is completed by means of different coordinate mapping methods.

[0159] In step S104, feature extraction is performed on the current frame image data to obtain image feature data.

[0160] It should be understood that in order to facilitate the generation of the first voxel feature and the second voxel feature, the features of the image data need to be extracted, i.e., image feature data.

[0161] In step S105, a target sampling point in the visual perception space is obtained, and the target sampling point is mapped to the image space.

[0162] It should be noted that in the present embodiment, whether the generated voxel feature needs to be assisted by additional data to define the first voxel feature and the second voxel feature. If the generated voxel feature is completed only by coordinate conversion and feature mapping between calibration parameters, it is called the generation of the first voxel feature. The first voxel feature can retain the corresponding relationship of the original image feature information, and can obtain relatively accurate scene geometric perception by means of calibration parameters. If the generated voxel feature needs to be assisted by additional data, and then assisted by calibration parameters to complete the calibration process, it is called the generation of the second voxel feature.

[0163] It should be understood that based on the above first voxel feature and second voxel feature, the target sampling point is used to generate the first voxel feature. After obtaining the target sampling point in the visual perception space, the target function is used to map the target sampling point to the image space. The target function can be <·>.

[0164] In step S106, the first voxel feature is generated based on the voxel container, the first mapping relationship, the image feature data, and the image space after mapping the sampling point.

[0165] The voxel container is a division manner of a three-dimensional space, which divides the space into many small cubic units, and these units are called voxels.

[0166] It can be understood that after determining the image space after mapping the sampling point, the first voxel feature is generated according to the mapping relationship from the camera space to the visual perception space, the image feature data, and the voxel container.

[0167]

[0168] wherein V fast represents the first voxel feature, represents the voxel container, and <·> represents the objective function, f cams represents the image feature data, and Eq(1) represents the first mapping relationship from the image acquisition space to the visual perception space.

[0169] It should be noted that after mapping the target sampling point to the image space using the objective function, the target sampling point is aligned with the voxel container , and the feature is filled to generate the first voxel feature.

[0170] In step S107, the second voxel feature is generated according to the second mapping relationship.

[0171] It can be understood that after obtaining the second mapping relationship from the visual perception space to the image acquisition space, the second voxel feature needs to be generated by combining the corresponding image feature of the image feature data.

[0172] Further, in step S107, the device feature data of the image acquisition device is obtained by performing characteristic detection, the depth estimation data of the image data is determined, and the second voxel feature is generated according to the depth estimation data, the device feature data, the image feature data, and the mapping relationship from the visual perception space to the camera space by using the shift-multiply strategy.

[0173] The device feature parameters of the image acquisition device can include the size of the light-sensitive element, the lens aperture, the focal length, the sensitivity, and the like; and the depth estimation data of the current frame image data refers to the distance information of each pixel point of the current frame image data to the image acquisition device.

[0174] It should be understood that in order to effectively improve the generation of the second voxel feature, the device feature data needs to be used, at this time the characteristic detection is performed on the image acquisition device, and then the more determined voxel feature, that is, the second voxel feature, is generated according to the depth estimation data, the device feature data, the image feature data and the mapping relationship from the visual perception space to the camera space in a way of shift multiplication strategy.

[0175] Referring to FIG. 3, FIG. 3 is a schematic diagram of the overall flow of generating the voxel feature, specifically, it is divided into two branches, wherein the first branch is used to generate the first voxel feature, that is, the fast voxel feature shown in the figure, and the second branch is used to generate the second voxel feature, that is, the slow voxel feature shown in the figure. After obtaining the target sampling point in the visual perception space and mapping the target sampling point to the image space, the voxel container According to the first mapping relationship and the image feature data, the first voxel feature is generated and output. Similarly, after detecting the device feature data of the image acquisition device, the depth estimation data of the image data is further determined, then the intermediate voxel feature is obtained by using the shift multiplication strategy for calculation, and then the second voxel feature is obtained by using the BEVPool operation to process the intermediate voxel feature, and the second voxel feature is output.

[0176] In this embodiment, the intrinsic parameters and the extrinsic parameters of the image acquisition device are obtained, the first mapping relationship from the image acquisition space to the visual perception space is determined according to the first conversion data from the image acquisition device to the vehicle coordinate system, the intrinsic parameters and the extrinsic parameters, the second mapping relationship from the visual perception space to the image acquisition space is determined according to the extrinsic parameters, the second conversion data from the vehicle coordinate system to the global coordinate system, the third conversion data from the visual perception space to the vehicle coordinate system and the fourth conversion data from the global coordinate system to the vehicle coordinate system, the image feature data is obtained by performing feature extraction on the current frame image data, the target sampling point in the visual perception space is obtained and mapped to the image space, the first voxel feature is generated based on the voxel container, the first mapping relationship, the image feature data and the image space after the sampling point is mapped, the second voxel feature is generated according to the second mapping relationship, and in the above manner, the first mapping relationship between the visual perception space and the image acquisition space is determined by using the intrinsic parameters and the extrinsic parameters of the image acquisition device and the multi-dimensional conversion data, then the target sampling point in the visual perception space is mapped to the image space, and the first voxel feature and the second voxel feature are respectively generated by using the voxel container and the depth estimation data of the image data, so as to effectively improve the accuracy of generating the first voxel feature and the second voxel feature.

[0177] The present disclosure also provides an occupancy network prediction device, please refer to FIG. 4, the occupancy network prediction device comprises:

[0178] The generating module 10 is configured to generate first voxel features and second voxel features according to current frame image data of a surrounding environment of a vehicle, wherein the first voxel features and the second voxel features are generated in different manners;

[0179] The determining module 20 is configured to determine a feature distribution structure between the first voxel features and the second voxel features;

[0180] The fusion module 30 is configured to perform feature fusion on the first voxel features and the second voxel features according to the feature distribution structure to obtain target voxel fusion features;

[0181] The prediction module 40 is configured to acquire a plurality of historical frame image data of the surrounding environment of the vehicle, determine target scale fusion voxel features according to the plurality of historical multi-frame data and the target voxel fusion features, and predict object occupancy rate information in the surrounding environment of the vehicle through an occupancy network according to the target scale fusion voxel features.

[0182] In an embodiment, when the generating module 10 generates the first voxel features and the second voxel features according to the current frame image data of the surrounding environment of the vehicle, the generating module 10 is specifically configured to:

[0183] generate the first voxel features according to the current frame image data of the surrounding environment of the vehicle;

[0184] estimate the current frame image data to obtain depth estimation data; and

[0185] generate the second voxel features based on the current frame image data and the depth estimation data.

[0186] In an embodiment, when the generating module 10 generates the first voxel features and the second voxel features according to the current frame image data of the surrounding environment of the vehicle, the generating module 10 is specifically configured to:

[0187] acquire intrinsic parameters and extrinsic parameters of an image acquisition device;

[0188] determine a first mapping relationship from an image acquisition space to a visual perception space according to first conversion data from the image acquisition device to the vehicle, the intrinsic parameters and the extrinsic parameters;

[0189] determine a second mapping relationship from the visual perception space to the image acquisition space according to the extrinsic parameters, second conversion data from the vehicle to a global coordinate system, third conversion data from the visual perception space to a vehicle coordinate system, and fourth conversion data from the global coordinate system to the vehicle coordinate system;

[0190] perform feature extraction on the current frame image data to obtain image feature data;

[0191] acquire a target sampling point of a visual perception space, and map the target sampling point to an image space;

[0192] generate a first voxel feature based on the voxel container, the first mapping relationship, the image feature data, and the image space to which the sampling point is mapped;

[0193] generate a second voxel feature according to the second mapping relationship.

[0194] In an embodiment, the generating module 10, when generating the second voxel feature according to the second mapping relationship, is specifically configured to:

[0195] perform characteristic detection on the image acquisition device to obtain device feature data;

[0196] determine depth estimation data of the current frame image data;

[0197] generate a second voxel feature according to the depth estimation data, the device feature data, the image feature data, and the second mapping relationship.

[0198] In an embodiment, the fusion module 30, when performing feature fusion on the first voxel feature and the second voxel feature according to the feature distribution structure to obtain a target voxel fusion feature, is specifically configured to:

[0199] determine a voxel feature fusion strategy according to the feature distribution structure, wherein the voxel feature fusion strategy includes a receptive field fusion strategy and a focus fusion strategy;

[0200] group voxel features to obtain a first group of voxel features and a second group of voxel features, wherein the voxel features include the first voxel feature and the second voxel feature;

[0201] splice the first group of voxel features according to a target dimension based on the receptive field fusion strategy to obtain a first spliced voxel feature;

[0202] perform overlap region fusion on the first spliced voxel feature through target convolution to obtain a first fusion feature;

[0203] compress first voxel features in the second group of voxel features in a height direction;

[0204] determine a reconstructed voxel feature according to the compressed second group of voxel features;

[0205] splice the reconstructed voxel feature and second voxel features in the second group of voxel features to obtain a second spliced voxel feature;

[0206] perform feature fusion on the second spliced voxel feature to obtain a second fusion feature;

[0207] determine a target voxel fusion feature according to the first fusion feature and the second fusion feature.

[0208] In an embodiment, the prediction module 40 is specifically configured to:

[0209] determine a plurality of historical image features according to a plurality of historical frame image data, and perform feature alignment on the plurality of historical image features, wherein the plurality of aligned historical image features at least include a first frame historical image feature and a second frame historical image feature;

[0210] determine a coordinate mapping from the first frame historical image feature to the second frame historical image feature;

[0211] obtain a dimension of the voxel feature under the current frame image data, and determine a sampling point of a voxel space corresponding to the current frame image data according to the dimension;

[0212] perform spatial conversion on the sampling point based on the coordinate mapping;

[0213] obtain a sampling voxel feature according to the converted sampling point;

[0214] determine a target scale fusion voxel feature according to the sampling voxel feature and the target voxel fusion feature.

[0215] In an embodiment, the prediction module 40 is specifically configured to:

[0216] determine a first frame offset sampling feature and a second frame offset sampling feature according to the sampling voxel feature;

[0217] splice the first frame offset sampling feature and the second frame offset sampling feature to obtain a spliced offset sampling feature;

[0218] perform cascade fusion on the spliced offset sampling feature to obtain a cascade fusion feature;

[0219] perform scale fusion on the target voxel fusion feature and the cascade fusion feature to obtain a target scale fusion voxel feature.

[0220] The embodiment generates first voxel features and second voxel features according to current frame image data of the surrounding environment of the vehicle; determines a feature distribution structure between the first voxel features and the second voxel features; performs feature fusion on the first voxel features and the second voxel features according to the feature distribution structure to obtain target voxel fusion features; determines target scale fusion voxel features according to a plurality of historical frame image data and the target voxel fusion features, and performs occupancy network prediction according to the target scale fusion voxel features; in the above manner, the multi-view image data in a single frame scene is processed into scene-aware first voxel features and second voxel features, and feature fusion is performed by using the analyzed feature distribution structure, and the limit relationship between multiple frames is fully considered, thereby effectively improving the accuracy of object occupancy information prediction in the surrounding environment of the vehicle, and enhancing the stability and robustness of environmental perception.

[0221] The occupancy network prediction device provided by the present disclosure adopts the occupancy network prediction method in the above embodiment, and can solve the technical problem of low accuracy of occupancy network prediction in a complex environment in the prior art. Compared with the prior art, the occupancy network prediction device provided by the present disclosure has the same beneficial effects as the occupancy network prediction method provided by the above embodiment, and other technical features in the occupancy network prediction device are the same as the features disclosed in the above embodiment method, which will not be repeated here.

[0222] The present disclosure provides an occupancy network prediction device, which comprises at least one processor and a memory connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the occupancy network prediction method in the above embodiment one.

[0223] Reference is made to FIG. 5, which shows a structural schematic diagram of an occupancy network prediction device suitable for implementing the embodiments of the present disclosure. The occupancy network prediction device in the embodiments of the present disclosure can include but is not limited to mobile terminals such as mobile phones, notebook computers, digital broadcast receivers, PDAs (Personal Digital Assistant), PADs (Portable Application Description), PMPs (Portable Media Player), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and the like, and fixed terminals such as digital TVs, desktop computers, and the like. The occupancy network prediction device shown in FIG. 5 is only an example, and should not bring any limitation to the functions and use range of the embodiments of the present disclosure.

[0224] As shown in FIG. 5, the occupancy network prediction device can include a processing apparatus 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read only memory (ROM) 1002 or a program loaded from a storage apparatus 1003 into a random access memory (RAM) 1004. In the RAM 1004, various programs and data required for the operation of the occupancy network prediction device are also stored. The processing apparatus 1001, the ROM 1002, and the RAM 1004 are connected to each other through a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Generally, the following systems can be connected to the I / O interface 1006: an input apparatus 1007 including, for example, a touch screen, a touch pad, a keyboard, a mouse, an image sensor, a microphone, an accelerometer, a gyroscope, etc.; an output apparatus 1008 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; the storage apparatus 1003 including, for example, a magnetic tape, a hard disk, etc.; and a communication apparatus 1009. The communication apparatus 1009 can allow the occupancy network prediction device to communicate wirelessly or by wire with other devices to exchange data. Although the occupancy network prediction device having various systems is shown in the drawing, it should be understood that all of the shown systems are not required to be implemented or possessed. More or less systems can be alternatively implemented or possessed.

[0225] In particular, the processes described above with reference to the flowcharts can be implemented as a computer software program according to embodiments of the present disclosure. For example, embodiments of the present disclosure include a computer program product comprising a computer program carried on a computer readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network by a communication apparatus, or installed from the storage apparatus 1003, or installed from the ROM 1002. When the computer program is executed by the processing apparatus 1001, the above-mentioned functions defined in the methods of embodiments of the present disclosure are performed.

[0226] The occupancy network prediction device provided by the present disclosure adopts the occupancy network prediction method in the above-mentioned embodiments, and can solve the technical problem of low accuracy of occupancy network prediction in a complex environment in the prior art. Compared with the prior art, the occupancy network prediction device provided by the present disclosure has the same beneficial effects as the occupancy network prediction method provided by the above-mentioned embodiments, and other technical features in the occupancy network prediction device are the same as the features disclosed in the previous embodiment method, which will not be repeated here.

[0227] It should be understood that various aspects disclosed herein can be implemented in hardware, software, firmware, or combinations thereof, to achieve the various aspects. In the description above, specific terminology is used to describe particular features, configurations, materials or characteristics. But such terminology is used only in a descriptive sense and not for purposes of limitation.

[0228] The above description is merely that of a specific implementation of the present disclosure, but the scope of protection of the present disclosure is not limited thereto. Any person skilled in the art can easily think of changes or replacements within the technical scope disclosed by the present disclosure, and all such changes or replacements should be encompassed within the scope of protection of the present disclosure. Therefore, the scope of protection of the present disclosure should be subject to the scope of protection of the claims.

[0229] The present disclosure provides a computer-readable storage medium having stored thereon computer-readable program instructions (i.e., a computer program) for performing the occupancy network prediction method in the above-described embodiments.

[0230] The computer-readable storage medium provided by the present disclosure may, for example, be a U disk, but is not limited to an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, system, or device, or any combination thereof. More specific examples of the computer-readable storage medium can include, but are not limited to, an electrical connection having one or more conductive wires, a portable computer diskette, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM) or flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present embodiment, the computer-readable storage medium can be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium can be transmitted in any suitable medium, including but not limited to electrical wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.

[0231] The above-described computer-readable storage medium can be included in the occupancy network prediction device; or can exist separately without being assembled into the occupancy network prediction device.

[0232] Computer program code for carrying out operations of the present disclosure can be written in any combination of one or more programming languages, including an object oriented programming language such as Java, Smalltalk, C++ or the like, and conventional procedural programming languages, such as the "C" programming language or similar programming languages. The program code can execute entirely on the user's computer, partly on the user's computer, as a stand-alone software package, partly on the user's computer and partly on a remote computer or entirely on the remote computer or server. In the latter scenario, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or the connection can be made to an external computer (for example, through the Internet using an Internet Service Provider).

[0233] The flow diagrams and the block diagrams in the drawings are illustrations of architectures, functionalities, and operations of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flow diagrams or block diagrams can represent a module, a segment, or a portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that in some alternative implementations, the functions noted in the blocks can occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently or the blocks may

[0234] The modules involved in the embodiments of the present disclosure can be implemented in the form of software or in the form of hardware. In some cases, the name of the module does not constitute a limitation on the module itself.

[0235] The readable storage medium provided by the present disclosure is a computer readable storage medium, which stores computer readable program instructions (i.e., a computer program) for executing the above-mentioned occupancy network prediction method, and can solve the technical problem of low accuracy of the prior art in predicting the occupancy network in a complex environment. Compared with the prior art, the computer readable storage medium provided by the present disclosure has the same beneficial effects as the occupancy network prediction method provided by the above-mentioned embodiments, and will not be described here.

[0236] The present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the occupancy network prediction method as described above.

[0237] The computer program product provided by the present disclosure can solve the technical problem of low accuracy of the prior art in predicting the occupancy network in a complex environment. Compared with the prior art, the beneficial effects of the computer program product provided by the present disclosure are the same as those of the occupancy network prediction method provided by the above-mentioned embodiments, and are not described here.

[0238] The above only describes some embodiments of the present disclosure, and does not limit the patent scope of the present disclosure. Any equivalent structural transformation, direct / indirect application in other related technical fields, or the like made by using the contents of the present disclosure and the drawings within the technical concept of the present disclosure are included in the patent protection scope of the present disclosure.

Claims

1. An occupancy network prediction method, comprising: generating first voxel features and second voxel features from current frame image data of a surrounding environment of a vehicle, wherein the first voxel features and the second voxel features are generated in different ways; determining a feature distribution structure between the first voxel features and the second voxel features; performing feature fusion on the first voxel features and the second voxel features according to the feature distribution structure to obtain target voxel fusion features; and obtaining a plurality of historical frame image data of the surrounding environment of the vehicle, determining target scale fusion voxel features from the plurality of historical frame image data and the target voxel fusion features, and predicting occupancy rate information of objects in the surrounding environment of the vehicle through an occupancy network according to the target scale fusion voxel features.

2. The method of claim 1, wherein, The generating of the first voxel features and the second voxel features from the current frame image data of the surrounding environment of the vehicle comprises: generating the first voxel features from the current frame image data of the surrounding environment of the vehicle; performing estimation on the current frame image data to obtain depth estimation data; and generating the second voxel features based on the current frame image data and the depth estimation data.

3. The method of claim 1, wherein, The generating of the first voxel features and the second voxel features from the current frame image data of the surrounding environment of the vehicle comprises: obtaining intrinsic parameters and extrinsic parameters of an image acquisition device; determining a first mapping relationship from an image acquisition space to a visual perception space according to first conversion data from the image acquisition device to a vehicle coordinate system, the intrinsic parameters and the extrinsic parameters; determining a second mapping relationship from the visual perception space to the image acquisition space according to the extrinsic parameters, second conversion data from the vehicle coordinate system to a global coordinate system, third conversion data from the visual perception space to the vehicle coordinate system, and fourth conversion data from the global coordinate system to the vehicle coordinate system; performing feature extraction on the current frame image data to obtain image feature data; obtaining target sampling points in the visual perception space and mapping the target sampling points to an image space; generating the first voxel features based on a voxel container, the first mapping relationship, the image feature data, and the image space with the sampling points mapped thereon; and generating the second voxel features according to the second mapping relationship.

4. The method of claim 3, wherein, The generating of the second voxel features according to the second mapping relationship comprises: performing characteristic detection on the image acquisition device to obtain device feature data; determining depth estimation data of the current frame image data; and generating the second voxel features according to the depth estimation data, the device feature data, the image feature data, and the second mapping relationship.

5. The method of claim 1, wherein, The performing of the feature fusion on the first voxel features and the second voxel features according to the feature distribution structure to obtain the target voxel fusion features comprises: determining a voxel feature fusion strategy according to the feature distribution structure, wherein the voxel feature fusion strategy comprises a receptive field fusion strategy and a focus fusion strategy; grouping voxel features to obtain first group voxel features and second group voxel features, wherein the voxel features comprise the first voxel features and the second voxel features; splicing the first set of voxel features based on the receptive field fusion strategy and a target dimension, to obtain first spliced voxel features; performing overlap region fusion on the first spliced voxel features through target convolution, to obtain first fused features; compressing first voxel features in the second set of voxel features in a height direction; determining reconstructed voxel features according to the compressed second set of voxel features; splicing the reconstructed voxel features with second voxel features in the second set of voxel features, to obtain second spliced voxel features; performing feature fusion on the second spliced voxel features, to obtain second fused features; and determining target voxel fusion features according to the first fused features and the second fused features.

6. The method of any one of claims 1 to 5, wherein, The determining of the target scale fusion voxel features according to the plurality of historical frame image data and the target voxel fusion features comprises: determining a plurality of historical image features according to a plurality of historical frame image data, and performing feature alignment on the plurality of historical image features, wherein the plurality of aligned historical image features at least include first frame historical image features and second frame historical image features; determining coordinate mapping from the first frame historical image features to the second frame historical image features; obtaining a dimension of the voxel features under current frame image data, and determining sampling points of a voxel space corresponding to the current frame image data according to the dimension; performing spatial conversion on the sampling points based on the coordinate mapping; obtaining sampling voxel features according to the converted sampling points; and determining target scale fusion voxel features according to the sampling voxel features and the target voxel fusion features.

7. The method of claim 6, wherein, The determining of the target scale fusion voxel features according to the sampling voxel features and the target voxel fusion features comprises: determining first frame offset sampling features and second frame offset sampling features according to the sampling voxel features; splicing the first frame offset sampling features and the second frame offset sampling features, to obtain spliced offset sampling features; performing cascade fusion on the spliced offset sampling features, to obtain cascade fused features; and performing scale fusion on the target voxel fusion features and the cascade fused features, to obtain target scale fusion voxel features.

8. An occupancy network prediction apparatus, comprising: a generation module configured to generate first voxel features and second voxel features according to current frame image data of a surrounding environment of a vehicle, wherein the first voxel features and the second voxel features are generated in different ways; a determination module configured to determine a feature distribution structure between the first voxel features and the second voxel features; a fusion module configured to perform feature fusion on the first voxel features and the second voxel features according to the feature distribution structure, to obtain target voxel fusion features; and a prediction module configured to obtain a plurality of historical frame image data of the surrounding environment of the vehicle, determine target scale fusion voxel features according to a plurality of the historical frame data and the target voxel fusion features, and predict occupancy rate information of an object in the surrounding environment of the vehicle through an occupancy network according to the target scale fusion voxel features.

9. The apparatus of claim 8, wherein, The generating module, when generating the first voxel feature and the second voxel feature according to current frame image data of a surrounding environment of a vehicle, is specifically configured to: generate a first voxel feature according to current frame image data of a surrounding environment of a vehicle; estimate the current frame image data to obtain depth estimation data; and generate a second voxel feature based on the current frame image data and the depth estimation data.

10. The apparatus of claim 8, wherein, The generating module, when generating the first voxel feature and the second voxel feature according to current frame image data of a surrounding environment of a vehicle, is specifically configured to: obtain intrinsic parameters and extrinsic parameters of an image acquisition device; determine a first mapping relationship from an image acquisition space to a visual perception space according to first conversion data from the image acquisition device to the vehicle, the intrinsic parameters, and the extrinsic parameters; determine a second mapping relationship from the visual perception space to the image acquisition space according to the extrinsic parameters, second conversion data from the vehicle to a global coordinate system, third conversion data from the visual perception space to the vehicle coordinate system, and fourth conversion data from the global coordinate system to the vehicle coordinate system; perform feature extraction on the current frame image data to obtain image feature data; obtain a target sampling point in the visual perception space and map the target sampling point to the image space; generate a first voxel feature based on a voxel container, the first mapping relationship, the image feature data, and the image space after the sampling point is mapped; and generate a second voxel feature according to the second mapping relationship.

11. The apparatus of claim 10, wherein, The generating module, when generating the second voxel feature according to the second mapping relationship, is specifically configured to: perform characteristic detection on the image acquisition device to obtain device feature data; determine depth estimation data of the current frame image data; generate a second voxel feature according to the depth estimation data, the device feature data, the image feature data, and the second mapping relationship.

12. The apparatus of claim 8, wherein, The fusion module, when performing feature fusion on the first voxel feature and the second voxel feature according to the feature distribution structure to obtain target voxel fusion features, is specifically configured to: determine a voxel feature fusion strategy according to the feature distribution structure, wherein the voxel feature fusion strategy includes a receptive field fusion strategy and a focus fusion strategy; group voxel features to obtain first group voxel features and second group voxel features, wherein the voxel features include the first voxel feature and the second voxel feature; splice the first group voxel features according to a target dimension based on the receptive field fusion strategy to obtain first spliced voxel features; perform overlapping region fusion on the first spliced voxel features through target convolution to obtain first fusion features; compress first voxel features in the second group voxel features in a height direction; determine reconstructed voxel features according to the compressed second group voxel features; splice the reconstructed voxel features and second voxel features in the second group voxel features to obtain second spliced voxel features; perform feature fusion on the second spliced voxel features to obtain second fusion features; and determine target voxel fusion features according to the first fusion features and the second fusion features.

13. The apparatus of any one of claims 8-12, wherein, In a process of determining the target scale fusion voxel feature according to the plurality of historical frame image data and the target voxel fusion feature, the prediction module is specifically configured to: determine a plurality of historical image features according to the plurality of historical frame image data, and perform feature alignment on the plurality of historical image features, wherein the plurality of aligned historical image features at least include a first frame historical image feature and a second frame historical image feature; determine a coordinate mapping from the first frame historical image feature to the second frame historical image feature; obtain a dimension of the voxel feature under the current frame image data, and determine a sampling point of a voxel space corresponding to the current frame image data according to the dimension; perform spatial conversion on the sampling point based on the coordinate mapping; obtain a sampling voxel feature according to the converted sampling point; and determine the target scale fusion voxel feature according to the sampling voxel feature and the target voxel fusion feature.

14. The apparatus of claim 13, wherein, In a process of determining the target scale fusion voxel feature according to the sampling voxel feature and the target voxel fusion feature, the prediction module is specifically configured to: determine a first frame offset sampling feature and a second frame offset sampling feature according to the sampling voxel feature; splice the first frame offset sampling feature and the second frame offset sampling feature to obtain a spliced offset sampling feature; perform cascade fusion on the spliced offset sampling feature to obtain a cascade fusion feature; and perform scale fusion on the target voxel fusion feature and the cascade fusion feature to obtain the target scale fusion voxel feature. a memory, a processor, and a computer program stored on the memory and executable on the processor, the computer program being configured to implement the steps of the occupancy network prediction method according to any one of claims 1 to 7.

15. An occupancy network prediction device comprising:

16. A storage medium, which is a computer-readable storage medium, and has stored thereon a computer program, the computer program being executable by a processor to implement the steps of the occupancy network prediction method according to any one of claims 1 to 7.

17. A computer program product, comprising a computer program, the computer program being executable by a processor to implement the steps of the occupancy network prediction method according to any one of claims 1 to 7. ​

Images