Network occupancy prediction method and apparatus, device, storage medium, and product
By generating fast and slow voxel features, determining their feature distribution structure, and performing feature fusion, and combining historical multi-frame data with an improved pyramid model of the convolutional module, the problem of accuracy in occupancy network prediction under complex environments is solved, and the prediction accuracy and stability of environmental perception are improved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- VOYAH AUTOMOBILE TECH CO LTD
- Filing Date
- 2024-07-24
- Publication Date
- 2026-08-04
AI Technical Summary
Existing technologies have low accuracy in predicting occupancy networks in complex environments.
By generating fast and slow voxel features, the feature distribution structure is determined, and feature fusion is performed. The occupancy network prediction is then performed by combining historical multi-frame data and a pyramid model with an improved convolutional module.
This improves the accuracy of occupation network prediction and the stability and robustness of environmental perception.
Smart Images
Figure CN118864873B_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of autonomous driving technology, and in particular to methods, apparatus, devices, storage media, and products for predicting network occupancy. Background Technology
[0002] Vehicles need to perceive their surroundings in real time during operation, such as road structures, traffic participants, and obstacles, to ensure driving safety and smoothness. Currently, common methods for environmental perception are point cloud processing based on LiDAR or stereo matching based on vision. However, LiDAR struggles to capture detailed information, and vision-based stereo matching is greatly affected by factors such as lighting and occlusion, resulting in low accuracy in perceiving the surrounding environment. To address this, single-frame occupancy network prediction methods have emerged. However, single-frame occupancy network prediction methods have very limited ability to understand the scene, resulting in low accuracy in occupancy network prediction in complex environments.
[0003] The above content is only used to help understand the technical solution of this application and does not represent an admission that the above content is prior art. Summary of the Invention
[0004] The main objective of this application is to provide a method, apparatus, device, storage medium, and product for predicting occupancy networks, aiming to solve the technical problem that the accuracy of existing technologies in predicting occupancy networks in complex environments is low.
[0005] To achieve the above objectives, this application proposes a method for predicting occupied networks, the method comprising:
[0006] Generate fast voxel features and slow voxel features based on image data of the surrounding environment of the target object;
[0007] Determine the feature distribution structure between the fast voxel features and the slow voxel features;
[0008] Based on the feature distribution structure, the fast voxel features and the slow voxel features are fused to obtain the target voxel fusion features;
[0009] The target scale fused voxel features are determined based on historical multi-frame data and the target voxel fusion features, and the occupancy network prediction is performed based on the target scale fused voxel features.
[0010] In one embodiment, the step of generating fast voxel features and slow voxel features based on image data of the surrounding environment of the target object includes:
[0011] Acquire the intrinsic and extrinsic parameters of the camera device used to capture image data of the surrounding environment of the target object;
[0012] The mapping relationship between the camera space and the visual perception space is determined based on the internal parameters, the external parameters, and the conversion data from the camera device to the vehicle space.
[0013] The mapping relationship between the visual perception space and the camera space is determined based on the external parameters, the transformation data from the vehicle space to the global coordinate system, the transformation data from the visual perception space to the vehicle coordinate system, and the transformation data from the global coordinate system to the vehicle coordinate system.
[0014] Feature extraction is performed on the image data to obtain image feature data;
[0015] Obtain target sampling points in the visual perception space and map the target sampling points to the image space;
[0016] Fast voxel features are generated based on the voxel container, the mapping relationship from the camera space to the visual perception space, the image feature data, and the image space after the mapping sampling points.
[0017] Slow voxel features are generated based on the mapping relationship from the visual perception space to the camera space.
[0018] In one embodiment, the step of generating slow voxel features based on the mapping relationship from the visual perception space to the camera space includes:
[0019] Perform characteristic testing on the camera equipment to obtain equipment feature data;
[0020] Determine the depth estimation data for the image data;
[0021] A shift-multiplication strategy is used to generate slow voxel features based on the depth estimation data, the device feature data, the image feature data, and the mapping relationship from the visual perception space to the camera space.
[0022] In one embodiment, the step of fusing the fast voxel features and the slow voxel features according to the feature distribution structure to obtain the target voxel fused features includes:
[0023] A voxel feature fusion strategy is determined based on the feature distribution structure, wherein the voxel feature fusion strategy includes a receptive region fusion strategy and a focus fusion strategy;
[0024] The fast voxel features and the slow voxel features are grouped separately to obtain the first group of voxel features and the second group of voxel features.
[0025] Based on the receptive region fusion strategy, the first group of voxel features are spliced together according to the target dimension using a preset merging method to obtain the first spliced voxel features.
[0026] The overlapping regions of the first spliced voxel features are fused using target convolution.
[0027] The fast voxel features in the second group of voxel features are compressed along the height direction;
[0028] The reconstructed voxel features are determined based on the compressed second set of voxel features;
[0029] The reconstructed voxel features are spliced together with the slow voxel features in the second set of voxel features to obtain the second spliced voxel features.
[0030] Feature fusion is performed on the second spliced voxel features;
[0031] The target voxel fusion features are determined based on the fusion results of the first spliced voxel features and the fusion results of the second spliced voxel features.
[0032] In one embodiment, the step of determining the target-scale fused voxel features based on historical multi-frame data and the target voxel fusion features includes:
[0033] Historical multi-frame features are determined based on the historical multi-frame data, and the historical multi-frame data is aligned, wherein the aligned historical multi-frame data includes at least the first frame of historical data and the second frame of historical data.
[0034] Set the mapping parameters for the sensing devices in the second frame;
[0035] Calculate the coordinate mapping from the first frame to the second frame based on the mapping parameters;
[0036] Obtain the dimension of the voxel features in the current frame data, and determine the sampling points in the voxel space of the current frame based on the dimension;
[0037] The sampling points are spatially transformed based on the coordinate mapping;
[0038] Based on the grid sampling strategy, the sampling voxel features are obtained from the transformed sampling points;
[0039] The target scale fused voxel features are determined based on the sampled voxel features and the target voxel fusion features.
[0040] In one embodiment, the step of determining the target-scale fused voxel features based on the sampled voxel features and the target voxel fusion features includes:
[0041] The first frame offset sampling features and the second frame offset sampling features are determined based on the sampling voxel features;
[0042] The offset sampling features of the first frame and the offset sampling features of the second frame are concatenated to obtain the concatenated offset sampling features;
[0043] The spliced offset sampling features are cascaded and fused based on the target convolution to obtain cascaded fused features;
[0044] By improving the pyramid model of the convolution module, the target voxel fusion feature and the cascaded fusion feature are scale-fused to obtain the target scale fused voxel feature.
[0045] Furthermore, to achieve the above objectives, this application also proposes a network occupancy prediction device, which includes:
[0046] The generation module is used to generate fast voxel features and slow voxel features based on image data of the surrounding environment of the target object.
[0047] A determination module is used to determine the feature distribution structure between the fast voxel features and the slow voxel features;
[0048] The fusion module is used to fuse the fast voxel features and the slow voxel features according to the feature distribution structure to obtain the target voxel fused features.
[0049] The prediction module is used to determine the target scale fused voxel features based on historical multi-frame data and the target voxel fusion features, and to perform occupancy network prediction based on the target scale fused voxel features.
[0050] In addition, to achieve the above objectives, this application also proposes a network occupancy prediction device, the device comprising: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the network occupancy prediction method as described above.
[0051] In addition, to achieve the above objectives, this application also proposes a storage medium, which is a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, it implements the steps of the occupation network prediction method described above.
[0052] In addition, to achieve the above objectives, this application also provides a computer program product, which includes a computer program that, when executed by a processor, implements the steps of the occupation network prediction method described above.
[0053] One or more technical solutions proposed in this application have at least the following technical effects: generating fast voxel features and slow voxel features based on image data of the surrounding environment of the target object; determining the feature distribution structure between the fast voxel features and the slow voxel features; performing feature fusion on the fast voxel features and the slow voxel features according to the feature distribution structure to obtain target voxel fusion features; determining target scale fusion voxel features based on historical multi-frame data and the target voxel fusion features, and performing occupancy network prediction based on the target scale fusion voxel features; through the above method, multi-view image data in a single-frame scene is processed into scene-aware fast voxel features and slow voxel features, and feature fusion is performed using the analyzed feature distribution structure, fully considering the limit relationship between multiple frames, combining historical multi-frame data and a pyramid model with an improved convolution module for scale fusion, thereby effectively improving the accuracy of occupancy network prediction and enhancing the stability and robustness of environmental perception. Attached Figure Description
[0054] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.
[0055] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, for those skilled in the art, other drawings can be obtained based on these drawings without creative effort.
[0056] Figure 1 This is a flowchart illustrating an embodiment of the network prediction method of this application.
[0057] Figure 2 This is a flowchart illustrating Embodiment 2 of the network prediction method of this application;
[0058] Figure 3 A schematic diagram of the overall process for generating voxel features in the occupancy network prediction method provided in Embodiment 2 of this application;
[0059] Figure 4 This is a schematic diagram of the module structure of the network prediction device according to an embodiment of this application;
[0060] Figure 5 This is a schematic diagram of the device structure of the hardware operating environment involved in the network prediction method in this application embodiment.
[0061] The purpose, features, and advantages of this application will be further explained in conjunction with the embodiments and with reference to the accompanying drawings. Detailed Implementation
[0062] It should be noted that the executing entity in this embodiment can be a computing service device with data processing, network communication, and program execution functions, such as a tablet computer, personal computer, or mobile phone, or an electronic device or occupancy prediction controller capable of performing the above functions. The following description uses an occupancy prediction controller as an example to illustrate this embodiment and the subsequent embodiments.
[0063] Based on this, embodiments of this application provide a method for predicting network occupancy, referring to... Figure 1 , Figure 1 This is a flowchart illustrating the first embodiment of the network prediction method of this application.
[0064] In this embodiment, the occupancy network prediction method includes steps S10 to S40:
[0065] Step S10: Generate fast voxel features and slow voxel features based on the image data of the surrounding environment of the target object.
[0066] It should be noted that the image data refers to the multi-view image data captured by a camera device set on the target object, which can be a vehicle. In this case, the surrounding environment of the target environment can be a complex 3D environment. In such a complex 3D environment, the accuracy of the single-frame occupancy network prediction is relatively low. Furthermore, in this embodiment, voxel features are divided into fast voxel features and slow voxel features. The difference between the two lies in whether additional data is needed to assist in the generation process.
[0067] Step S20: Determine the feature distribution structure between the fast voxel features and the slow voxel features.
[0068] It is understandable that the feature distribution structure refers to the distribution structure of voxel features between fast voxel features and slow voxel features. This feature distribution structure can be a distribution structure with repeating space in geometric position. If it is this distribution structure, it indicates that fast voxels and slow voxels will exhibit similar spatial characteristics.
[0069] Step S30: Based on the feature distribution structure, perform feature fusion on the fast voxel features and the slow voxel features to obtain the target voxel fusion features.
[0070] It should be understood that the target voxel fusion feature refers to the feature after fusing fast voxel features and slow voxel features. For fast voxel features and slow voxel features, there are two types of voxel feature fusion strategies, namely the receptive region fusion strategy and the receptive region fusion strategy.
[0071] Further, step S30 includes: determining a voxel feature fusion strategy based on the feature distribution structure, wherein the voxel feature fusion strategy includes a receptive region fusion strategy and a focus fusion strategy; grouping the fast voxel features and the slow voxel features respectively to obtain a first group of voxel features and a second group of voxel features; splicing the first group of voxel features according to the target dimension using a preset merging method based on the receptive region fusion strategy to obtain a first spliced voxel feature; performing overlapping region fusion on the first spliced voxel features through target convolution; compressing the fast voxel features in the second group of voxel features according to the height direction; determining reconstructed voxel features based on the compressed second group of voxel features; splicing the reconstructed voxel features with the slow voxel features in the second group of voxel features to obtain a second spliced voxel feature; performing feature fusion on the second spliced voxel feature; and determining a target voxel fusion feature based on the fusion result of the first spliced voxel feature and the fusion result of the second spliced voxel feature.
[0072] Understandably, after obtaining the fast and slow voxel features, they are grouped into the first group and the second group of voxel features. When performing feature fusion using the receptive region fusion strategy, a preset merging method is needed to concatenate the first group of voxel features according to the target dimension. This preset merging method can be concatenation, and the target dimension can be C. cams In the calculation settings, the dimension will remain D=C. depth This allows for the acquisition of new features, and the above method ensures the rapid acquisition of voxel features V. fast With slow voxel characteristics V slow In 3D space, D×H×W, or Z, X, Y, ensures overlapping and fusion of spatial locations. Finally, a target convolution is used to fuse the overlapping regions, ensuring the generation of redundant features and reducing computational cost. This target convolution can be a single-layer 3D convolution.
[0073] It should be noted that, because objects in real space possess geometric information such as height and width, while fast voxel features V... fast It is obtained merely by simple geometric mapping and feature filling, and its description of this geometric information is weaker than that of slow voxel features V. slow Considering this key issue, the fast voxel feature V in the second group of voxel features is first... fast Compression is performed along the height direction. The compressed features can be viewed as the vertical feature distribution of voxel features, which can describe the location of the object. Then, repeated operations are used to obtain reconstructed voxel features, which can be specifically represented as:
[0074]
[0075] Here, repeat(.) represents a copy operation on data of dimension 1, and D, H, W represent the voxel feature dimensions.
[0076] It should be noted that, through the above calculations, a rough description of the object in the height direction can be obtained without losing the object's true spatial description, thus completing voxel feature reconstruction. Then, the reconstructed voxel V... fast Compared with the slow voxel feature V in the second group of voxel features slow The data is spliced to enrich the slow voxel features and improve the accuracy of occupancy prediction.
[0077] It should be understood that, assuming the current frame is t, and the historical frames are t-1,...,tk, where k represents the number of historical frames used, and the receptive region fusion strategy is used between t and t-1 (the first group of voxel features), and between t-2,...,tk (the second group of voxel features), the receptive region fusion strategy is used. By grouping the feature fusion strategies between the current frame and the historical frames, it is ensured that the historical frame data can obtain as much perceptual data as possible, while also enabling the current frame to have a more generalized representation of the scene. The final target voxel fusion feature can be expressed as:
[0078]
[0079] in, To represent the target voxel fusion features, use F r F represents the fusion result of the first splicing voxel features. f This indicates the fusion result of the second splicing voxel features.
[0080] Step S40: Determine the target scale fused voxel features based on historical multi-frame data and the target voxel fusion features, and perform occupancy network prediction based on the target scale fused voxel features.
[0081] It should be noted that, in order to enrich voxel features and fully consider the cascading relationship between multiple frames, the target scale fused voxel features are determined by combining historical multi-frame data, and then the target scale fused voxel features are used for occupancy network prediction, thereby effectively improving the accuracy of occupancy network prediction.
[0082] Further, the step of determining the target scale fused voxel features based on historical multi-frame data and the target voxel fusion features includes: determining historical multi-frame features based on the historical multi-frame data, and aligning the historical multi-frame data, wherein the aligned historical multi-frame data includes at least the first frame historical data and the second frame historical data; setting the mapping parameters of the sensing device within the second frame; calculating the coordinate mapping from the first frame to the second frame based on the mapping parameters; obtaining the dimension of the voxel features in the current frame data, and determining the sampling points in the voxel space of the current frame based on the dimension; performing spatial transformation on the sampling points based on the coordinate mapping; obtaining the sampling voxel features based on the transformed sampling points using a grid sampling strategy; and determining the target scale fused voxel features based on the sampling voxel features and the target voxel fusion features.
[0083] It should be understood that, in order to ensure enhanced correlation between different frames and continuously improve the perceptual performance of voxel features used for occupancy prediction through cascading, after obtaining historical multi-frame features, it is necessary to align the historical multi-frame features and then cascade and fuse the aligned historical multi-frame data. Taking the first frame historical data and the second frame historical data as an example, the first frame historical data can also be called t-frame data, and the second frame historical data can also be called t-1 frame data. The mapping parameters of the sensing device in the second frame include, but are not limited to, the transformation data from the camera device to the vehicle coordinate system. Transformation data from vehicle coordinate system to global coordinate system By using the transformation data from the camera equipment to the vehicle coordinate system Transformation data from the vehicle coordinate system to the global coordinate system The coordinate mapping from frame t-1 to frame t is obtained as follows:
[0084]
[0085] Among them, M t-1→t This represents the coordinate mapping from frame t-1 data to frame t data. This represents the transformation data from the camera equipment to the vehicle's coordinate system. This represents the transformation data from the vehicle coordinate system to the global coordinate system.
[0086] Understandably, after obtaining the coordinate mapping from frame t-1 to frame t, it is necessary to further determine the sampling point in the voxel space of the current frame, which can be a sampled 3D point.
[0087] It should be noted that after obtaining the sampling points in the voxel space of the current frame, it is necessary to transform these sampling points into the visual perception space using coordinate mapping. Specifically, this can be defined as... Then, the data from frame t-1 is sampled into the data from frame t, specifically as follows:
[0088]
[0089] in, This means sampling the data from frame t-1 into the data from frame t. This indicates that the sampling points are being converted to the visual perception space, M t-1→t This represents the coordinate mapping from frame t-1 data to frame t data.
[0090] It should be understood that after obtaining the transformed sampling points, the sampling voxel features are obtained using a grid sampling strategy, specifically:
[0091]
[0092] Among them, V t-1→t Indicates the characteristics of the sampled voxels. This indicates that the data from frame t-1 is sampled into the data from frame t, f gs This indicates the grid sampling strategy.
[0093] Further, the step of determining the target scale fused voxel feature based on the sampled voxel feature and the target voxel fusion feature includes: determining a first frame offset sampling feature and a second frame offset sampling feature based on the sampled voxel feature; concatenating the first frame offset sampling feature and the second frame offset sampling feature to obtain a concatenated offset sampling feature; performing cascaded fusion on the concatenated offset sampling feature based on the target convolution to obtain a cascaded fused feature; and performing scale fusion on the target voxel fusion feature and the cascaded fused feature by improving the pyramid model of the convolution module to obtain the target scale fused voxel feature.
[0094] Understandably, after obtaining the sampled voxel features of the data at time t-1, the interference of redundant features is reduced to ensure that the perceptual features are more focused on the scene objects. After obtaining the first frame offset sampling features and the second frame offset sampling features, in order to ensure the efficiency of the calculation, the first frame offset sampling features and the second frame offset sampling features are concatenated and then cascaded using a target convolution. This target convolution can be a single-layer 3D convolution.
[0095] It should be understood that, in order to further improve the ability to perceive objects of different scales, a multi-scale fusion method is adopted for scale fusion. In the scale fusion process, an improved pyramid model with convolutional modules is used. Specifically, the 2D convolutional modules of the original pyramid model are improved to 3D convolutional modules to adapt to the data changes of the model. After the fusion is completed, the target scale fused voxel features are obtained, and the target scale fused voxel features are used for the final occupancy network prediction.
[0096] This embodiment generates fast voxel features and slow voxel features based on image data of the surrounding environment of the target object; determines the feature distribution structure between the fast and slow voxel features; fuses the fast and slow voxel features according to the feature distribution structure to obtain target voxel fusion features; determines target scale fusion voxel features based on historical multi-frame data and the target voxel fusion features, and performs occupancy network prediction based on the target scale fusion voxel features; through the above method, multi-view image data in a single-frame scene is processed into scene-aware fast and slow voxel features, and feature fusion is performed using the analyzed feature distribution structure, fully considering the limiting relationships between multiple frames, combining historical multi-frame data and a pyramid model with an improved convolution module for scale fusion, thereby effectively improving the accuracy of occupancy network prediction and enhancing the stability and robustness of environmental perception.
[0097] Based on the first embodiment of this application, in the second embodiment of this application, the content that is the same as or similar to that in the first embodiment described above can be referred to the above description, and will not be repeated hereafter. Based on this, please refer to... Figure 2 Step S10 includes steps S101 to S107:
[0098] Step S101: Obtain the internal and external parameters of the camera device used to capture image data of the surrounding environment of the target object.
[0099] It should be noted that the camera device can be a high-definition camera set on the target object. Through mapping and transformation, the image data of multiple views can be unified into the visual perception space, which is more conducive to autonomous driving perception tasks in the current scenario. The intrinsic parameters of the camera device include, but are not limited to, focal length, pixel size, coordinate axis tilt and distortion parameters, while the extrinsic parameters of the camera include, but are not limited to, rotation matrix and translation vector.
[0100] Step S102: Determine the mapping relationship between the camera space and the visual perception space based on the internal parameters, the external parameters, and the conversion data from the camera device to the vehicle space.
[0101] It should be understood that after obtaining the intrinsic and extrinsic parameters, coordinate transformation is performed by combining the data from the camera device to the vehicle space to complete the mutual mapping between the camera device space and the visual perception space, thereby determining the mapping relationship from the camera space to the visual perception space. Specifically:
[0102]
[0103] Among them, P BEV P represents the visual perception space. cams Indicates the camera equipment space, E represents external parameters, I represents the external parameters. -1 This represents the inverse matrix of the intrinsic parameters.
[0104] It should be noted that the above formula is used to establish a mapping relationship from the camera space to the visual perception space in order to complete the sampling and alignment between visual perception features and image features.
[0105] Step S103: Determine the mapping relationship from the visual perception space to the camera space based on the external parameters, the conversion data from the vehicle space to the global coordinate system, the conversion data from the visual perception space to the vehicle coordinate system, and the conversion data from the global coordinate system to the vehicle coordinate system.
[0106] It is understandable that global coordinate data in the global coordinate system includes, but is not limited to, inertial navigation data, GPS positioning data, and navigation data. After obtaining the extrinsic parameters, the transformation data from the vehicle space to the global coordinate system, the transformation data from the visual perception space to the vehicle coordinate system, and the transformation data from the global coordinate system to the vehicle coordinate system are combined to realize the transformation from the visual perception space to the camera device coordinate system, so as to determine the mapping relationship from the visual perception space to the camera space. Specifically:
[0107]
[0108] Among them, P cams P represents the space for the camera equipment. BEV This represents the visual perception space, and E represents the extrinsic parameter. This represents the data transformed from the vehicle space to the global coordinate system. This represents the data transformed from the visual perception space to the vehicle's coordinate system. This represents the data transformed from the global coordinate system to the vehicle coordinate system.
[0109] It should be noted that the above formula utilizes intermediate values between the vehicle coordinate system and the global coordinate system to achieve information transfer between different representation spaces. In summary, by employing different coordinate mapping methods, geometric information transfer under the calibration data is achieved.
[0110] Step S104: Extract features from the image data to obtain image feature data.
[0111] It should be understood that in order to facilitate the generation of fast voxel features and slow voxel features, it is necessary to extract features from the image data, namely image feature data.
[0112] Step S105: Obtain target sampling points in the visual perception space and map the target sampling points to the image space.
[0113] It should be noted that in this implementation, the generation of voxel features is differentiated into fast and slow voxel features based on whether additional data is needed to assist in defining them. If the generated voxel features are generated solely through coordinate transformations and feature mappings between calibration parameters, this is called fast voxel feature generation. Fast voxel features can preserve the correspondence of the original image feature information and can achieve relatively accurate scene geometry perception with the help of calibration parameters. If the generated voxel features require additional data and calibration parameters to complete the calibration process, this is called slow voxel feature generation.
[0114] It should be understood that, based on the fast voxel features and slow voxel features mentioned above, it is necessary to generate fast voxel features using target sampling points. After obtaining the target sampling points in the visual perception space, the target sampling points are mapped to the image space using an objective function, which can be <·>.
[0115] Step S106: Generate fast voxel features based on the voxel container, the mapping relationship from the camera space to the visual perception space, the image feature data, and the image space after the mapping sampling points.
[0116] Understandably, after determining the image space with mapped sampling points, a voxel container is used to generate fast voxel features based on the mapping relationship from the camera space to the visual perception space and the image feature data. Specifically:
[0117]
[0118] Among them, V fast Indicates rapid voxel characteristics, The voxel container is represented by <·>, and the objective function f is represented by f. cams Eq(1) represents the image feature data and represents the mapping relationship from the camera space to the visual perception space.
[0119] It should be noted that after mapping the target sampling points to the image space using the objective function, and with the voxel container... Alignment and feature filling are performed to generate fast voxel features.
[0120] Step S107: Generate slow voxel features based on the mapping relationship from the visual perception space to the camera space.
[0121] Understandably, after obtaining the mapping relationship from the visual perception space to the camera space, it is necessary to learn and generate slow voxel features by combining the image features corresponding to the image feature data.
[0122] Further, step S107 includes: performing feature detection on the camera device to obtain device feature data; determining depth estimation data of the image data; and generating slow voxel features using a shift-multiplication strategy based on the depth estimation data, the device feature data, the image feature data, and the mapping relationship from the visual perception space to the camera space.
[0123] It should be understood that in order to effectively improve the generation of slow voxel features, device feature data is required. At this time, the characteristics of the camera device are detected, and then a shift-multiplication strategy is used to generate more definite voxel features based on depth estimation data, device feature data, image feature data, and the mapping relationship from visual perception space to camera space. These are the slow voxel features.
[0124] refer to Figure 3 , Figure 3 The overall process for generating voxel features is illustrated in the diagram. Specifically, it consists of two paths: the first path generates fast voxel features, and the second path generates slow voxel features. After acquiring target sampling points in the visual perception space and mapping them to the image space, voxel containers are used... Based on the mapping relationship from the camera space to the visual perception space and the image feature data, a fast voxel feature is generated and output. Similarly, after detecting the device feature data of the camera device, the depth estimation data of the image data is further determined. Then, a shift-multiplication strategy is used to calculate the intermediate voxel feature. The intermediate voxel feature is then processed by the BEVPool operation to obtain the slow voxel feature, and the slow voxel feature is output.
[0125] This embodiment acquires the intrinsic and extrinsic parameters of a camera device used to capture image data of the surrounding environment of a target object; determines the mapping relationship from the camera space to the visual perception space based on the intrinsic and extrinsic parameters and the camera-to-vehicle space conversion data; determines the mapping relationship from the visual perception space to the camera space based on the extrinsic parameters, the vehicle-to-global coordinate system conversion data, the visual perception space-to-vehicle coordinate system conversion data, and the global coordinate system-to-vehicle coordinate system conversion data; extracts features from the image data to obtain image feature data; acquires target sampling points in the visual perception space and maps these target sampling points to the image space; and uses voxel containers... The mapping relationship from the camera space to the visual perception space, the image feature data, and the image space after mapping the sampling points generate fast voxel features; slow voxel features are generated according to the mapping relationship from the visual perception space to the camera space. In this way, the mapping relationship between the visual perception space and the camera space is determined using the intrinsic and extrinsic parameters of the camera device and multi-dimensional transformation data. Then, the target sampling points in the visual perception space are mapped to the image space, and fast voxel features and slow voxel features are generated using the voxel container and the depth estimation data of the image data, respectively. This can effectively improve the accuracy of generating fast voxel features and slow voxel features.
[0126] This application also provides a network occupancy prediction device; please refer to... Figure 4 The network occupancy prediction device includes:
[0127] The generation module 10 is used to generate fast voxel features and slow voxel features based on image data of the surrounding environment of the target object.
[0128] The determination module 20 is used to determine the feature distribution structure between the fast voxel features and the slow voxel features.
[0129] The fusion module 30 is used to perform feature fusion on the fast voxel features and the slow voxel features according to the feature distribution structure to obtain the target voxel fusion features.
[0130] The prediction module 40 is used to determine the target scale fused voxel features based on historical multi-frame data and the target voxel fusion features, and to perform occupancy network prediction based on the target scale fused voxel features.
[0131] This embodiment generates fast voxel features and slow voxel features based on image data of the surrounding environment of the target object; determines the feature distribution structure between the fast and slow voxel features; fuses the fast and slow voxel features according to the feature distribution structure to obtain target voxel fusion features; determines target scale fusion voxel features based on historical multi-frame data and the target voxel fusion features, and performs occupancy network prediction based on the target scale fusion voxel features; through the above method, multi-view image data in a single-frame scene is processed into scene-aware fast and slow voxel features, and feature fusion is performed using the analyzed feature distribution structure, fully considering the limiting relationships between multiple frames, combining historical multi-frame data and a pyramid model with an improved convolution module for scale fusion, thereby effectively improving the accuracy of occupancy network prediction and enhancing the stability and robustness of environmental perception.
[0132] The possession network prediction device provided in this application, employing the possession network prediction method in the above embodiments, can solve the technical problem of low accuracy in possession network prediction under complex environments in the prior art. Compared with the prior art, the beneficial effects of the possession network prediction device provided in this application are the same as those of the possession network prediction method provided in the above embodiments, and other technical features in the possession network prediction device are the same as those disclosed in the methods of the above embodiments, and will not be repeated here.
[0133] In one embodiment, the generation module 10 is further configured to: acquire intrinsic and extrinsic parameters of a camera device used to capture image data of the surrounding environment of a target object; determine a mapping relationship from camera space to visual perception space based on the intrinsic and extrinsic parameters and the camera device-to-vehicle space conversion data; determine a mapping relationship from visual perception space to camera space based on the extrinsic parameters, the vehicle space-to-global coordinate system conversion data, the visual perception space-to-vehicle coordinate system conversion data, and the global coordinate system-to-vehicle coordinate system conversion data; extract features from the image data to obtain image feature data; acquire target sampling points in the visual perception space and map the target sampling points to the image space; generate fast voxel features based on the voxel container, the mapping relationship from camera space to visual perception space, the image feature data, and the image space after mapping the sampling points; and generate slow voxel features based on the mapping relationship from visual perception space to camera space.
[0134] In one embodiment, the generation module 10 is further configured to perform characteristic detection on the camera device to obtain device feature data; determine depth estimation data of the image data; and generate slow voxel features using a shift-multiplication strategy based on the depth estimation data, the device feature data, the image feature data, and the mapping relationship from the visual perception space to the camera space.
[0135] In one embodiment, the fusion module 30 is further configured to determine a voxel feature fusion strategy based on the feature distribution structure, wherein the voxel feature fusion strategy includes a receptive region fusion strategy and a focus fusion strategy; group the fast voxel features and the slow voxel features respectively to obtain a first group of voxel features and a second group of voxel features; based on the receptive region fusion strategy, stitch the first group of voxel features together according to the target dimension using a preset merging method to obtain a first stitched voxel feature; perform overlapping region fusion on the first stitched voxel feature through target convolution; compress the fast voxel features in the second group of voxel features according to the height direction; determine the reconstructed voxel features based on the compressed second group of voxel features; stitch the reconstructed voxel features together with the slow voxel features in the second group of voxel features to obtain a second stitched voxel feature; perform feature fusion on the second stitched voxel feature; and determine the target voxel fusion feature based on the fusion result of the first stitched voxel feature and the fusion result of the second stitched voxel feature.
[0136] In one embodiment, the prediction module 40 is further configured to: determine historical multi-frame features based on the historical multi-frame data; align the historical multi-frame data, wherein the aligned historical multi-frame data includes at least a first frame of historical data and a second frame of historical data; set mapping parameters for the sensing device within the second frame; calculate the coordinate mapping from the first frame to the second frame based on the mapping parameters; obtain the dimension of the voxel features in the current frame data; determine the sampling points in the voxel space of the current frame based on the dimension; perform spatial transformation on the sampling points based on the coordinate mapping; obtain the sampling voxel features based on the transformed sampling points using a grid sampling strategy; and determine the target scale fused voxel features based on the sampling voxel features and the target voxel fusion features.
[0137] In one embodiment, the prediction module 40 is further configured to determine a first frame offset sampling feature and a second frame offset sampling feature based on the sampled voxel features; concatenate the first frame offset sampling feature and the second frame offset sampling feature to obtain a concatenated offset sampling feature; perform cascaded fusion of the concatenated offset sampling feature based on the target convolution to obtain a cascaded fused feature; and perform scale fusion of the target voxel fusion feature and the cascaded fused feature by improving the pyramid model of the convolution module to obtain a target scale fused voxel feature.
[0138] This application provides a network occupancy prediction device, which includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, which are executed by the at least one processor to enable the at least one processor to perform the network occupancy prediction method in the above embodiment 1.
[0139] The following is for reference. Figure 5 This document illustrates a structural diagram of a network occupancy prediction device suitable for implementing embodiments of this application. The network occupancy prediction device in these embodiments may include, but is not limited to, mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (Personal Digital Assistants), PADs (Portable Application Description), PMPs (Portable Media Players), in-vehicle terminals (e.g., in-vehicle navigation terminals), and fixed terminals such as digital TVs and desktop computers. Figure 5 The illustrated network occupancy prediction device is merely an example and should not impose any limitations on the functionality and scope of use of the embodiments of this application.
[0140] like Figure 5 As shown, the occupancy prediction device may include a processing unit 1001 (e.g., a central processing unit, a graphics processing unit, etc.) that can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 1002 or a program loaded from a storage device 1003 into a random access memory (RAM) 1004. The RAM 1004 also stores various programs and data required for the operation of the occupancy prediction device. The processing unit 1001, ROM 1002, and RAM 1004 are interconnected via a bus 1005. An input / output (I / O) interface 1006 is also connected to the bus. Typically, the following systems can be connected to the I / O interface 1006: input devices 1007 including, for example, a touchscreen, touchpad, keyboard, mouse, image sensor, microphone, accelerometer, gyroscope, etc.; output devices 1008 including, for example, a liquid crystal display (LCD), speaker, vibrator, etc.; storage devices 1003 including, for example, magnetic tape, hard disk, etc.; and communication devices 1009. Communication device 1009 allows the occupancy network prediction device to communicate wirelessly or wiredly with other devices to exchange data. Although the figures show occupancy network prediction devices with various systems, it should be understood that implementation or possession of all the systems shown is not required. More or fewer systems may be implemented alternatively.
[0141] Specifically, according to the embodiments disclosed in this application, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments disclosed in this application include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device, or installed from storage device 1003, or installed from ROM 1002. When the computer program is executed by processing device 1001, it performs the functions defined in the methods of the embodiments disclosed in this application.
[0142] The possession network prediction device provided in this application, employing the possession network prediction method in the above embodiments, can solve the technical problem of low accuracy in possession network prediction under complex environments in the prior art. Compared with the prior art, the beneficial effects of the possession network prediction device provided in this application are the same as those of the possession network prediction method provided in the above embodiments, and other technical features in the possession network prediction device are the same as those disclosed in the method of the previous embodiment, and will not be repeated here.
[0143] It should be understood that the various parts disclosed in this application can be implemented using hardware, software, firmware, or a combination thereof. In the description of the above embodiments, specific features, structures, materials, or characteristics can be combined in any suitable manner in one or more embodiments or examples.
[0144] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.
[0145] This application provides a computer-readable storage medium having computer-readable program instructions (i.e., a computer program) stored thereon, the computer-readable program instructions being used to execute the occupancy network prediction method in the above embodiments.
[0146] The computer-readable storage medium provided in this application may be, for example, a USB flash drive, but is not limited to, electrical, magnetic, optical, electromagnetic, infrared, or semiconductor systems, devices, or any combination thereof. More specific examples of computer-readable storage media may include, but are not limited to: electrical connections having one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination thereof. In this embodiment, the computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, system, or device. The program code contained on the computer-readable storage medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (Radio Frequency), etc., or any suitable combination thereof.
[0147] The aforementioned computer-readable storage medium may be included in the possession network prediction device; or it may exist independently and not assembled into the possession network prediction device.
[0148] Computer program code for performing the operations of this application can be written in one or more programming languages or a combination thereof, including object-oriented programming languages such as Java, Smalltalk, and C++, and conventional procedural programming languages such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a Local Area Network (LAN) or a Wide Area Network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).
[0149] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this application. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.
[0150] The modules described in the embodiments of this application can be implemented in software or hardware. The names of the modules do not necessarily limit the functionality of the unit itself.
[0151] The readable storage medium provided in this application is a computer-readable storage medium that stores computer-readable program instructions (i.e., a computer program) for executing the above-described occupancy network prediction method. This addresses the technical problem of low accuracy in occupancy network prediction under complex environments in existing technologies. Compared to existing technologies, the beneficial effects of the computer-readable storage medium provided in this application are the same as those of the occupancy network prediction method provided in the above embodiments, and will not be elaborated upon here.
[0152] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of the occupation network prediction method described above.
[0153] The computer program product provided in this application can solve the technical problem of low accuracy in occupancy network prediction under complex environments in existing technologies. Compared with the prior art, the beneficial effects of the computer program product provided in this application are the same as those of the occupancy network prediction method provided in the above embodiments, and will not be repeated here.
[0154] The above description is only a part of the embodiments of this application and does not limit the patent scope of this application. All equivalent structural transformations made under the technical concept of this application and using the contents of the specification and drawings of this application, or direct / indirect applications in other related technical fields, are included in the patent protection scope of this application.
Claims
1. An occupancy network prediction method characterized by, The method includes: Generate fast voxel features and slow voxel features based on image data of the surrounding environment of the target object; Determine the feature distribution structure between the fast voxel features and the slow voxel features; Based on the feature distribution structure, the fast voxel features and the slow voxel features are fused to obtain the target voxel fusion features; The target scale fused voxel features are determined based on historical multi-frame data and the target voxel fusion features, and the occupancy network prediction is performed based on the target scale fused voxel features. The step of generating fast voxel features and slow voxel features based on image data of the surrounding environment of the target object includes: Acquire the intrinsic and extrinsic parameters of the camera device used to capture image data of the surrounding environment of the target object; The mapping relationship between the camera space and the visual perception space is determined based on the internal parameters, the external parameters, and the conversion data from the camera device to the vehicle space. The mapping relationship between the visual perception space and the camera space is determined based on the external parameters, the transformation data from the vehicle space to the global coordinate system, the transformation data from the visual perception space to the vehicle coordinate system, and the transformation data from the global coordinate system to the vehicle coordinate system. Feature extraction is performed on the image data to obtain image feature data; Obtain target sampling points in the visual perception space and map the target sampling points to the image space; Fast voxel features are generated based on the voxel container, the mapping relationship from the camera space to the visual perception space, the image feature data, and the image space after the mapping sampling points. Slow voxel features are generated based on the mapping relationship from the visual perception space to the camera space; The step of fusing the fast voxel features and the slow voxel features according to the feature distribution structure to obtain the target voxel fused features includes: A voxel feature fusion strategy is determined based on the feature distribution structure, wherein the voxel feature fusion strategy includes a receptive region fusion strategy and a focus fusion strategy; The fast voxel features and the slow voxel features are grouped separately to obtain the first group of voxel features and the second group of voxel features. Based on the receptive region fusion strategy, the first group of voxel features are spliced together according to the target dimension using a preset merging method to obtain the first spliced voxel features. The overlapping regions of the first spliced voxel features are fused using target convolution. The fast voxel features in the second group of voxel features are compressed along the height direction; The reconstructed voxel features are determined based on the compressed second set of voxel features; The reconstructed voxel features are spliced together with the slow voxel features in the second set of voxel features to obtain the second spliced voxel features. Feature fusion is performed on the second spliced voxel features; The target voxel fusion features are determined based on the fusion results of the first spliced voxel features and the fusion results of the second spliced voxel features.
2. The method of claim 1, wherein, The step of generating slow voxel features based on the mapping relationship from the visual perception space to the camera space includes: Perform characteristic testing on the camera equipment to obtain equipment feature data; Determine the depth estimation data for the image data; A shift-multiplication strategy is used to generate slow voxel features based on the depth estimation data, the device feature data, the image feature data, and the mapping relationship from the visual perception space to the camera space.
3. The method of any one of claims 1 to 2, wherein, The step of determining the target scale fused voxel features based on historical multi-frame data and the target voxel fusion features includes: Historical multi-frame features are determined based on the historical multi-frame data, and the historical multi-frame data is aligned, wherein the aligned historical multi-frame data includes at least the first frame of historical data and the second frame of historical data. Set the mapping parameters for the sensing devices in the second frame; Calculate the coordinate mapping from the first frame to the second frame based on the mapping parameters; Obtain the dimension of the voxel features in the current frame data, and determine the sampling points in the voxel space of the current frame based on the dimension; The sampling points are spatially transformed based on the coordinate mapping; Based on the grid sampling strategy, the sampling voxel features are obtained from the transformed sampling points; The target scale fused voxel features are determined based on the sampled voxel features and the target voxel fusion features.
4. The method of claim 3, wherein, The step of determining the target scale fused voxel features based on the sampled voxel features and the target voxel fusion features includes: The first frame offset sampling features and the second frame offset sampling features are determined based on the sampling voxel features; The offset sampling features of the first frame and the offset sampling features of the second frame are concatenated to obtain the concatenated offset sampling features; The spliced offset sampling features are cascaded and fused based on the target convolution to obtain cascaded fused features; By improving the pyramid model of the convolution module, the target voxel fusion feature and the cascaded fusion feature are scale-fused to obtain the target scale fused voxel feature.
5. An occupancy network prediction apparatus characterized by comprising: The device includes: The generation module is used to generate fast voxel features and slow voxel features based on image data of the surrounding environment of the target object. A determination module is used to determine the feature distribution structure between the fast voxel features and the slow voxel features; The fusion module is used to fuse the fast voxel features and the slow voxel features according to the feature distribution structure to obtain the target voxel fused features. The prediction module is used to determine the target scale fused voxel features based on historical multi-frame data and the target voxel fusion features, and to perform occupancy network prediction based on the target scale fused voxel features. The generation module is also used to acquire the internal and external parameters of the camera device used to capture image data of the surrounding environment of the target object; The mapping relationship between the camera space and the visual perception space is determined based on the internal parameters, the external parameters, and the conversion data from the camera device to the vehicle space. The mapping relationship between the visual perception space and the camera space is determined based on the external parameters, the transformation data from the vehicle space to the global coordinate system, the transformation data from the visual perception space to the vehicle coordinate system, and the transformation data from the global coordinate system to the vehicle coordinate system. Feature extraction is performed on the image data to obtain image feature data; Obtain target sampling points in the visual perception space and map the target sampling points to the image space; Fast voxel features are generated based on the voxel container, the mapping relationship from the camera space to the visual perception space, the image feature data, and the image space after the mapping sampling points. Slow voxel features are generated based on the mapping relationship from the visual perception space to the camera space; The fusion module is further configured to determine a voxel feature fusion strategy based on the feature distribution structure, wherein the voxel feature fusion strategy includes a receptive region fusion strategy and a focusing fusion strategy; The fast voxel features and the slow voxel features are grouped separately to obtain the first group of voxel features and the second group of voxel features. Based on the receptive region fusion strategy, the first group of voxel features are spliced together according to the target dimension using a preset merging method to obtain the first spliced voxel features. The overlapping regions of the first spliced voxel features are fused using target convolution. The fast voxel features in the second group of voxel features are compressed along the height direction; The reconstructed voxel features are determined based on the compressed second set of voxel features; The reconstructed voxel features are spliced together with the slow voxel features in the second set of voxel features to obtain the second spliced voxel features. Feature fusion is performed on the second spliced voxel features; The target voxel fusion features are determined based on the fusion results of the first spliced voxel features and the fusion results of the second spliced voxel features.
6. An occupancy network prediction device, comprising: The device includes: a memory, a processor, and a computer program stored in the memory and executable on the processor, the computer program being configured to implement the steps of the occupation network prediction method as described in any one of claims 1 to 4.
7. A storage medium, characterized by The storage medium is a computer-readable storage medium, and a computer program is stored on the storage medium. When the computer program is executed by a processor, it implements the steps of the occupation network prediction method as described in any one of claims 1 to 4.
8. A computer program product, characterised in that, The computer program product includes a computer program that, when executed by a processor, implements the steps of the occupation network prediction method as described in any one of claims 1 to 4.