Semantic map occupation prediction method and device, storage medium and electronic device

Abstract

The invention provides a semantic map occupation prediction method and device, a storage medium and an electronic device, and the method comprises the steps: training a first prediction model of a first vehicle model through employing visual annotation data; acquiring original data of a second vehicle type; analyzing the data type of the original data; if the original data is the target camera parameter of the second vehicle type, generating first training data of the second vehicle type according to the target camera parameter and the visual annotation data; if the original data is unlabeled original image data collected by a second vehicle type, generating second training data by adopting the original image data; and training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type. According to the invention, the technical problem of weak generalization ability of a vision-driven semantic map prediction model during cross-vehicle-type application in related technologies is solved, and the performance of the vision-driven semantic map prediction model during cross-domain migration is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, in particular to a prediction method and device of an occupancy semantic map, a storage medium and an electronic device. BACKGROUND

[0002] In related technologies, three-dimensional occupancy semantic map prediction based on vision can solve the problem of long-tail obstacle detection of automatic driving, especially after introducing the occupancy network, which has triggered extensive research. Although significant progress has been made in three-dimensional occupancy prediction based on vision in recent years, the first vehicle model still faces challenges in feature representation, annotation cost and practical application when migrating to the second vehicle model. Specifically: (1) it is difficult to obtain three-dimensional features from two-dimensional image features. Three-dimensional occupancy prediction based on vision aims to fully perceive and understand the three-dimensional scene through image input only, but the lack of depth and geometric information in the image is a major obstacle to learning three-dimensional feature representation. (2) Expensive fine-grained annotation. 3D occupancy prediction involves the occupancy state and semantic category of voxels, so fine-grained semantic annotation is required for each voxel, which is both time-consuming and expensive. (3) Changing working conditions. The significant feature changes of the vehicle operating environment and the diversified sensor suite equipped (including different numbers and fields of view of cameras) cause the performance of the vision-driven occupancy semantic map prediction model to decrease significantly when migrating across domains, and the generalization ability of the model is poor.

[0003] In view of the above problems in the related art, no efficient and accurate solution has been found. SUMMARY

[0004] The present application provides a prediction method and device of an occupancy semantic map, a storage medium and an electronic device to solve the technical problems in the related art.

[0005] According to one embodiment of the present application, a prediction method of an occupancy semantic map is provided, comprising: training a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; obtaining original data of a second vehicle type; analyzing a data type of the original data; if the original data is a target camera parameter of the second vehicle type, generating a target time-series multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data, and determining a three-dimensional occupancy semantic ground truth map in the visual annotation data and the target time-series multi-view image as first training data of the second vehicle type; if the original data is unannotated original image data collected by the second vehicle type, generating self-supervised data by using the original image data and the first prediction model, and determining the self-supervised data as second training data of the second vehicle type; training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

[0006] Optionally, training the first prediction model of the first vehicle type by using the visual annotation data comprises: obtaining visual annotation data collected by the first vehicle type, wherein the visual annotation data comprises source time-series multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; encoding the source time-series multi-view images in an initial model to obtain two-dimensional multi-scale features; performing visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features; performing occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; calculating a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and updating parameters of the initial model based on the first loss value to obtain the first prediction model of the first vehicle type.

[0007] Optionally, generating the target time-series multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data comprises: generating a three-dimensional Gaussian rendering model by using the visual annotation data; generating a three-dimensional Gaussian display representation of the second vehicle type in each scene by using the three-dimensional Gaussian rendering model, wherein the three-dimensional Gaussian display representation comprises geometric characteristics, color and texture characteristics of the second vehicle type in each scene; analyzing a vehicle driving trajectory in the visual annotation data, and rendering the target time-series multi-view image of the second vehicle type in the three-dimensional Gaussian display representation based on the vehicle driving trajectory and the target camera parameter.

[0008] Optionally, generating the three-dimensional Gaussian rendering model using the visual annotation data comprises: obtaining source time-series multi-view images in the visual annotation data; inputting the source time-series multi-view images into the three-dimensional Gaussian rendering model to obtain initialized three-dimensional Gaussian point clouds, wherein each of the Gaussian point clouds comprises a center point coordinate, a covariance matrix, color and opacity information; parsing source camera parameters of the source time-series multi-view images, and projecting the three-dimensional Gaussian point clouds into a two-dimensional image plane using the source camera parameters, and rendering to obtain corresponding two-dimensional images; calculating a second loss value between the two-dimensional images and the source time-series multi-view images, and optimizing parameters of the three-dimensional Gaussian point clouds through the second loss value to obtain an optimized three-dimensional Gaussian rendering model.

[0009] Optionally, generating the self-supervised data using the original image data and the first prediction model comprises: performing semantic automatic labeling and depth estimation on the original image data using a visual language large model to correspondingly generate pseudo-label semantic maps and pseudo-label depth maps; generating predicted three-dimensional occupancy semantic features of the original image data using the first prediction model; inputting the predicted three-dimensional occupancy semantic features into pre-constructed semantic prediction modules and depth prediction modules, respectively, to correspondingly obtain predicted semantic maps and predicted depth maps, and inputting the predicted three-dimensional occupancy semantic features into the first prediction model to output occupancy semantic prediction maps; determining the pseudo-label semantic maps, the pseudo-label depth maps, the predicted semantic maps, the predicted depth maps, and the occupancy semantic prediction maps as the self-supervised data.

[0010] Optionally, the second training data comprises pseudo-label semantic maps, pseudo-label depth maps, predicted semantic maps, predicted depth maps, and occupancy semantic prediction maps, and training the first prediction model using the second training data to obtain a second prediction model for the second vehicle type comprises: adjusting the first prediction model using the pseudo-label semantic maps and the predicted semantic maps to obtain a first intermediate model; adjusting the first intermediate model using the pseudo-label depth maps and the predicted depth maps to obtain a second intermediate model; and adjusting the second intermediate model using the occupancy semantic prediction maps to obtain the second prediction model.

[0011] Optionally, adjusting the first prediction model using the pseudo-label semantic maps and the predicted semantic maps to obtain a first intermediate model comprises: calculating a third loss value between the pseudo-label semantic maps and the predicted semantic maps; and optimizing semantic perception capability of the first prediction model according to the third loss value to obtain the first intermediate model.

[0012] Optionally, the first intermediate model is adjusted by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model, including: calculating a fourth loss value between the pseudo-label depth map and the predicted depth map; and optimizing the depth perception capability of the first intermediate model according to the fourth loss value to obtain a second intermediate model.

[0013] Optionally, the second intermediate model is adjusted by using the occupancy semantic prediction map to obtain a second prediction model, including: reading a first semantic category with the highest confidence and a second semantic category with the second highest confidence in each grid of the occupancy semantic prediction map; calculating a semantic difference value between the first semantic category and the second semantic category; determining whether the semantic difference value is greater than a preset threshold; if the semantic difference value is greater than the preset threshold, rendering the occupancy semantic prediction map to obtain a rendered semantic map and a rendered depth map; determining whether the rendered semantic map is aligned with the pseudo-label semantic map and whether the rendered depth map is aligned with the pseudo-label depth map; and if the rendered semantic map is aligned with the pseudo-label semantic map and the rendered depth map is aligned with the pseudo-label depth map, performing supervised training on the second intermediate model by using the original image data and the occupancy semantic prediction map to obtain a second prediction model.

[0014] According to another embodiment of the present application, a prediction device for an occupancy semantic map is provided, including: a first training module configured to train a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is configured to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; an acquisition module configured to acquire original data of a second vehicle type; an analysis module configured to analyze a data type of the original data; a generation module configured to, if the original data is target camera parameters of the second vehicle type, generate target time-series multi-view images of the second vehicle type according to the target camera parameters and the visual annotation data, and determine a three-dimensional occupancy semantic ground truth map in the visual annotation data and the target time-series multi-view images as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generate self-supervised data by using the original image data and the first prediction model, and determine the self-supervised data as second training data of the second vehicle type; and a second training module configured to train the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is configured to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

[0015] Optionally, the first training module comprises: an acquisition unit configured to acquire visual annotation data collected by the first vehicle model, wherein the visual annotation data comprises source time-series multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; a processing unit configured to encode the source time-series multi-view images in an initial model to obtain two-dimensional multi-scale features, perform visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features, and perform occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; and a training unit configured to calculate a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and update parameters of the initial model based on the first loss value to obtain a first prediction model of the first vehicle model.

[0016] Optionally, the generation module comprises: a first generation unit configured to generate a three-dimensional Gaussian rendering model using the visual annotation data; a second generation unit configured to generate a three-dimensional Gaussian display representation of the second vehicle model in each scene using the three-dimensional Gaussian rendering model, wherein the three-dimensional Gaussian display representation comprises geometric characteristics, color, and texture characteristics of the second vehicle model in each scene; and a rendering unit configured to parse a vehicle driving trajectory in the visual annotation data, and render a target time-series multi-view image of the second vehicle model in the three-dimensional Gaussian display representation based on the vehicle driving trajectory and the target camera parameters.

[0017] Optionally, the first generation unit comprises: an acquisition subunit configured to acquire source time-series multi-view images in the visual annotation data; an acquisition subunit configured to input the source time-series multi-view images into a three-dimensional Gaussian rendering model to obtain initialized three-dimensional Gaussian point clouds, wherein each of the Gaussian point clouds comprises a center point coordinate, a covariance matrix, color, and opacity information; a rendering subunit configured to parse source camera parameters of the source time-series multi-view images, project the three-dimensional Gaussian point clouds into a two-dimensional image plane using the source camera parameters, and render corresponding two-dimensional images; and a generation subunit configured to calculate a second loss value between the two-dimensional images and the source time-series multi-view images, and optimize parameters of the three-dimensional Gaussian point clouds using the second loss value to obtain an optimized three-dimensional Gaussian rendering model.

[0018] Optionally, the generating module comprises: a third generating unit configured to perform semantic automatic labeling and depth estimation on the original image data by using a visual language large model, to generate a pseudo-label semantic map and a pseudo-label depth map; a fourth generating unit configured to generate predicted three-dimensional occupancy semantic features of the original image data by using the first prediction model; a processing unit configured to input the predicted three-dimensional occupancy semantic features into a pre-constructed semantic prediction module and a depth prediction module respectively, to obtain a predicted semantic map and a predicted depth map, and input the predicted three-dimensional occupancy semantic features into the first prediction model to output an occupancy semantic prediction map; and a determining unit configured to determine the pseudo-label semantic map, the pseudo-label depth map, the predicted semantic map, the predicted depth map, and the occupancy semantic prediction map as self-supervised data.

[0019] Optionally, the second training data comprises a pseudo-label semantic map, a pseudo-label depth map, a predicted semantic map, a predicted depth map, and an occupancy semantic prediction map, and the second training module comprises: a first adjusting unit configured to adjust the first prediction model by using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model; a second adjusting unit configured to adjust the first intermediate model by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model; and a third adjusting unit configured to adjust the second intermediate model by using the occupancy semantic prediction map to obtain a second prediction model.

[0020] Optionally, the first adjusting unit comprises: a calculating sub-unit configured to calculate a third loss value between the pseudo-label semantic map and the predicted semantic map; and an optimizing sub-unit configured to optimize semantic perception ability of the first prediction model according to the third loss value to obtain the first intermediate model.

[0021] Optionally, the second adjusting unit comprises: a calculating sub-unit configured to calculate a fourth loss value between the pseudo-label depth map and the predicted depth map; and an optimizing sub-unit configured to optimize depth perception ability of the first intermediate model according to the fourth loss value to obtain the second intermediate model.

[0022] Optionally, the third adjusting unit comprises: a reading subunit, configured to read a first semantic category with the highest confidence and a second semantic category with the second highest confidence in each grid of the occupancy semantic prediction map; a calculating subunit, configured to calculate a semantic difference between the first semantic category and the second semantic category; a judging subunit, configured to judge whether the semantic difference is greater than a preset threshold; a rendering subunit, configured to render the occupancy semantic prediction map to obtain a rendered semantic map and a rendered depth map if the semantic difference is greater than the preset threshold; the judging subunit is further configured to judge whether the rendered semantic map is aligned with the pseudo-label semantic map and whether the rendered depth map is aligned with the pseudo-label depth map; and a training subunit, configured to perform supervised training on the second intermediate model by using the original image data and the occupancy semantic prediction map to obtain a second prediction model if the rendered semantic map is aligned with the pseudo-label semantic map and the rendered depth map is aligned with the pseudo-label depth map.

[0023] According to another aspect of the embodiments of the present application, a storage medium is also provided, which comprises a stored program, and the program performs the above steps when running.

[0024] According to another aspect of the embodiments of the present application, an electronic device is also provided, which comprises a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; the memory is used to store a computer program; and the processor is used to perform the steps in the above method by running the program stored in the memory.

[0025] The embodiments of the present application also provide a computer program product comprising instructions which, when the program is executed on a computer, cause the computer to carry out the steps of the above method.

[0026] The present application has the following beneficial effects: 1. A three-dimensional Gaussian rendering module is introduced to render a time-series multi-view image that meets target camera parameters, which can be used to optimize model performance, thereby eliminating the influence of target camera parameters on model performance and improving the generalization performance of the model under the second vehicle type. 2. A semantic prediction module and a depth prediction module are introduced to process three-dimensional occupancy semantic features under the second vehicle type, and the multi-view pseudo-label semantic map and depth map obtained through the visual language model are aligned for supervision, thereby improving the semantic and depth perception generalization performance of the model for the second vehicle type image. 3. Two pseudo-label screening principles are proposed, which can screen out high-quality occupancy semantic map pseudo-labels for the occupancy semantic map prediction model under the second vehicle type, thereby improving the generalization performance of the model under the second vehicle type, saving the cost of manual annotation of data, and improving the data utilization rate. BRIEF DESCRIPTION OF DRAWINGS

[0027] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings: Figure 1 is a hardware structure block diagram of a car according to an embodiment of the application; Figure 2 is a flow chart of a prediction method of an occupancy semantic map according to an embodiment of the application; Figure 3 is a training schematic diagram of an occupancy semantic map prediction model of a source domain in an embodiment of the application; Figure 4 is a training optimization schematic diagram of a three-dimensional Gaussian display representation of a certain scene of a source domain in an embodiment of the application; Figure 5 is a training schematic diagram of an occupancy semantic map prediction model of a target domain in an embodiment of the application; Figure 6 is a structure block diagram of a prediction device of an occupancy semantic map according to an embodiment of the application. DETAILED DESCRIPTION

[0028] In order to make the technical personnel in the art better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those of ordinary skill in the art without creative labor should fall within the scope of protection of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0029] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, product or device including a series of steps or units does not necessarily limit to those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to these processes, methods, products or devices.

[0030] Embodiment 1 The method provided in the embodiment of the application can be executed in a car, a server, a processor, a safety controller, an automatic driving / assisted driving / smart driving controller or similar processing device. Taking the car as an example, Figure 1 is a hardware structure block diagram of a car according to an embodiment of the application. As shown in Figure 1 , the car can include one or more (only one is shown in Figure 1 ) processors 102 (the processor 102 can include, but is not limited to, a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the car can further include a transmission device 106 for communication function and an input / output device 108. Those skilled in the art can understand that Figure 1 the structure shown is only schematic, which does not limit the structure of the car. For example, the car can include more or fewer components than those shown in Figure 1 , or have a different configuration from that shown in Figure 1 .

[0031] The memory 104 can be used to store car programs, such as software programs of application software and modules, for example, a car program corresponding to the prediction method of an occupancy semantic map of a car according to an embodiment of the application. The processor 102 executes various functional applications and data processing by running the car program stored in the memory 104, that is, implements the method described above. The memory 104 can include a high-speed random access memory, and can further include a non-volatile memory, such as one or more magnetic storage devices, flash memories or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, and these remote memories can be connected to the car through a network. Examples of the network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0032] The transmission device 106 is used to receive or send data via a network. Specific examples of the network can include a wireless network provided by a communication provider of the car. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, NIC for short), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (Radio Frequency, RF for short) module, which is used to communicate with the Internet in a wireless manner.

[0033] In the embodiment, a prediction method of an occupancy semantic map is provided, Figure 2 is a flowchart of a prediction method of an occupancy semantic map according to an embodiment of the application, as shown inFigure 2 As shown, the flow includes the following steps: Step S201, training a first prediction model of a first vehicle type using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; Optionally, the visual annotation data is a time-series multi-view image collected by a camera of a sample vehicle of the first vehicle type, and a corresponding three-dimensional occupancy semantic ground truth map.

[0034] The first prediction model and the second prediction model of the embodiment are occupancy semantic map models applied to the first vehicle type and the second vehicle type respectively. The first vehicle type belongs to the source domain of the occupancy semantic map model, and the second vehicle type belongs to the target domain of the occupancy semantic map model. The scheme of the embodiment is used to realize the generalization ability of the occupancy semantic map model from the source domain to the target domain.

[0035] The occupancy semantic map of the embodiment is a three-dimensional visual representation of a vehicle driving scene in the field of intelligent driving perception. The entire driving scene is divided into minimum unit cubic grids, and each grid is assigned a semantic label.

[0036] Step S202, obtaining original data of a second vehicle type; Optionally, the original data is unlabeled data from the second vehicle type, which can be target camera parameters of the second vehicle type, or unlabeled original image data collected by a camera of the vehicle of the second vehicle type.

[0037] Step S203, analyzing the data type of the original data; Step S204, if the original data is target camera parameters of the second vehicle type, generating target time-series multi-view images of the second vehicle type according to the target camera parameters and the visual annotation data, and determining the target time-series multi-view images and the three-dimensional occupancy semantic ground truth map in the visual annotation data as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generating self-supervised data using the original image data and the first prediction model, and determining the self-supervised data as second training data of the second vehicle type; Step S205, training the first prediction model using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

[0038] By the above steps, the first prediction model of the first vehicle type is trained by using the visual annotation data, wherein the first prediction model is used to predict the occupancy semantic map of the first vehicle type according to the visual image of the first vehicle type; the original data of the second vehicle type is obtained; the data type of the original data is analyzed; if the original data is the target camera parameter of the second vehicle type, the target time sequence multi-view image of the second vehicle type is generated according to the target camera parameter and the visual annotation data, and the target time sequence multi-view image and the three-dimensional occupancy semantic ground truth map in the visual annotation data are determined as the first training data of the second vehicle type; if the original data is the unlabeled original image data collected by the second vehicle type, the self-supervised data is generated by using the original image data and the first prediction model, and the self-supervised data is determined as the second training data of the second vehicle type; the first prediction model is trained by using the first training data and / or the second training data, and the second prediction model of the second vehicle type is obtained, wherein the second prediction model is used to predict the occupancy semantic map of the second vehicle type according to the visual image of the second vehicle type, the generalization of semantic feature fusion is enhanced, the generalization ability of deep feature fusion is improved, and the generalization of multi-source feature utilization is optimized, so that the generalization ability of the model in the target domain is comprehensively improved, the technical problem of weak generalization ability of the visual-driven occupancy semantic map prediction model in cross-vehicle application in the related art is solved, and the performance of the visual-driven occupancy semantic map prediction model in cross-domain migration is ensured.

[0039] In the embodiment, training the first prediction model of the first vehicle type by using the visual annotation data comprises: obtaining visual annotation data collected by the first vehicle type, wherein the visual annotation data comprises source time sequence multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; encoding the source time sequence multi-view images in an initial model to obtain two-dimensional multi-scale features; performing visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features; performing occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; calculating a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and updating parameters of the initial model based on the first loss value to obtain the first prediction model of the first vehicle type.

[0040] Figure 3 For the training schematic diagram of the occupancy semantic map prediction model of the source domain in the embodiment, the implementation process of the source domain training is divided into three modules: an image feature encoder module, a visual conversion module, and an occupancy semantic decoding module, the visual conversion module outputs three-dimensional occupancy semantic features, the occupancy semantic decoding module outputs an occupancy semantic prediction map, and the training is performed in combination with an occupancy semantic ground truth module, so that the modules are connected with each other and jointly complete the complete process from a two-dimensional image to a three-dimensional occupancy semantic prediction. The process comprises: First, the input multi-view image set is processed by the image feature encoder module to extract features. This module uses existing image feature extraction networks, such as ResNet, Swin Transformer, and ConvNeXt, to extract deep-level feature information from each image. Each image is encoded into a high-dimensional feature vector , which can be represented by the formula:

[0041] In the formula, represents the image feature extraction network. Through this process, the features of all images are combined into a feature matrix , providing high-quality feature input for subsequent three-dimensional space mapping. The implementation of this module ensures that the semantic and geometric information of each pixel in the image is preserved and transmitted in a structured form.

[0042] Next, the visual conversion module maps two-dimensional image features to three-dimensional space through learning to generate three-dimensional occupancy semantic feature representations. This module combines geometric projection and depth estimation methods to convert the feature matrix into source domain three-dimensional occupancy semantic features . The specific formula is:

[0043] where, includes the intrinsic and extrinsic parameters of the camera, which defines the geometric relationship from the image plane to the three-dimensional space, represents the image depth information obtained through the depth estimation method. The depth estimation method can use a publicly available pre-trained model, such as ZeroDepth. Through such projection, each three-dimensional coordinate in space can be accurately mapped to a feature value . This module realizes accurate mapping from the feature plane to the three-dimensional space, providing spatialized semantic feature representation for occupancy semantic decoding.

[0044] Finally, the occupancy semantic decoding module is used to analyze the three-dimensional occupancy semantic features and segment them into different semantic categories. This module uses a three-dimensional convolutional network (3D CNN) to perform semantic classification on each three-dimensional occupancy semantic feature and predict its class probability . The specific formula is:

[0045] The output semantic distribution probability This is a semantic segmentation result in three-dimensional space, where each occupied value represents a raster position. The predicted probabilities belong to different categories. Through this process, the semantic structure of complex scenes is clearly presented, facilitating subsequent analysis and application.

[0046] If a model trained in the source domain is directly applied to occupancy semantic map prediction for data in other target domains, the model performance will drop significantly. Therefore, it is necessary to improve the generalization performance of the model in the target domain.

[0047] In one implementation scenario of this embodiment, where only the target camera parameters of the second vehicle model are available but the original image data is unavailable, generating a target temporal multi-view image of the second vehicle model based on the target camera parameters and the visual annotation data includes: generating a 3D Gaussian rendering model using the visual annotation data; generating a 3D Gaussian display representation of the second vehicle model in each scene using the 3D Gaussian rendering model, wherein the 3D Gaussian display representation includes the geometric characteristics, color, and texture characteristics of the second vehicle model in each scene; parsing the vehicle driving trajectory in the visual annotation data, and rendering the target temporal multi-view image of the second vehicle model in the 3D Gaussian display representation based on the vehicle driving trajectory and the target camera parameters.

[0048] In one example, generating a 3D Gaussian rendering model using the visually labeled data includes: acquiring source temporal multi-view images from the visually labeled data; inputting the source temporal multi-view images into the 3D Gaussian rendering model to obtain an initialized 3D Gaussian point cloud, wherein each Gaussian point cloud contains center point coordinates, covariance matrix, color, and opacity information; parsing the source camera parameters of the source temporal multi-view images, and using the source camera parameters to project the 3D Gaussian point cloud onto a 2D image plane, and rendering the corresponding 2D image; calculating a second loss value between the 2D image and the source temporal multi-view images, and optimizing the parameters of the 3D Gaussian point cloud using the second loss value to obtain an optimized 3D Gaussian rendering model.

[0049] Figure 4 This is a schematic diagram illustrating the training and optimization of a 3D Gaussian display representation of a scene in the source domain in an embodiment of the present invention, such as... Figure 4 As shown, this step is achieved through a Gaussian point cloud initialization module, a projection module, an adaptive density control module, and a differentiable rasterization module. The projection module also needs to consider camera parameters. The differentiable rasterization module generates a Gaussian-rasterized six-view image corresponding to the original six-view image. The first step achieves efficient rendering of the 3D scene through a purely vision-driven approach. The specific implementation process is as follows: Firstly, the time sequence multi-view images of a certain scene are input into the Gaussian point cloud initialization module. Through the SfM algorithm in the module, common features in the images can be identified, and the motion path of the camera and the three-dimensional point cloud structure of the scene can be estimated. After obtaining the three-dimensional point cloud structure, the Gaussian point cloud initialization module converts each point cloud into a three-dimensional Gaussian point cloud, which is used as the basis for training model optimization. Each three-dimensional Gaussian point cloud is composed of a center point coordinate, a covariance matrix, color information and opacity information, which is represented as:

[0050] In the formula, is the three-dimensional space position of each Gaussian point , which defines the position of the point in space; is the three-dimensional Gaussian point space distribution shape and size, which is used to describe the distribution range and directionality of the point in the three-dimensional space; is the color information contained in each Gaussian point , which represents the RGB color value, used for color expression in rendering; is the transparency weight of each Gaussian point, represented by , which is used to control the contribution intensity of the point to the rendering result.

[0051] Secondly, the three-dimensional Gaussian points are projected from the world coordinate system to the two-dimensional image plane by using the projection module. Based on the camera's intrinsic and extrinsic parameters, the projection formula of the Gaussian point cloud is represented as:

[0052] In the formula, is the projection coordinate of the three-dimensional Gaussian point on the two-dimensional plane; is the projected covariance matrix; is the perspective transformation matrix of the camera, which converts the point from the world coordinate system to the camera space, where is the rotation matrix, is the translation vector; is the Jacobian matrix, which is the affine approximation of the projection transformation, used to convert the point from the camera space to the two-dimensional image plane.

[0053] Then, the Gaussian points on the two-dimensional image plane are input into the differentiable rasterization module for image rendering. During rendering, the color value of each pixel is obtained by the weighted contribution of all projected Gaussians:

[0054] In the formula, is the pixel coordinate in the image plane; is the coordinate vector of the corresponding pixel; , is the two-dimensional projection coordinate and two-dimensional covariance matrix of the Gaussian point cloud. The rendered image is compared with the ground truth image, and the rendering loss is calculated by the following formula:

[0055] wherein, represents the rendering loss; represents the image ground truth; represents the rendered image; represents the structural similarity calculation function; , represents the weight of the loss.

[0056] Finally, when the Gaussian is initialized and a certain number of iterations, for example, 1000 iterations, are performed, the adjustment of the Gaussian point cloud density is started. The adjustment principle is to judge whether the gradient of each three-dimensional Gaussian point cloud is greater than a certain threshold. For example, when the gradient of a certain Gaussian point cloud is greater than or equal to 200, the Gaussian point cloud is operated and processed. The operation and processing includes cloning, segmentation and pruning. The specific operation needs to be determined according to the conditions. When the scale of the Gaussian point cloud is large, the Gaussian point cloud is segmented, that is, the Gaussian point is divided into multiple sub-Gaussian points (for example, the center is uniformly offset), and the scale and transparency of each sub-Gaussian are reduced; when the scale is small, it indicates that the Gaussian point cloud distribution range is too small and may not effectively cover the area in the scene, so the cloning operation is performed, that is, multiple copies of the Gaussian point cloud are copied, and the center position is adjusted; when the transparency of the Gaussian point is less than a certain threshold (for example, 0.005), it is considered that the Gaussian point cloud has little contribution to the rendering result, and the pruning operation can be performed, that is, it is removed. Through this dynamic adjustment, the conditions for adjusting the Gaussian point cloud density are completed, which can optimize the number and distribution range of Gaussian points in the rendering process, improve the rendering quality, and save computing resources. The scale of the Gaussian point cloud is calculated as follows:

[0057] wherein, , , is the diagonal element of the covariance matrix of the Gaussian point cloud.

[0058] The second step is to obtain multi-view images that meet the camera parameters of the target domain. Through the cooperative action of the Gaussian point cloud initialization module, the projection module, the adaptive density control module and the differentiable rasterization module, the three-dimensional Gaussian display representation of each scene is generated. On this basis, when the camera parameters of the target domain are given, the multi-view images that meet the camera parameters of the target domain can be rendered. Specifically, in the embodiment, the Gaussian point cloud is initialized by the Gaussian point cloud initialization module, and the projection module is used to project the Gaussian point cloud into the image plane to obtain the two-dimensional projection coordinate and the two-dimensional covariance matrix of the Gaussian point cloud. Then, the adaptive density control module is used to adjust the density of the Gaussian point cloud, and the differentiable rasterization module is used to render the image. The specific process is as follows. Figure 4The camera parameter shown in the middle is replaced by the camera parameter of the target domain, and the three-dimensional Gaussian point is mapped to the two-dimensional image plane according to the target domain camera parameter by using the projection module. Then, the projection result is rendered by the differentiable rasterization module to generate a time sequence multi-view image conforming to the target domain camera parameter. The time sequence multi-view image generated above is input into the semantic occupancy map prediction model to optimize the performance of the model. By adjusting the matching of the generated image and the target domain camera parameter, the negative influence of the target domain camera parameter on the model performance can be effectively eliminated, thereby improving the generalization performance of the model in the target domain.

[0059] In another implementation scenario of the embodiment, the original data is unlabeled original image data collected by the second vehicle model, and generating self-supervised data using the original image data and the first prediction model includes: performing semantic automatic labeling and depth estimation on the original image data using a visual language large model to correspondingly generate a pseudo-label semantic map and a pseudo-label depth map; generating predicted three-dimensional occupancy semantic features of the original image data using the first prediction model; inputting the predicted three-dimensional occupancy semantic features into pre-constructed semantic prediction modules and depth prediction modules respectively to correspondingly obtain a predicted semantic map and a predicted depth map, and inputting the predicted three-dimensional occupancy semantic features into the first prediction model to output an occupancy semantic prediction map; and determining the pseudo-label semantic map, the pseudo-label depth map, the predicted semantic map, the predicted depth map, and the occupancy semantic prediction map as self-supervised data.

[0060] The implementation scenario proposes a method based on large model pseudo-labeling and multi-stage optimization. On the basis of training of the aforementioned occupancy semantic map prediction model (first prediction model), three modules are added, which are a visual language large model for automatically labeling the semantics and depth of an image, a semantic prediction module for predicting semantic information of an image, and a depth prediction module for predicting depth information of an image. By combining semantic automatic labeling, depth estimation, and pseudo-label screening, the adaptability of the model trained on source domain data to the target domain is gradually improved.

[0061] In this implementation scenario, the second training data includes a pseudo-label semantic map, a pseudo-label depth map, a predicted semantic map, a predicted depth map, and an occupancy semantic prediction map, and training the first prediction model using the second training data to obtain a second prediction model of the second vehicle model includes: adjusting the first prediction model using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model; adjusting the first intermediate model using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model; and adjusting the second intermediate model using the occupancy semantic prediction map to obtain the second prediction model.

[0062] In an embodiment, the first prediction model is adjusted using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model, including: calculating a third loss value between the pseudo-label semantic map and the predicted semantic map; optimizing semantic perception ability of the first prediction model according to the third loss value to obtain the first intermediate model.

[0063] The multi-view image of the target domain is automatically semantically labeled by using the visual language large model to obtain a pseudo semantic map of the multi-view image; the multi-view image of the target domain is processed by the image feature encoding module and the visual conversion module to obtain three-dimensional occupancy semantic features; the three-dimensional occupancy semantic features are input into the semantic prediction module to obtain a predicted semantic map of the multi-view image; the predicted semantic map is compared with the multi-view pseudo semantic map to calculate a semantic loss; the semantic loss is optimized by back propagation to optimize the occupancy semantic map prediction model, which can improve the semantic perception generalization performance of the model in the target domain.

[0064] In an embodiment, the first intermediate model is adjusted using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model, including: calculating a fourth loss value between the pseudo-label depth map and the predicted depth map; optimizing depth perception ability of the first intermediate model according to the fourth loss value to obtain the second intermediate model.

[0065] The multi-view image of the target domain is automatically semantically labeled by using the visual language large model to obtain a pseudo semantic map of the multi-view image; the multi-view image of the target domain is processed by the image feature encoding module and the visual conversion module to obtain three-dimensional occupancy semantic features; the three-dimensional occupancy semantic features are input into the semantic prediction module to obtain a predicted semantic map of the multi-view image; the predicted semantic map is compared with the multi-view pseudo semantic map to calculate a semantic loss; the semantic loss is optimized by back propagation to optimize the occupancy semantic map prediction model, which can improve the semantic perception generalization performance of the model in the target domain.

[0066] In an embodiment, the second intermediate model is adjusted using the occupancy semantic prediction map to obtain a second prediction model, including: reading a first semantic class with the highest confidence and a second semantic class with the second highest confidence in each grid of the occupancy semantic prediction map; calculating a semantic difference value between the first semantic class and the second semantic class; judging whether the semantic difference value is greater than a preset threshold; if the semantic difference value is greater than the preset threshold, using the occupancy semantic prediction map to render a rendered semantic map and a rendered depth map; judging whether the rendered semantic map is aligned with the pseudo-label semantic map and whether the rendered depth map is aligned with the pseudo-label depth map; if the rendered semantic map is aligned with the pseudo-label semantic map and the rendered depth map is aligned with the pseudo-label depth map, using the original image data and the occupancy semantic prediction map to supervise training of the second intermediate model to obtain the second prediction model.

[0067] The target domain three-dimensional occupancy semantic feature is input into the occupancy semantic decoding module trained in the source domain, to obtain an occupancy semantic prediction map in the target domain, high-quality pseudo-occupancy grid labels are screened out according to two principles, and are added to the next iteration cycle, and a supervision training is performed on the occupancy semantic prediction model of the second vehicle type.

[0068] The first pseudo-label screening principle is a class confidence difference. The specific implementation manner is as follows: first, the difference between the first class (the highest confidence class) and the second class (the second highest confidence class) of each grid in the occupancy semantic prediction map is calculated; then, a threshold is set, when the difference is greater than the threshold, it is considered that the prediction result is reliable, and can be used as a pseudo-label to train the model. The second pseudo-label screening principle is a semantic and spatial consistency constraint. The specific implementation manner is as follows: the occupancy semantic prediction map is rendered into a multi-view image rendering semantic map and a rendering depth map through a rendering module; the rendering semantic map and the rendering depth map are respectively aligned with the pseudo semantic map and the depth map obtained through the visual language large model to select pseudo labels with consistent semantics and space.

[0069] Figure 5 is a training schematic diagram of the target domain occupancy semantic map prediction model in the embodiment of the present application, on the basis of Figure 3 , three modules are added, which are a visual language large model for automatically labeling the semantics and depth of the image, a semantic prediction module for predicting the semantic information of the image, and a depth prediction module for predicting the depth information of the image. The training process includes: The first step is to add the multi-view pseudo semantic map to the occupancy semantic map prediction model trained in the source domain. Specifically, first, the visual language large model (for example, SAM 2) is used to automatically label the semantics of the multi-view image in the target domain, to generate high-quality pseudo semantic maps. These pseudo semantic maps serve as approximate semantic labels of the target domain image, and provide a supervision signal for the training of the model. Specifically, the multi-view image set in the target domain is input into the visual language large model for pixel-by-pixel semantic segmentation, to generate a pseudo semantic map set corresponding to the multi-view image. The calculation formula is as follows:

[0070] These pseudo semantic maps can provide high-quality supervision signals for training in the case that there is a lack of real labeled data in the target domain. Next, the multi-view image in the target domain is processed through the image feature encoding module and the visual conversion module, to generate target domain three-dimensional occupancy semantic features , and the specific implementation is consistent with that in the source domain, and will not be repeated. Finally, the generated three-dimensional occupancy semantic features are input into the semantic prediction module , generate a predicted semantic map of the multi-view image :

[0071] Subsequently, the semantic map output by the semantic prediction module is compared pixel by pixel with the corresponding pseudo semantic map to calculate a semantic loss , which measures the difference between the predicted result and the pseudo label. Through back propagation, the semantic loss optimizes the parameters of the image feature encoding module, the visual conversion module, and the semantic prediction module. Through the above steps, the multi-view pseudo semantic map is introduced into the training of the semantic map prediction model, and is optimized in combination with the semantic loss, effectively improving the semantic perception ability of the model in the target domain and significantly enhancing the generalization performance of the model.

[0072] In the second step, the multi-view depth map is added to the training of the aforementioned semantic map prediction model to improve the depth perception generalization performance of the model in the target domain. Specifically, first, a visual language large model (such as DepthAnythingV2) is used to estimate the depth of the multi-view image in the target domain to generate high-quality multi-view depth maps. These depth maps serve as approximate geometric labels for the target domain images, providing a supervision signal for the training of the model. Specifically, a set of multi-view images in the target domain is input into the visual language large model to perform pixel-by-pixel depth estimation and generate a set of depth maps corresponding to the multi-view images . The calculation formula is as follows:

[0073] These depth maps can effectively describe the scene geometry information of the target domain images, providing geometric constraints for subsequent model optimization. Next, the multi-view images in the target domain are processed by the image feature encoding module and the visual conversion module to generate three-dimensional semantic features of the target domain. The specific implementation here is consistent with the training of the source domain and will not be repeated. Finally, the generated three-dimensional semantic features are input into the depth prediction module to generate predicted depth maps of the multi-view images :

[0074] Subsequently, the depth map output by the depth prediction module is compared pixel by pixel with the depth map estimated by the large model to calculate a depth loss , which measures the difference between the predicted result and the depth map estimated by the large model. Through back propagation, the depth loss The parameters of the image feature encoding module, the visual conversion module, and the semantic prediction module are optimized. Through the above steps, the multi-view depth map is introduced into the training of the occupancy semantic map prediction model, and the geometric perception ability of the model is optimized in combination with the depth loss, thereby effectively improving the understanding of the geometric information of the target domain scene by the model and significantly enhancing the depth perception generalization performance of the model.

[0075] In the third step, the obtained pseudo-occupancy grid label is added to the training of the aforementioned occupancy semantic map prediction model to improve the occupancy semantic prediction ability of the model in the target domain. The implementation content is described below in combination with specific steps and formulas: First, the target domain three-dimensional occupancy semantic features generated in the aforementioned step are input into the occupancy semantic decoding module trained in the source domain to generate an occupancy semantic prediction map in the target domain .

[0076]

[0077] wherein, is the occupancy semantic prediction map in the target domain, is the spatial dimension of the three-dimensional grid, is the number of categories, represents the prediction probability of the grid position belonging to the category . Next, according to the two pseudo-label screening principles, high-quality pseudo-occupancy grid labels are screened from the target domain occupancy semantic prediction map, and are added to the next iteration cycle to supervise the training of the target domain occupancy semantic map prediction model.

[0078] The first pseudo-label screening principle is based on the category confidence difference, and selects the grid with a higher prediction confidence as the pseudo-label. The specific implementation manner is as follows: For each grid , the difference between the first category (the category with the highest prediction probability) and the second category (the category with the second highest prediction probability) is calculated:

[0079] wherein, , .

[0080] A threshold value is set, and when , it is considered that the prediction result of the grid is reliable and can be added to the training set as a pseudo-label. The screening result is:

[0081] wherein, ​represents the pseudo label screened according to the confidence difference value, represents that the grid is not selected.

[0082] The second pseudo label screening principle is based on semantic and spatial consistency constraints, and is used to screen high-quality grid pseudo labels from the pseudo semantic map and the pseudo depth Figure 1 map generated by the visual language large model. This principle uses the alignment of the occupancy semantic prediction map and the pseudo label generated in the image space in terms of semantics and geometry to ensure that the screened pseudo labels have higher reliability.

[0083] Specifically, the target domain occupancy semantic prediction map is rendered into an image space semantic map and a depth map by a rendering module , denoted as rendered semantic map and rendered depth map , respectively.

[0084] The rendering module can map each occupancy grid in three-dimensional space to a pixel in image space, and ensure a one-to-one correspondence between three-dimensional grids and image pixels. For a certain grid in the target domain , its rendering result corresponds to a certain pixel in the image space. For each pixel , if the rendered semantic category and depth value are completely consistent with the pseudo semantic map and the pseudo depth map generated by the visual language large model, then the corresponding three-dimensional occupancy grid is considered to be a credible pseudo label. The specific screening conditions are:

[0085] The grid label that meets the above conditions will be selected as a pseudo label, and its semantic category is:

[0086] Otherwise, the grid label will be discarded and not used as a pseudo label for training.

[0087] Finally, the pseudo labels obtained by the two screening principles and are integrated to obtain a high-quality pseudo label set :

[0088] The screened pseudo labels The occupancy semantic map prediction model is supervised and optimized to be added to the next iteration training, so as to improve the occupancy semantic prediction capability of the model on the target domain scene.

[0089] By using the scheme of the embodiment, a pure visual occupancy semantic map prediction generalization framework suitable for automatic driving is constructed through source domain training and target domain image generation based on three-dimensional Gaussian rendering, and the adaptability and prediction accuracy of the model in multi-domain and multi-vehicle scenes are effectively improved, thereby providing a strong guarantee for the safety and reliability of the automatic driving system. First, the initial model training is completed in the source domain using the labeled data, so as to ensure that the model can effectively learn the visual features, depth information and semantic relationship of the source domain, thereby laying a foundation for subsequent generalization. Second, in the case that only the camera parameters of the target domain are available without original data, three-dimensional Gaussian rendering is used to generate new visual images conforming to the camera parameters of the target domain, and the model is trained to adapt to different perspective changes. Finally, in the case that the original data of the target domain is not manually labeled, pseudo semantic maps and depth maps are generated by combining a large model, multi-scale features are extracted and a unified three-dimensional voxel feature map is constructed, pseudo labels are generated through confidence screening, and self-supervised training is performed, so as to improve the generalization performance of the model in multi-domain and multi-vehicle scenes, effectively reduce the dependence on manual labeling, and enhance the robustness and adaptability of the model.

[0090] The scheme of the embodiment fully utilizes the unlabeled data of the target domain, realizes the generalization enhancement of semantic feature fusion, the generalization capability improvement of depth feature fusion, and the generalization optimization of multi-source feature utilization, and thereby comprehensively improves the generalization capability of the model in the target domain.

[0091] From the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be implemented by means of software and a general hardware platform, and of course, it can also be implemented by hardware, but in many cases, the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, or an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device) to execute the method described in each embodiment of the present application.

[0092] Embodiment 2 In this embodiment, an occupancy semantic map prediction device is also provided, which is used to implement the above embodiments and preferred embodiments, and has been described above. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware or a combination of software and hardware is also possible and contemplated.

[0093] Figure 6 is a structure block diagram of a prediction device for an occupancy semantic map according to an embodiment of the present application, as shown in the figure, the device comprises: Figure 6 a first training module 61, configured to train a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; an acquisition module 62, configured to acquire original data of a second vehicle type; an analysis module 63, configured to analyze a data type of the original data; a generation module 64, configured to, if the original data is a target camera parameter of the second vehicle type, generate a target time-series multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data, and determine a three-dimensional occupancy semantic ground truth map in the target time-series multi-view image and the visual annotation data as first training data of the second vehicle type; if the original data is unannotated original image data collected by the second vehicle type, generate self-supervised data by using the original image data and the first prediction model, and determine the self-supervised data as second training data of the second vehicle type; a second training module 65, configured to train the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

[0094] Optionally, the first training module comprises: an acquisition unit, configured to acquire visual annotation data collected by a first vehicle type, wherein the visual annotation data comprises source time-series multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; a processing unit, configured to encode the source time-series multi-view images in an initial model to obtain two-dimensional multi-scale features; perform visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features; perform occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; and a training unit, configured to calculate a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and update parameters of the initial model based on the first loss value to obtain a first prediction model of the first vehicle type.

[0095] ​Optionally, the generating module comprises: a first generating unit configured to generate a three-dimensional Gaussian rendering model using the visual annotation data; a second generating unit configured to generate a three-dimensional Gaussian display representation of the second vehicle model in each scene using the three-dimensional Gaussian rendering model, wherein the three-dimensional Gaussian display representation comprises geometric characteristics, color and texture characteristics of the second vehicle model in each scene; and a rendering unit configured to parse a vehicle driving track in the visual annotation data, and render a target time-series multi-view image of the second vehicle model in the three-dimensional Gaussian display representation based on the vehicle driving track and the target camera parameter.

[0096] Optionally, the first generating unit comprises: an obtaining subunit configured to obtain a source time-series multi-view image in the visual annotation data; an obtaining subunit configured to input the source time-series multi-view image into a three-dimensional Gaussian rendering model to obtain an initialized three-dimensional Gaussian point cloud, wherein each Gaussian point cloud comprises a center point coordinate, a covariance matrix, color and opacity information; a rendering subunit configured to parse a source camera parameter of the source time-series multi-view image, and project the three-dimensional Gaussian point cloud into a two-dimensional image plane using the source camera parameter, and render a corresponding two-dimensional image; and a generating subunit configured to calculate a second loss value between the two-dimensional image and the source time-series multi-view image, and optimize parameters of the three-dimensional Gaussian point cloud using the second loss value to obtain an optimized three-dimensional Gaussian rendering model.

[0097] Optionally, the generating module comprises: a third generating unit configured to perform semantic automatic annotation and depth estimation on the original image data using a visual language large model, to correspondingly generate a pseudo-label semantic map and a pseudo-label depth map; a fourth generating unit configured to generate a predicted three-dimensional occupancy semantic feature of the original image data using the first prediction model; a processing unit configured to input the predicted three-dimensional occupancy semantic feature into a pre-constructed semantic prediction module and a depth prediction module, respectively, to correspondingly obtain a predicted semantic map and a predicted depth map, and input the predicted three-dimensional occupancy semantic feature into the first prediction model to output an occupancy semantic prediction map; and a determining unit configured to determine the pseudo-label semantic map, the pseudo-label depth map, the predicted semantic map, the predicted depth map, and the occupancy semantic prediction map as self-supervised data.

[0098] Optionally, the second training data comprises a pseudo-label semantic map, the pseudo-label depth map, the predicted semantic map, the predicted depth map, and the occupancy semantic prediction map, and the second training module comprises: a first adjusting unit configured to adjust the first prediction model by using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model; a second adjusting unit configured to adjust the first intermediate model by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model; and a third adjusting unit configured to adjust the second intermediate model by using the occupancy semantic prediction map to obtain a second prediction model.

[0099] Optionally, the first adjusting unit comprises: a calculating sub-unit configured to calculate a third loss value between the pseudo-label semantic map and the predicted semantic map; and an optimizing sub-unit configured to optimize semantic perception capability of the first prediction model according to the third loss value to obtain the first intermediate model.

[0100] Optionally, the second adjusting unit comprises: a calculating sub-unit configured to calculate a fourth loss value between the pseudo-label depth map and the predicted depth map; and an optimizing sub-unit configured to optimize depth perception capability of the first intermediate model according to the fourth loss value to obtain the second intermediate model.

[0101] Optionally, the third adjusting unit comprises: a reading sub-unit configured to read a first semantic class with the highest confidence and a second semantic class with the second highest confidence of each grid in the occupancy semantic prediction map; a calculating sub-unit configured to calculate a semantic difference value between the first semantic class and the second semantic class; a judging sub-unit configured to judge whether the semantic difference value is greater than a preset threshold; a rendering sub-unit configured to, if the semantic difference value is greater than the preset threshold, render the occupancy semantic prediction map to obtain a rendered semantic map and a rendered depth map; the judging sub-unit is further configured to judge whether the rendered semantic map is aligned with the pseudo-label semantic map and whether the rendered depth map is aligned with the pseudo-label depth map; and a training sub-unit configured to, if the rendered semantic map is aligned with the pseudo-label semantic map and the rendered depth map is aligned with the pseudo-label depth map, perform supervised training on the second intermediate model by using the original image data and the occupancy semantic prediction map to obtain the second prediction model.

[0102] It should be noted that each of the above modules can be implemented by software or hardware, and for the latter, the following implementation manners can be used, but are not limited thereto: all of the above modules are located in the same processor; or the above modules are located in different processors in any combination.

[0103] Embodiment 3 The embodiment of the present application also provides a storage medium, wherein the storage medium stores a computer program, and the computer program is configured to execute the steps in any one of the method embodiments when running.

[0104] Optionally, in the embodiment, the storage medium is configured to store a computer program for executing the following steps: S1, training a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; S2, obtaining original data of a second vehicle type; S3, analyzing a data type of the original data; S4, if the original data is a target camera parameter of the second vehicle type, generating a target time sequence multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data, and determining the target time sequence multi-view image and a three-dimensional occupancy semantic true value map in the visual annotation data as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generating self-supervised data by using the original image data and the first prediction model, and determining the self-supervised data as second training data of the second vehicle type; S5, training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

[0105] Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various storage media that can store a computer program.

[0106] The embodiment of the present application also provides an electronic device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to execute the computer program to execute the steps in any one of the method embodiments.

[0107] Optionally, the electronic device can further comprise a transmission device and an input and output device, wherein the transmission device is connected with the processor, and the input and output device is connected with the processor.

[0108] Optionally, in the embodiment, the processor can be configured to execute the following steps by using the computer program: S1, training a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; S2, obtaining original data of a second vehicle type; S3, analyzing a data type of the original data; S4, if the original data is a target camera parameter of the second vehicle type, generating a target time sequence multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data, and determining the target time sequence multi-view image and a three-dimensional occupancy semantic ground truth map in the visual annotation data as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generating self-supervised data by using the original image data and the first prediction model, and determining the self-supervised data as second training data of the second vehicle type; S5, training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

[0109] Optionally, specific examples in the embodiments can refer to the examples described in the above embodiments and optional implementation manners, and the embodiments will not be described here again.

[0110] The apparatus embodiments described above are only schematic, wherein the units as shown in the separate components can or can not be physically separate, and the components as shown in the units can or can not be physical units, i.e., can be located in one place, or can be distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0111] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be realized by means of software plus a general hardware platform, and of course can also be realized by hardware. Based on such understanding, the above technical solutions can be embodied in the form of a software product, which can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0112] It should be understood that the terminology used herein is for the purpose of describing particular exemplary embodiments only and is not intended to be limiting. Unless the context clearly indicates otherwise, the singular forms “a,” “an,” and “described” as used herein may also include the plural forms. The terms “comprising,” “including,” “containing,” and “having” are inclusive and therefore indicate the presence of the stated features, steps, operations, elements, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, elements, components, and / or combinations thereof. The method steps, processes, and operations described herein are not construed as requiring them to be performed in a particular order described or illustrated unless the order of performance is explicitly indicated. It should also be understood that additional or alternative steps may be used.

[0113] The above description is merely a specific embodiment of the present invention, enabling those skilled in the art to understand or implement the invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the invention. Therefore, the present invention is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A prediction method of an occupancy semantic map, characterized in that, The method comprises the following steps: training a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; obtaining original data of a second vehicle type; analyzing a data type of the original data; if the original data is a target camera parameter of the second vehicle type, generating a target time sequence multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data, and determining a three-dimensional occupancy semantic ground truth map in the visual annotation data and the target time sequence multi-view image as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generating self-supervised data by using the original image data and the first prediction model, and determining the self-supervised data as second training data of the second vehicle type; training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

2. The method of claim 1, wherein, The method of training a first prediction model of a first vehicle type by using visual annotation data comprises the following steps: obtaining visual annotation data collected by a first vehicle type, wherein the visual annotation data comprises source time sequence multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; encoding the source time sequence multi-view images in an initial model to obtain two-dimensional multi-scale features, performing visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features, and performing occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; calculating a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and updating parameters of the initial model based on the first loss value to obtain a first prediction model of the first vehicle type.

3. The method of claim 1, wherein, The method of generating a target time sequence multi-view image of a second vehicle type according to a target camera parameter and visual annotation data comprises the following steps: generating a three-dimensional Gaussian rendering model by using the visual annotation data; generating a three-dimensional Gaussian display representation of the second vehicle type in each scene by using the three-dimensional Gaussian rendering model, wherein the three-dimensional Gaussian display representation comprises geometric characteristics, color and texture characteristics of the second vehicle type in each scene; analyzing a vehicle driving trajectory in the visual annotation data, and rendering a target time sequence multi-view image of the second vehicle type in the three-dimensional Gaussian display representation based on the vehicle driving trajectory and the target camera parameter.

4. The method of claim 3, wherein, The method of generating a three-dimensional Gaussian rendering model by using the visual annotation data comprises the following steps: obtaining source time sequence multi-view images in the visual annotation data; inputting the source time sequence multi-view images into a three-dimensional Gaussian rendering model to obtain initialized three-dimensional Gaussian point clouds, wherein each Gaussian point cloud contains a center point coordinate, a covariance matrix, color and opacity information; analyzing source camera parameters of the source time sequence multi-view images, and projecting the three-dimensional Gaussian point clouds into a two-dimensional image plane by using the source camera parameters, and rendering to obtain corresponding two-dimensional images. Calculate a second loss value between the two-dimensional image and the source time-series multi-view image, and optimize parameters of the three-dimensional Gaussian point cloud through the second loss value to obtain an optimized three-dimensional Gaussian rendering model.

5. The method of claim 1, wherein, Generating self-supervised data by using the original image data and the first prediction model comprises: Performing semantic automatic labeling and depth estimation on the original image data by using a visual language large model to correspondingly generate a pseudo-label semantic map and a pseudo-label depth map; Generating predicted three-dimensional occupancy semantic features of the original image data by using the first prediction model; Inputting the predicted three-dimensional occupancy semantic features into pre-constructed semantic prediction modules and depth prediction modules respectively to correspondingly obtain a predicted semantic map and a predicted depth map, and inputting the predicted three-dimensional occupancy semantic features into the first prediction model to output an occupancy semantic prediction map; Determining the pseudo-label semantic map, the pseudo-label depth map, the predicted semantic map, the predicted depth map, and the occupancy semantic prediction map as self-supervised data.

6. The method of claim 1, wherein, The second training data comprises a pseudo-label semantic map, a pseudo-label depth map, a predicted semantic map, a predicted depth map, and an occupancy semantic prediction map, and the first prediction model is trained by using the second training data to obtain a second prediction model of the second vehicle type, comprising: Adjusting the first prediction model by using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model; Adjusting the first intermediate model by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model; Adjusting the second intermediate model by using the occupancy semantic prediction map to obtain a second prediction model.

7. The method of claim 6, wherein, Adjusting the first prediction model by using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model, comprising: Calculating a third loss value between the pseudo-label semantic map and the predicted semantic map; Optimizing semantic perception capability of the first prediction model according to the third loss value to obtain a first intermediate model.

8. The method of claim 6, wherein, Adjusting the first intermediate model by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model, comprising: Calculating a fourth loss value between the pseudo-label depth map and the predicted depth map; Optimizing depth perception capability of the first intermediate model according to the fourth loss value to obtain a second intermediate model.

9. The method of claim 6, wherein, Adjusting the second intermediate model by using the occupancy semantic prediction map to obtain a second prediction model, comprising: Reading a first semantic class with the highest confidence and a second semantic class with the second highest confidence of each grid in the occupancy semantic prediction map; Calculating a semantic difference value between the first semantic class and the second semantic class; Determining whether the semantic difference value is greater than a preset threshold value; If the semantic difference value is greater than the preset threshold value, rendering a rendering semantic map and a rendering depth map by using the occupancy semantic prediction map; Determining whether the rendering semantic map is aligned with the pseudo-label semantic map and determining whether the rendering depth map is aligned with the pseudo-label depth map; If the rendering semantic map is aligned with the pseudo-label semantic map and the rendering depth map is aligned with the pseudo-label depth map, the second prediction model is obtained by supervised training the second intermediate model using the original image data and the occupancy semantic prediction map.

10. A prediction device of an occupancy semantic map, characterized by, The method comprises the following steps: The first training module is configured to train a first prediction model of a first vehicle type using visual annotation data, wherein the first prediction model is configured to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type. The acquisition module is configured to acquire original data of a second vehicle type. The analysis module is configured to analyze a data type of the original data. The generation module is configured to, if the original data is target camera parameters of the second vehicle type, generate target time-series multi-view images of the second vehicle type according to the target camera parameters and the visual annotation data, and determine the target time-series multi-view images and a three-dimensional occupancy semantic ground truth map in the visual annotation data as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generate self-supervised data using the original image data and the first prediction model, and determine the self-supervised data as second training data of the second vehicle type. The second training module is configured to train the first prediction model using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is configured to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

11. A storage medium, characterized by The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 9 when running. 12.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the computer program to execute the method described in any one of claims 1 to 9.

Images