Method and device for predicting occupancy semantic map, storage medium, and electronic device

Abstract

The application provides a prediction method and device of an occupancy semantic map, a storage medium and an electronic device, and the method comprises the following steps: training a first prediction model of a first vehicle type by using visual labeling data; obtaining original data of a second vehicle type; analyzing the data type of the original data; if the original data is target camera parameters of the second vehicle type, generating first training data of the second vehicle type according to the target camera parameters and the visual labeling data; if the original data is unlabeled original image data collected by the second vehicle type, generating second training data by using the original image data; and training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type. The technical problem of weak generalization ability of a vision-driven occupancy semantic map prediction model in cross-vehicle application in the related art is solved, and the performance of the vision-driven occupancy semantic map prediction model in cross-domain migration is ensured.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of vehicles, in particular to a prediction method and device of an occupancy semantic map, a storage medium and an electronic device. BACKGROUND

[0002] In related technologies, three-dimensional occupancy semantic map prediction based on vision can solve the problem of long-tail obstacle detection of automatic driving, especially after introducing the occupancy network, which has triggered extensive research. Although significant progress has been made in three-dimensional occupancy prediction based on vision in recent years, the first vehicle model still faces challenges in feature representation, annotation cost and practical application when migrating to the second vehicle model. Specifically: (1) it is difficult to obtain three-dimensional features from two-dimensional image features. Three-dimensional occupancy prediction based on vision aims to fully perceive and understand the three-dimensional scene through image input only, but the lack of depth and geometric information in the image is a major obstacle to learning three-dimensional feature representation. (2) Expensive fine-grained annotation. 3D occupancy prediction involves the occupancy state and semantic category of voxels, so fine-grained semantic annotation is required for each voxel, which is both time-consuming and expensive. (3) Changing working conditions. The significant feature changes of the vehicle operating environment and the diversified sensor suite equipped (including different numbers and fields of view of cameras) cause the performance of the vision-driven occupancy semantic map prediction model to decrease significantly when migrating across domains, and the generalization ability of the model is poor.

[0003] In view of the above problems in the related art, no efficient and accurate solution has been found. SUMMARY

[0004] The present application provides a prediction method and device of an occupancy semantic map, a storage medium and an electronic device to solve the technical problems in the related art.

[0005] According to one embodiment of the present application, a prediction method of an occupancy semantic map is provided, comprising: training a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; obtaining original data of a second vehicle type; analyzing a data type of the original data; if the original data is a target camera parameter of the second vehicle type, generating a target time-series multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data, and determining a three-dimensional occupancy semantic ground truth map in the visual annotation data and the target time-series multi-view image as first training data of the second vehicle type; if the original data is unannotated original image data collected by the second vehicle type, generating self-supervised data by using the original image data and the first prediction model, and determining the self-supervised data as second training data of the second vehicle type; training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

[0006] Optionally, training the first prediction model of the first vehicle type by using the visual annotation data comprises: obtaining visual annotation data collected by the first vehicle type, wherein the visual annotation data comprises source time-series multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; encoding the source time-series multi-view images in an initial model to obtain two-dimensional multi-scale features; performing visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features; performing occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; calculating a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and updating parameters of the initial model based on the first loss value to obtain the first prediction model of the first vehicle type.

[0007] Optionally, generating the target time-series multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data comprises: generating a three-dimensional Gaussian rendering model by using the visual annotation data; generating a three-dimensional Gaussian display representation of the second vehicle type in each scene by using the three-dimensional Gaussian rendering model, wherein the three-dimensional Gaussian display representation comprises geometric characteristics, color and texture characteristics of the second vehicle type in each scene; analyzing a vehicle driving trajectory in the visual annotation data, and rendering the target time-series multi-view image of the second vehicle type in the three-dimensional Gaussian display representation based on the vehicle driving trajectory and the target camera parameter.

[0008] Optionally, generating the three-dimensional Gaussian rendering model using the visual annotation data comprises: obtaining source time-series multi-view images in the visual annotation data; inputting the source time-series multi-view images into the three-dimensional Gaussian rendering model to obtain initialized three-dimensional Gaussian point clouds, wherein each of the Gaussian point clouds comprises a center point coordinate, a covariance matrix, color and opacity information; parsing source camera parameters of the source time-series multi-view images, and projecting the three-dimensional Gaussian point clouds into a two-dimensional image plane using the source camera parameters, and rendering to obtain corresponding two-dimensional images; calculating a second loss value between the two-dimensional images and the source time-series multi-view images, and optimizing parameters of the three-dimensional Gaussian point clouds through the second loss value to obtain an optimized three-dimensional Gaussian rendering model.

[0009] Optionally, generating the self-supervised data using the original image data and the first prediction model comprises: performing semantic automatic labeling and depth estimation on the original image data using a visual language large model to correspondingly generate pseudo-label semantic maps and pseudo-label depth maps; generating predicted three-dimensional occupancy semantic features of the original image data using the first prediction model; inputting the predicted three-dimensional occupancy semantic features into pre-constructed semantic prediction modules and depth prediction modules, respectively, to correspondingly obtain predicted semantic maps and predicted depth maps, and inputting the predicted three-dimensional occupancy semantic features into the first prediction model to output occupancy semantic prediction maps; determining the pseudo-label semantic maps, the pseudo-label depth maps, the predicted semantic maps, the predicted depth maps, and the occupancy semantic prediction maps as the self-supervised data.

[0010] Optionally, the second training data comprises pseudo-label semantic maps, pseudo-label depth maps, predicted semantic maps, predicted depth maps, and occupancy semantic prediction maps, and training the first prediction model using the second training data to obtain a second prediction model for the second vehicle type comprises: adjusting the first prediction model using the pseudo-label semantic maps and the predicted semantic maps to obtain a first intermediate model; adjusting the first intermediate model using the pseudo-label depth maps and the predicted depth maps to obtain a second intermediate model; and adjusting the second intermediate model using the occupancy semantic prediction maps to obtain the second prediction model.

[0011] Optionally, adjusting the first prediction model using the pseudo-label semantic maps and the predicted semantic maps to obtain a first intermediate model comprises: calculating a third loss value between the pseudo-label semantic maps and the predicted semantic maps; and optimizing semantic perception capability of the first prediction model according to the third loss value to obtain the first intermediate model.

[0012] Optionally, the first intermediate model is adjusted by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model, including: calculating a fourth loss value between the pseudo-label depth map and the predicted depth map; and optimizing the depth perception capability of the first intermediate model according to the fourth loss value to obtain a second intermediate model.

[0013] Optionally, the second intermediate model is adjusted by using the occupancy semantic prediction map to obtain a second prediction model, including: reading a first semantic category with the highest confidence and a second semantic category with the second highest confidence in each grid of the occupancy semantic prediction map; calculating a semantic difference value between the first semantic category and the second semantic category; determining whether the semantic difference value is greater than a preset threshold; if the semantic difference value is greater than the preset threshold, rendering the occupancy semantic prediction map to obtain a rendered semantic map and a rendered depth map; determining whether the rendered semantic map is aligned with the pseudo-label semantic map and whether the rendered depth map is aligned with the pseudo-label depth map; and if the rendered semantic map is aligned with the pseudo-label semantic map and the rendered depth map is aligned with the pseudo-label depth map, performing supervised training on the second intermediate model by using the original image data and the occupancy semantic prediction map to obtain a second prediction model.

[0014] According to another embodiment of the present application, a prediction device for an occupancy semantic map is provided, including: a first training module configured to train a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is configured to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; an acquisition module configured to acquire original data of a second vehicle type; an analysis module configured to analyze a data type of the original data; a generation module configured to, if the original data is target camera parameters of the second vehicle type, generate target time-series multi-view images of the second vehicle type according to the target camera parameters and the visual annotation data, and determine a three-dimensional occupancy semantic ground truth map in the visual annotation data and the target time-series multi-view images as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generate self-supervised data by using the original image data and the first prediction model, and determine the self-supervised data as second training data of the second vehicle type; and a second training module configured to train the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is configured to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

[0015] Optionally, the first training module comprises: an acquisition unit configured to acquire visual annotation data collected by the first vehicle model, wherein the visual annotation data comprises source time-series multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; a processing unit configured to encode the source time-series multi-view images in an initial model to obtain two-dimensional multi-scale features, perform visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features, and perform occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; and a training unit configured to calculate a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and update parameters of the initial model based on the first loss value to obtain a first prediction model of the first vehicle model.

[0016] Optionally, the generation module comprises: a first generation unit configured to generate a three-dimensional Gaussian rendering model using the visual annotation data; a second generation unit configured to generate a three-dimensional Gaussian display representation of the second vehicle model in each scene using the three-dimensional Gaussian rendering model, wherein the three-dimensional Gaussian display representation comprises geometric characteristics, color, and texture characteristics of the second vehicle model in each scene; and a rendering unit configured to parse a vehicle driving trajectory in the visual annotation data, and render a target time-series multi-view image of the second vehicle model in the three-dimensional Gaussian display representation based on the vehicle driving trajectory and the target camera parameters.

[0017] Optionally, the first generation unit comprises: an acquisition subunit configured to acquire source time-series multi-view images in the visual annotation data; an acquisition subunit configured to input the source time-series multi-view images into a three-dimensional Gaussian rendering model to obtain initialized three-dimensional Gaussian point clouds, wherein each of the Gaussian point clouds comprises a center point coordinate, a covariance matrix, color, and opacity information; a rendering subunit configured to parse source camera parameters of the source time-series multi-view images, project the three-dimensional Gaussian point clouds into a two-dimensional image plane using the source camera parameters, and render corresponding two-dimensional images; and a generation subunit configured to calculate a second loss value between the two-dimensional images and the source time-series multi-view images, and optimize parameters of the three-dimensional Gaussian point clouds using the second loss value to obtain an optimized three-dimensional Gaussian rendering model.

[0018] Optionally, the generating module comprises: a third generating unit configured to perform semantic automatic labeling and depth estimation on the original image data by using a visual language large model, to generate a pseudo-label semantic map and a pseudo-label depth map; a fourth generating unit configured to generate predicted three-dimensional occupancy semantic features of the original image data by using the first prediction model; a processing unit configured to input the predicted three-dimensional occupancy semantic features into a pre-constructed semantic prediction module and a depth prediction module respectively, to obtain a predicted semantic map and a predicted depth map, and input the predicted three-dimensional occupancy semantic features into the first prediction model to output an occupancy semantic prediction map; and a determining unit configured to determine the pseudo-label semantic map, the pseudo-label depth map, the predicted semantic map, the predicted depth map, and the occupancy semantic prediction map as self-supervised data.

[0019] Optionally, the second training data comprises a pseudo-label semantic map, a pseudo-label depth map, a predicted semantic map, a predicted depth map, and an occupancy semantic prediction map, and the second training module comprises: a first adjusting unit configured to adjust the first prediction model by using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model; a second adjusting unit configured to adjust the first intermediate model by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model; and a third adjusting unit configured to adjust the second intermediate model by using the occupancy semantic prediction map to obtain a second prediction model.

[0020] Optionally, the first adjusting unit comprises: a calculating sub-unit configured to calculate a third loss value between the pseudo-label semantic map and the predicted semantic map; and an optimizing sub-unit configured to optimize semantic perception ability of the first prediction model according to the third loss value to obtain the first intermediate model.

[0021] Optionally, the second adjusting unit comprises: a calculating sub-unit configured to calculate a fourth loss value between the pseudo-label depth map and the predicted depth map; and an optimizing sub-unit configured to optimize depth perception ability of the first intermediate model according to the fourth loss value to obtain the second intermediate model.

[0022] Optionally, the third adjusting unit comprises: a reading subunit, configured to read a first semantic category with the highest confidence and a second semantic category with the second highest confidence in each grid of the occupancy semantic prediction map; a calculation subunit, configured to calculate a semantic difference between the first semantic category and the second semantic category; a judgment subunit, configured to judge whether the semantic difference is greater than a preset threshold; a rendering subunit, configured to, if the semantic difference is greater than the preset threshold, render the occupancy semantic prediction map to obtain a rendered semantic map and a rendered depth map; the judgment subunit, configured to judge whether the rendered semantic map is aligned with the pseudo-label semantic map and whether the rendered depth map is aligned with the pseudo-label depth map; and a training subunit, configured to, if the rendered semantic map is aligned with the pseudo-label semantic map and the rendered depth map is aligned with the pseudo-label depth map, perform supervised training on the second intermediate model by using the original image data and the occupancy semantic prediction map to obtain a second prediction model.

[0023] According to another aspect of the embodiments of the present application, a storage medium is also provided, which includes a stored program, and the program performs the above steps when running.

[0024] According to another aspect of the embodiments of the present application, an electronic device is also provided, which includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory complete mutual communication through the communication bus; the memory is used to store a computer program; and the processor is used to perform the steps in the above method by running the program stored in the memory.

[0025] The embodiments of the present application also provide a computer program product including instructions, which, when running on a computer, cause the computer to perform the steps in the above method.

[0026] The present application has the following beneficial effects:

[0027] 1. A three-dimensional Gaussian rendering module is introduced to render a time-series multi-view image conforming to target camera parameters, which can be used to optimize model performance, thereby eliminating the influence of target camera parameters on model performance and improving the generalization performance of the model under the second vehicle type.

[0028] 2. A semantic prediction module and a depth prediction module are introduced to process three-dimensional occupancy semantic features under the second vehicle type, and the multi-view pseudo-label semantic map and depth map obtained through the visual language model are aligned for supervision, thereby improving the semantic and depth perception generalization performance of the model for the second vehicle type image.

[0029] 3. Two pseudo-label screening principles are proposed, high-quality pseudo labels of the semantic map can be screened out for the semantic map prediction model of the second vehicle type, which improves the generalization performance of the model under the second vehicle type, saves the cost of manual annotation of data, and improves the data utilization. BRIEF DESCRIPTION OF DRAWINGS

[0030] The accompanying drawings, which are included to provide a further understanding of the application and are incorporated in and constitute a part of this application, illustrate embodiments of the application and together with the description serve to explain the application. In the drawings:

[0031] Figure 1 is a hardware structure block diagram of a car according to an embodiment of the application;

[0032] Figure 2 is a flow chart of a prediction method of a semantic map according to an embodiment of the application;

[0033] Figure 3 is a training schematic diagram of a semantic map prediction model of a source domain in an embodiment of the application;

[0034] Figure 4 is a three-dimensional Gaussian display representation training optimization schematic diagram of a certain scene of a source domain in an embodiment of the application;

[0035] Figure 5 is a training schematic diagram of a semantic map prediction model of a target domain in an embodiment of the application;

[0036] Figure 6 is a structure block diagram of a prediction device of a semantic map according to an embodiment of the application. DETAILED DESCRIPTION

[0037] In order to enable persons skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be described clearly and completely below with reference to the accompanying drawings in the embodiments of the present application. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by persons skilled in the art without creative work should fall within the protection scope of the present application. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.

[0038] It should be noted that the terms "first", "second" and the like in the description and in the claims of the present application as well as above-mentioned appended drawings are intended to distinguish between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the use of the terms so-termed first, second, etc., is interchangeable so that the embodiments of the application described herein are capable of accomplishing the same object for example in different orders than illustrated or described herein. Also, the terms "comprises", "comprising", "includes", "including" and the like are to be construed in an inclusive rather than an exclusive sense, that is, in the sense of "including, but not limited to", to cover a process, method, object or apparatus including at least the recited steps or units, and any additional steps or units that are either expressly recited in the specification or are inherent in the process, method, object or apparatus.

[0039] Embodiment 1

[0040] The method embodiments provided by the embodiments of the present application can be executed in a car, a server, a processor, a safety controller, an automatic driving / assisted driving / smart driving controller or similar processing devices. Taking the case of running on a car, Figure 1 is a hardware structure block diagram of a car of an embodiment of the present application. As shown in Figure 1 , the car can include one or more (only one is shown in Figure 1 ) processor 102 (the processor 102 can include but is not limited to a processing device such as a microprocessor MCU or a programmable logic device FPGA) and a memory 104 for storing data. Optionally, the above-mentioned car can also include a transmission device 106 for communication function and an input and output device 108. Those skilled in the art can understand that Figure 1 The structure shown is only schematic, which does not limit the structure of the above-mentioned car. For example, the car can also include more or fewer components than those shown in Figure 1 , or have a different configuration from Figure 1 .

[0041] The memory 104 can be used to store car programs, such as software programs of application software and modules, such as the car program corresponding to the prediction method of the occupancy semantic map of a car in the embodiments of the present application. The processor 102 executes various functional applications and data processing by running the car program stored in the memory 104, that is, implements the above-mentioned method. The memory 104 can include a high-speed random access memory, and can also include a non-volatile memory, such as one or more magnetic storage devices, flash memories, or other non-volatile solid-state memories. In some examples, the memory 104 can further include a memory remotely arranged with respect to the processor 102, which can be connected to the car through a network. Examples of the above-mentioned network include but are not limited to the Internet, an intranet, a local area network, a mobile communication network and a combination thereof.

[0042] The transmission device 106 is configured to receive or send data via a network. The network can include a wireless network provided by a communication provider of the vehicle. In one example, the transmission device 106 includes a network interface controller (NIC) that can be connected to other network devices through a base station to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module configured to communicate with the Internet wirelessly.

[0043] A prediction method of an occupancy semantic map is provided in the embodiment, Figure 2 is a flowchart of a prediction method of an occupancy semantic map according to an embodiment of the present application, as shown in the figure, the flow includes the following steps: Figure 2

[0044] In step S201, a first prediction model of a first vehicle type is trained using visual annotation data, wherein the first prediction model is configured to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type.

[0045] Optionally, the visual annotation data is time-series multi-view images collected by a camera of a sample vehicle of the first vehicle type, and a corresponding three-dimensional occupancy semantic ground truth map.

[0046] The first prediction model and the second prediction model of the embodiment are occupancy semantic map models applied to the first vehicle type and the second vehicle type, respectively. The first vehicle type belongs to a source domain of the occupancy semantic map model, and the second vehicle type belongs to a target domain of the occupancy semantic map model. The scheme of the embodiment is configured to realize the generalization capability of the occupancy semantic map model from the source domain to the target domain.

[0047] The occupancy semantic map of the embodiment is a three-dimensional visual representation of a vehicle driving scene in the field of intelligent driving perception. The entire driving scene is divided into minimum unit cubic grids, and each grid is assigned a semantic label.

[0048] In step S202, original data of the second vehicle type is obtained.

[0049] Optionally, the original data is unlabeled data from the second vehicle type, which can be target camera parameters of the second vehicle type, or unlabeled original image data collected by a camera of the vehicle of the second vehicle type.

[0050] In step S203, a data type of the original data is analyzed.

[0051] ​Step S204, if the original data is the target camera parameter of the second vehicle model, generating a target time sequence multi-view image of the second vehicle model according to the target camera parameter and the visual annotation data, and determining the target time sequence multi-view image and the three-dimensional occupancy semantic ground truth map in the visual annotation data as the first training data of the second vehicle model; if the original data is the unlabeled original image data collected by the second vehicle model, generating self-supervised data by using the original image data and the first prediction model, and determining the self-supervised data as the second training data of the second vehicle model.

[0052] Step S205, training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle model, wherein the second prediction model is used to predict the occupancy semantic map of the second vehicle model according to the visual image of the second vehicle model.

[0053] Through the above steps, the first prediction model of the first vehicle model is trained by using the visual annotation data, wherein the first prediction model is used to predict the occupancy semantic map of the first vehicle model according to the visual image of the first vehicle model; the original data of the second vehicle model is obtained; the data type of the original data is analyzed; if the original data is the target camera parameter of the second vehicle model, the target time sequence multi-view image of the second vehicle model is generated according to the target camera parameter and the visual annotation data, and the target time sequence multi-view image and the three-dimensional occupancy semantic ground truth map in the visual annotation data are determined as the first training data of the second vehicle model; if the original data is the unlabeled original image data collected by the second vehicle model, the self-supervised data is generated by using the original image data and the first prediction model, and the self-supervised data is determined as the second training data of the second vehicle model; the first prediction model is trained by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle model, wherein the second prediction model is used to predict the occupancy semantic map of the second vehicle model according to the visual image of the second vehicle model. The unlabelled data of the second vehicle model is fully utilized, the generalization of semantic feature fusion is enhanced, the generalization ability of deep feature fusion is improved, and the generalization of multi-source feature utilization is optimized, so as to comprehensively improve the generalization ability of the model in the target domain, solve the technical problem that the generalization ability of the visual-driven occupancy semantic map prediction model is weak in the cross-vehicle application in the related technology, and ensure the performance of the visual-driven occupancy semantic map prediction model in the cross-domain migration.

[0054] In the embodiment, training the first prediction model of the first vehicle type by using the visual annotation data comprises: acquiring visual annotation data collected by the first vehicle type, wherein the visual annotation data comprises source time sequence multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; encoding the source time sequence multi-view images in an initial model to obtain two-dimensional multi-scale features; performing visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features; performing occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; calculating a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and updating parameters of the initial model based on the first loss value to obtain the first prediction model of the first vehicle type.

[0055] Figure 3 For the training schematic diagram of the occupancy semantic map prediction model of the source domain in the embodiment, the implementation process of the source domain training is divided into three modules: an image feature encoder module, a visual conversion module, and an occupancy semantic decoding module. The visual conversion module outputs three-dimensional occupancy semantic features, and the occupancy semantic decoding module outputs an occupancy semantic prediction map, so as to train in combination with an occupancy semantic ground truth module. The modules are connected to each other and jointly complete a complete process from a two-dimensional image to three-dimensional occupancy semantic prediction. The process comprises:

[0056] First, the image feature encoder module is used to extract features from a set of input multi-view images. The module uses an existing image feature extraction network, such as ResNet, Swin Transformer, or ConvNeXt, to extract deep-level feature information from each image. Each image is encoded into a high-dimensional feature vector , which is expressed by the formula:

[0057]

[0058] In the formula, denotes the image feature extraction network. Through such processing, the features of all images are combined into a feature matrix , which provides high-quality feature input for subsequent three-dimensional space mapping. The implementation of the module ensures that the semantic and geometric information of each pixel in the image is retained and transmitted in a structured form.

[0059] Then, the visual conversion module is used to map the two-dimensional image features to a three-dimensional space in a learning manner to generate a three-dimensional occupancy semantic feature representation. The module converts the feature matrix into source domain three-dimensional occupancy semantic features by combining geometric projection and depth estimation methods. The specific formula is:

[0060]

[0061] wherein, including the intrinsic and extrinsic parameters of the camera, for defining the geometric relationship from the image plane to the three-dimensional space, representing the image depth information obtained by the depth estimation method. The depth estimation method can adopt a public pre-trained model, such as ZeroDepth. Through such projection, the three-dimensional coordinates in each space can be accurately mapped to a feature value This module realizes accurate mapping from the feature plane to the three-dimensional space, and provides spatialized semantic feature expression for occupancy semantic decoding.

[0062] Finally, using the occupancy semantic decoding module to parse the three-dimensional occupancy semantic features and segment them into different semantic categories. This module uses a three-dimensional convolutional network (3D CNN) to perform semantic classification on each three-dimensional occupancy semantic feature and predict its class probability . The specific formula is:

[0063]

[0064] The output semantic distribution probability is a semantic segmentation result in a three-dimensional space, where each occupancy value represents the predicted probability of the grid position belonging to different categories. Through this processing, the semantic structure of a complex scene is clearly presented, facilitating subsequent analysis and application.

[0065] If the model trained in the source domain is directly applied to other target domain data for occupancy semantic map prediction, the model performance will be greatly reduced. Therefore, the generalization performance of the model needs to be improved for the target domain.

[0066] In one implementation scenario of the embodiment, in a case where only the target camera parameters of the second vehicle model are available without original image data, generating the target time-series multi-view images of the second vehicle model according to the target camera parameters and the visual annotation data includes: generating a three-dimensional Gaussian rendering model using the visual annotation data; generating a three-dimensional Gaussian display representation of the second vehicle model in each scene using the three-dimensional Gaussian rendering model, wherein the three-dimensional Gaussian display representation includes geometric characteristics, color, and texture characteristics of the second vehicle model in each scene; parsing a vehicle driving trajectory in the visual annotation data, and rendering the target time-series multi-view images of the second vehicle model in the three-dimensional Gaussian display representation based on the vehicle driving trajectory and the target camera parameters.

[0067] In one example, generating the three-dimensional Gaussian rendering model using the visual annotation data includes: obtaining source time-series multi-view images in the visual annotation data; inputting the source time-series multi-view images into a three-dimensional Gaussian rendering model to obtain initialized three-dimensional Gaussian point clouds, wherein each of the Gaussian point clouds includes a center point coordinate, a covariance matrix, color and opacity information; analyzing source camera parameters of the source time-series multi-view images, and projecting the three-dimensional Gaussian point clouds into a two-dimensional image plane using the source camera parameters, and rendering to obtain corresponding two-dimensional images; calculating a second loss value between the two-dimensional images and the source time-series multi-view images, and optimizing parameters of the three-dimensional Gaussian point clouds through the second loss value to obtain an optimized three-dimensional Gaussian rendering model.

[0068] Figure 4 For the three-dimensional Gaussian display representation of a certain scene in the source domain in the embodiment of the application, a training and optimization schematic diagram is shown in Figure 4 The step is implemented through a Gaussian point cloud initialization module, a projection module, an adaptive density control module and a differentiable rasterization module. The projection module also needs to be combined with camera parameters. The differentiable rasterization module generates six views of Gaussian rendering corresponding to the original six views. The first step is to realize efficient rendering of a three-dimensional scene in a pure vision-driven manner. The specific implementation process is as follows:

[0069] First, input time-series multi-view images of a certain scene into a Gaussian point cloud initialization module. Through the SfM algorithm in the module, common features in the images can be identified, and the motion path of the camera and the three-dimensional point cloud structure of the scene can be estimated. After obtaining the three-dimensional point cloud structure, the Gaussian point cloud initialization module converts each point cloud into a three-dimensional Gaussian point cloud, which is used as a basic training model for optimization. Each three-dimensional Gaussian point cloud is composed of a center point coordinate, a covariance matrix, color information and opacity information, and is represented as:

[0070]

[0071] In the formula, is the three-dimensional space position of each Gaussian point , which defines the position of the point in space; is the three-dimensional Gaussian point space distribution shape and size, which is used to describe the distribution range and directionality of the point in the three-dimensional space; is the color information contained in each Gaussian point , which represents the RGB color value and is used for color expression during rendering; is the transparency weight of each Gaussian point, which is represented by , and is used to control the contribution intensity of the point to the rendering result.

[0072] Secondly, the three-dimensional Gaussian points are projected from the world coordinate system to the two-dimensional image plane by the projection module. Based on the intrinsic and extrinsic parameters of the camera, the projection formula of the Gaussian point cloud is expressed as:

[0073]

[0074] where, is the projection coordinate of the three-dimensional Gaussian point on the two-dimensional plane; is the covariance matrix after projection; is the perspective transformation matrix of the camera, which converts the point from the world coordinate system to the camera space, where is the rotation matrix, is the translation vector; is the Jacobian matrix, which is the affine approximation of the projection transformation, used to convert the point from the camera space to the two-dimensional image plane.

[0075] Then, the Gaussian points on the two-dimensional image plane are input to the differentiable rasterization module for image rendering. When rendering, the color value of each pixel is obtained by the weighted contribution of all projected Gaussians:

[0076]

[0077] where, is the pixel coordinate in the image plane; is the coordinate vector of the corresponding pixel; is the two-dimensional projection coordinate of the Gaussian point cloud and the two-dimensional covariance matrix. The rendered image is compared with the ground truth image, and the rendering loss is calculated by the following formula:

[0078]

[0079] where, represents the rendering loss; represents the image ground truth; represents the rendered image; represents the structural similarity calculation function; , represents the weight of the loss.

[0080] ​Finally, after initializing the Gaussian distribution and undergoing a certain number of iterations, such as 1000 iterations, the density of the Gaussian point cloud is adjusted. The adjustment principle is to determine whether the gradient of each 3D Gaussian point cloud is greater than a certain threshold. For example, if the gradient of a Gaussian point cloud is ≥200, it will be processed. Processing operations include cloning, segmentation, and pruning. The specific operation depends on the conditions. When the scale of the Gaussian point cloud is large, it is segmented, dividing the point into multiple sub-Gaussian points (e.g., at a uniform offset from the center), and the scale and opacity of each sub-Gaussian point are reduced. When the scale is small, it indicates that the distribution range of the Gaussian point cloud is too small and may not effectively cover the area in the scene; therefore, it is cloned, making multiple copies of the point cloud and adjusting the center position. When the opacity of a Gaussian point is less than a certain threshold (e.g., 0.005), it is considered that the point cloud contributes little to the rendering result, and pruning can be performed, i.e., it is removed. This dynamic adjustment sets the conditions for Gaussian point cloud density, allowing optimization of the number and distribution range of Gaussian points during rendering, thus improving rendering quality while conserving computational resources. The scale calculation for Gaussian point clouds is as follows:

[0081]

[0082] In the formula, , , Let be the diagonal elements of the covariance matrix of the Gaussian point cloud.

[0083] The second step is to obtain multi-view images that conform to the camera parameters of the target domain. This invention generates a 3D Gaussian display representation for each scene through the collaborative action of a Gaussian point cloud initialization module, a projection module, an adaptive density control module, and a differentiable rasterization module. Based on this, given the camera parameters of the target domain, multi-view images conforming to the target domain camera parameters can be rendered. Specifically, in this embodiment, Figure 4 The camera parameters shown are replaced with those of the target domain camera, and the 3D Gaussian points are mapped onto the 2D image plane using a projection module based on the target domain camera parameters. Subsequently, the projection result is rendered using a differentiable rasterization module to generate a temporal multi-view image that conforms to the target domain camera parameters. This generated temporal multi-view image is then input into the occupancy semantic map prediction model to optimize and fine-tune its performance. By adjusting the matching between the generated image and the target domain camera parameters, the negative impact of the target domain camera parameters on model performance can be effectively eliminated, thereby improving the model's generalization performance in the target domain.

[0084] In another implementation scenario of the embodiment, the original data is unlabeled original image data collected by the second vehicle model, and generating self-supervised data using the original image data and the first prediction model includes: performing semantic automatic labeling and depth estimation on the original image data using a visual language large model, thereby generating a pseudo-label semantic map and a pseudo-label depth map; generating predicted three-dimensional occupancy semantic features of the original image data using the first prediction model; inputting the predicted three-dimensional occupancy semantic features into a pre-constructed semantic prediction module and a depth prediction module respectively, thereby obtaining a predicted semantic map and a predicted depth map, and inputting the predicted three-dimensional occupancy semantic features into the first prediction model to output an occupancy semantic prediction map; and determining the pseudo-label semantic map, the pseudo-label depth map, the predicted semantic map, the predicted depth map, and the occupancy semantic prediction map as the self-supervised data.

[0085] The implementation scenario proposes a method based on large model pseudo-labeling and multi-stage optimization. Based on the training of the aforementioned occupancy semantic map prediction model (first prediction model), three modules are added, which are a visual language large model for automatically labeling the semantics and depth of an image, a semantic prediction module for predicting image semantic information, and a depth prediction module for predicting image depth information. By combining semantic automatic labeling, depth estimation, and pseudo-label filtering, the adaptability of the model trained on source domain data to the target domain is gradually improved.

[0086] In this implementation scenario, the second training data includes a pseudo-label semantic map, a pseudo-label depth map, a predicted semantic map, a predicted depth map, and an occupancy semantic prediction map, and training the first prediction model using the second training data to obtain a second prediction model for the second vehicle model includes: adjusting the first prediction model using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model; adjusting the first intermediate model using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model; and adjusting the second intermediate model using the occupancy semantic prediction map to obtain the second prediction model.

[0087] In one implementation, adjusting the first prediction model using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model includes: calculating a third loss value between the pseudo-label semantic map and the predicted semantic map; and optimizing the semantic perception ability of the first prediction model according to the third loss value to obtain a first intermediate model.

[0088] The multi-view image of the target domain is automatically labeled in semantics by using a visual language large model to obtain a pseudo semantic map of the multi-view image; the multi-view image of the target domain is processed by the image feature encoding module and the visual conversion module to obtain three-dimensional occupancy semantic features; the three-dimensional occupancy semantic features are input into a semantic prediction module to obtain a predicted semantic map of the multi-view image; the predicted semantic map is compared with the multi-view pseudo semantic map to calculate a semantic loss; and the semantic loss is used for reverse propagation optimization of the occupancy semantic map prediction model to improve the semantic perception generalization performance of the model in the target domain.

[0089] In one embodiment, the first intermediate model is adjusted by using the pseudo label depth map and the predicted depth map to obtain a second intermediate model, including: calculating a fourth loss value between the pseudo label depth map and the predicted depth map; and optimizing the depth perception ability of the first intermediate model according to the fourth loss value to obtain a second intermediate model.

[0090] The multi-view image of the target domain is automatically labeled in semantics by using a visual language large model to obtain a pseudo semantic map of the multi-view image; the multi-view image of the target domain is processed by the image feature encoding module and the visual conversion module to obtain three-dimensional occupancy semantic features; the three-dimensional occupancy semantic features are input into a semantic prediction module to obtain a predicted semantic map of the multi-view image; the predicted semantic map is compared with the multi-view pseudo semantic map to calculate a semantic loss; and the semantic loss is used for reverse propagation optimization of the occupancy semantic map prediction model to improve the semantic perception generalization performance of the model in the target domain.

[0091] In one embodiment, the second intermediate model is adjusted by using the occupancy semantic prediction map to obtain a second prediction model, including: reading a first semantic class with the highest confidence and a second semantic class with the second highest confidence in each grid of the occupancy semantic prediction map; calculating a semantic difference value between the first semantic class and the second semantic class; judging whether the semantic difference value is greater than a preset threshold value; if the semantic difference value is greater than the preset threshold value, using the occupancy semantic prediction map to render a rendered semantic map and a rendered depth map; judging whether the rendered semantic map is aligned with the pseudo label semantic map and whether the rendered depth map is aligned with the pseudo label depth map; if the rendered semantic map is aligned with the pseudo label semantic map and the rendered depth map is aligned with the pseudo label depth map, using the original image data and the occupancy semantic prediction map to supervise training of the second intermediate model to obtain a second prediction model.

[0092] The three-dimensional occupancy semantic features of the target domain are input into the occupancy semantic decoding module trained in the source domain to obtain an occupancy semantic prediction map in the target domain, high-quality pseudo occupancy grid labels are screened out according to two principles, and the high-quality pseudo occupancy grid labels are added to the next iteration cycle to supervise training of the occupancy semantic prediction model of the second vehicle type.

[0093] The first pseudo-label screening principle is the difference of class confidence. The specific implementation is: first, calculate the difference between the first class (the highest confidence class) and the second class (the second highest confidence class) occupying the semantic prediction map of each grid; then set a threshold, when the difference is greater than the threshold, it is considered that the prediction result is reliable and can be used as a pseudo-label to train the model. The second pseudo-label screening principle is the semantic and spatial consistency constraint. The specific implementation is: the occupancy semantic prediction map is rendered into a multi-view image rendering semantic map and a rendering depth map through a rendering module; the rendering semantic map and the rendering depth map are respectively aligned with the pseudo semantic map and the depth map obtained by the aforementioned visual language large model to select the pseudo label with consistent semantics and space.

[0094] Figure 5 is a training schematic diagram of the target domain occupancy semantic map prediction model in the embodiment of the present application, based on Figure 3 , three modules are added, which are a visual language large model for automatically labeling the semantics and depth of the image, a semantic prediction module for predicting the semantic information of the image, and a depth prediction module for predicting the depth information of the image. The training process includes:

[0095] The first step is to add the multi-view pseudo semantic map to the source domain pre-trained occupancy semantic map prediction model for training. Specifically, first, use the visual language large model (such as SAM 2) to automatically label the semantics of the multi-view image in the target domain to generate high-quality pseudo semantic maps. These pseudo semantic maps serve as approximate semantic labels for the target domain images, providing a supervision signal for model training. Specifically, input a set of multi-view images in the target domain , pass them through the visual language large model for pixel-by-pixel semantic segmentation to generate a set of pseudo semantic maps corresponding to the multi-view images . The calculation formula is:

[0096]

[0097] These pseudo semantic maps can provide high-quality supervision signals for training in the absence of real labeled data in the target domain. Next, the multi-view images in the target domain are processed by the image feature encoding module and the visual conversion module to generate three-dimensional occupancy semantic features of the target domain, where the specific implementation is consistent with the source domain training and will not be repeated. Finally, the generated three-dimensional occupancy semantic features are input into the semantic prediction module to generate the predicted semantic map of the multi-view image:

[0098]

[0099] Subsequently, the semantic map output by the semantic prediction module is corresponding pseudo semantic map a pixel-wise comparison is performed to calculate a semantic loss , which measures the difference between the prediction result and the pseudo label. Through back propagation, the semantic loss optimizes the parameters of the image feature encoding module, the visual conversion module, and the semantic prediction module. Through the above steps, the multi-view pseudo semantic map is introduced into the training of the occupancy semantic map prediction model, and is optimized in combination with the semantic loss, effectively improving the semantic perception ability of the model in the target domain and significantly enhancing the generalization performance of the model.

[0100] In the second step, the multi-view depth map is added to the training of the aforementioned occupancy semantic map prediction model to improve the depth perception generalization performance of the model in the target domain. Specifically, first, a visual language large model (such as DepthAnythingV2) is used to estimate the depth of the multi-view images in the target domain to generate high-quality multi-view depth maps. These depth maps serve as approximate geometric labels for the target domain images, providing a supervision signal for the training of the model. Specifically, a set of multi-view images in the target domain is input into the visual language large model to perform pixel-wise depth estimation and generate a set of depth maps corresponding to the multi-view images . The calculation formula is as follows:

[0101]

[0102] These depth maps can effectively describe the scene geometry information of the target domain images, providing geometric constraints for subsequent model optimization. Next, the multi-view images in the target domain are processed by the image feature encoding module and the visual conversion module to generate three-dimensional occupancy semantic features in the target domain. The specific implementation here is consistent with the source domain training and will not be repeated. Finally, the generated three-dimensional occupancy semantic features are input into the depth prediction module to generate predicted depth maps for the multi-view images :

[0103]

[0104] Subsequently, the depth maps output by the depth prediction module are compared with the depth maps estimated by the large model in a pixel-wise manner to calculate a depth loss , which measures the difference between the prediction result and the depth map estimated by the large model. Through back propagation, the depth loss The parameters of the image feature encoding module, the visual conversion module, and the semantic prediction module are optimized. Through the above steps, the multi-view depth map is introduced into the training of the occupancy semantic map prediction model, and the geometric perception ability of the model is optimized by combining the depth loss, which effectively improves the understanding of the geometric information of the target domain scene by the model and significantly enhances the depth perception generalization performance of the model.

[0105] In a third step, the obtained pseudo-occupancy grid labels are added to the training of the aforementioned occupancy semantic map prediction model to improve the occupancy semantic prediction ability of the model in the target domain. The implementation content is described below in combination with specific steps and formulas:

[0106] First, the target domain three-dimensional occupancy semantic features generated in the aforementioned step are input into the occupancy semantic decoding module trained in the source domain to generate an occupancy semantic prediction map in the target domain . .

[0107]

[0108] wherein, is the occupancy semantic prediction map in the target domain, is the spatial dimension of the three-dimensional grid, is the number of categories, represents the prediction probability of the grid position belonging to the category . Next, according to two pseudo-label screening principles, high-quality pseudo-occupancy grid labels are screened from the target domain occupancy semantic prediction map, and are added to the next iteration cycle to supervise the training of the target domain occupancy semantic map prediction model.

[0109] The first pseudo-label screening principle is based on the category confidence difference, and selects the grid with a higher prediction confidence as the pseudo-label. The specific implementation manner is as follows:

[0110] For each grid , the difference between the first category (the category with the highest prediction probability) and the second category (the category with the second highest prediction probability) is calculated:

[0111]

[0112] wherein, , .

[0113] A threshold value is set, and when , it is considered that the prediction result of the grid is reliable and can be added to the training set as a pseudo-label. The screening result is:

[0114]

[0115] wherein, represents the pseudo label screened according to the confidence difference value, represents that the grid is not selected.

[0116] The second pseudo label screening principle is based on semantic and spatial consistency constraints, and is used to screen high-quality grid pseudo labels from the pseudo semantic map and the pseudo depth Figure 1 map generated by the visual language large model. This principle uses the alignment of the semantic prediction map and the pseudo label generated in the image space in terms of semantics and geometry to ensure that the screened pseudo labels have higher reliability.

[0117] Specifically, the target domain occupancy semantic prediction map is rendered by a rendering module into an image space semantic map and a depth map, denoted as rendered semantic map and rendered depth map , respectively.

[0118]

[0119] The rendering module can map each occupancy grid in three-dimensional space to a pixel in image space, and ensure a one-to-one correspondence between three-dimensional grids and image pixels. For a certain grid in the target domain, its rendering result corresponds to a certain pixel in the image space. For each pixel , if the rendered semantic category and the depth value are completely consistent with the pseudo semantic map and the pseudo depth map generated by the visual language large model, then the corresponding three-dimensional occupancy grid is considered to be a credible pseudo label. The specific screening conditions are:

[0120]

[0121] The grid label that meets the above conditions will be selected as a pseudo label, and its semantic category is:

[0122]

[0123] Otherwise, the grid label will be discarded and not used as a pseudo label for training.

[0124] Finally, the pseudo labels and obtained by the two screening principles are integrated to obtain a high-quality pseudo label set :

[0125]

[0126] The pseudo-labels screened out are added to the next iteration training to supervise and optimize the occupancy semantic map prediction model, and the model's occupancy semantic prediction ability for the target domain scene is improved.

[0127] By using the scheme of the embodiment, a pure visual occupancy semantic map prediction generalization framework suitable for automatic driving is constructed through source domain training and target domain image generation based on three-dimensional Gaussian rendering, and the adaptability and prediction accuracy of the model in multi-domain and multi-vehicle scenes are effectively improved, thereby providing a strong guarantee for the safety and reliability of the automatic driving system. First, the initial model training is completed in the source domain using the labeled data to ensure that the model can effectively learn the visual features, depth information and semantic relationships of the source domain, thereby laying a foundation for subsequent generalization. Second, for the case where only the camera parameters of the target domain are available without original data, three-dimensional Gaussian rendering is used to generate new visual images that conform to the camera parameters of the target domain, and the new visual images are used for model training to adapt to different perspective changes. Finally, for the problem that the original data of the target domain is not manually labeled, pseudo semantic maps and depth maps are generated by combining a large model, multi-scale features are extracted and a unified three-dimensional voxel feature map is constructed, pseudo labels are generated through confidence screening for self-supervised training, thereby improving the generalization performance of the model in multi-domain and multi-vehicle scenes, effectively reducing the dependence on manual labeling, and enhancing the robustness and adaptability of the model.

[0128] The scheme of the embodiment fully utilizes the unlabeled data of the target domain, realizes the generalization enhancement of semantic feature fusion, the generalization performance improvement of depth feature fusion, and the generalization optimization of multi-source feature utilization, thereby comprehensively improving the generalization performance of the model in the target domain.

[0129] Through the above description of the embodiments, those skilled in the art can clearly understand that the method according to the above embodiments can be realized by means of software and a necessary general hardware platform, of course, it can also be realized by hardware, but in many cases the former is a better embodiment. Based on such understanding, the technical solutions of the present application can be embodied in the form of a software product, which is stored in a storage medium (such as a ROM / RAM, a magnetic disk, an optical disk), and includes a plurality of instructions for causing a terminal device (which can be a mobile phone, a computer, a server, or a network device, etc.) to execute the method described in each embodiment of the present application.

[0130] Embodiment 2

[0131] ​A prediction device of an occupancy semantic map is also provided in the embodiments, which is used to implement the above embodiments and preferred embodiments, and will not be described again. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the device described in the following embodiments is preferably implemented in software, implementation of hardware, or a combination of software and hardware, is also possible and contemplated.

[0132] Figure 6 is a structural block diagram of a prediction device of an occupancy semantic map according to an embodiment of the application, as shown in the figure, the device comprises: Figure 6

[0133] A first training module 61 is configured to train a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type.

[0134] A obtaining module 62 is configured to obtain original data of a second vehicle type.

[0135] An analyzing module 63 is configured to analyze a data type of the original data.

[0136] A generating module 64 is configured to, if the original data is a target camera parameter of the second vehicle type, generate a target time sequence multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data, and determine a three-dimensional occupancy semantic ground truth map in the target time sequence multi-view image and the visual annotation data as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generate self-supervised data by using the original image data and the first prediction model, and determine the self-supervised data as second training data of the second vehicle type.

[0137] A second training module 65 is configured to train the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

[0138] ​Optionally, the first training module comprises: an acquisition unit configured to acquire visual annotation data collected by the first vehicle model, wherein the visual annotation data comprises source time-series multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; a processing unit configured to encode the source time-series multi-view images in an initial model to obtain two-dimensional multi-scale features, perform visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features, and perform occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; and a training unit configured to calculate a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and update parameters of the initial model based on the first loss value to obtain a first prediction model of the first vehicle model.

[0139] Optionally, the generation module comprises: a first generation unit configured to generate a three-dimensional Gaussian rendering model using the visual annotation data; a second generation unit configured to generate a three-dimensional Gaussian display representation of the second vehicle model in each scene using the three-dimensional Gaussian rendering model, wherein the three-dimensional Gaussian display representation comprises geometric characteristics, color, and texture characteristics of the second vehicle model in each scene; and a rendering unit configured to parse a vehicle driving trajectory in the visual annotation data, and render a target time-series multi-view image of the second vehicle model in the three-dimensional Gaussian display representation based on the vehicle driving trajectory and the target camera parameters.

[0140] Optionally, the first generation unit comprises: an acquisition subunit configured to acquire source time-series multi-view images in the visual annotation data; an acquisition subunit configured to input the source time-series multi-view images into a three-dimensional Gaussian rendering model to obtain initialized three-dimensional Gaussian point clouds, wherein each of the Gaussian point clouds comprises a center point coordinate, a covariance matrix, color, and opacity information; a rendering subunit configured to parse source camera parameters of the source time-series multi-view images, project the three-dimensional Gaussian point clouds into a two-dimensional image plane using the source camera parameters, and render corresponding two-dimensional images; and a generation subunit configured to calculate a second loss value between the two-dimensional images and the source time-series multi-view images, and optimize parameters of the three-dimensional Gaussian point clouds using the second loss value to obtain an optimized three-dimensional Gaussian rendering model.

[0141] Optionally, the generating module comprises: a third generating unit configured to perform semantic automatic labeling and depth estimation on the original image data by using a visual language large model, to generate a pseudo-label semantic map and a pseudo-label depth map; a fourth generating unit configured to generate predicted three-dimensional occupancy semantic features of the original image data by using the first prediction model; a processing unit configured to input the predicted three-dimensional occupancy semantic features into a pre-constructed semantic prediction module and a depth prediction module respectively, to obtain a predicted semantic map and a predicted depth map, and input the predicted three-dimensional occupancy semantic features into the first prediction model to output an occupancy semantic prediction map; and a determining unit configured to determine the pseudo-label semantic map, the pseudo-label depth map, the predicted semantic map, the predicted depth map, and the occupancy semantic prediction map as self-supervised data.

[0142] Optionally, the second training data comprises a pseudo-label semantic map, a pseudo-label depth map, a predicted semantic map, a predicted depth map, and an occupancy semantic prediction map, and the second training module comprises: a first adjusting unit configured to adjust the first prediction model by using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model; a second adjusting unit configured to adjust the first intermediate model by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model; and a third adjusting unit configured to adjust the second intermediate model by using the occupancy semantic prediction map to obtain a second prediction model.

[0143] Optionally, the first adjusting unit comprises: a calculating sub-unit configured to calculate a third loss value between the pseudo-label semantic map and the predicted semantic map; and an optimizing sub-unit configured to optimize semantic perception capability of the first prediction model according to the third loss value to obtain the first intermediate model.

[0144] Optionally, the second adjusting unit comprises: a calculating sub-unit configured to calculate a fourth loss value between the pseudo-label depth map and the predicted depth map; and an optimizing sub-unit configured to optimize depth perception capability of the first intermediate model according to the fourth loss value to obtain the second intermediate model.

[0145] Optionally, the third adjusting unit comprises: a reading subunit, configured to read a first semantic category with the highest confidence and a second semantic category with the second highest confidence of each grid in the occupancy semantic prediction map; a calculation subunit, configured to calculate a semantic difference between the first semantic category and the second semantic category; a judgment subunit, configured to judge whether the semantic difference is greater than a preset threshold; a rendering subunit, configured to, if the semantic difference is greater than the preset threshold, render the occupancy semantic prediction map to obtain a rendered semantic map and a rendered depth map; the judgment subunit is further configured to judge whether the rendered semantic map is aligned with the pseudo-label semantic map and whether the rendered depth map is aligned with the pseudo-label depth map; and a training subunit, configured to, if the rendered semantic map is aligned with the pseudo-label semantic map and the rendered depth map is aligned with the pseudo-label depth map, perform supervised training on the second intermediate model by using the original image data and the occupancy semantic prediction map to obtain a second prediction model.

[0146] It should be noted that the above various modules can be implemented by software or hardware, and for the latter, the following implementation manners can be used, but are not limited thereto: the above modules are located in the same processor; or the above various modules are located in different processors in any combination.

[0147] Embodiment 3

[0148] Embodiments of the present application also provide a storage medium having a computer program stored therein, wherein the computer program is configured to execute the steps in any of the method embodiments described above when running.

[0149] Optionally, in the present embodiment, the storage medium can be configured to store a computer program for executing the following steps:

[0150] S1, training a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type;

[0151] S2, obtaining original data of a second vehicle type;

[0152] S3, analyzing a data type of the original data;

[0153] S4, if the original data is target camera parameters of the second vehicle model, generating target time sequence multi-view images of the second vehicle model according to the target camera parameters and the visual annotation data, and determining the target time sequence multi-view images and a three-dimensional occupancy semantic ground truth map in the visual annotation data as first training data of the second vehicle model; if the original data is unlabeled original image data collected by the second vehicle model, generating self-supervised data by using the original image data and the first prediction model, and determining the self-supervised data as second training data of the second vehicle model;

[0154] S5, training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle model, wherein the second prediction model is used for predicting an occupancy semantic map of the second vehicle model according to visual images of the second vehicle model.

[0155] Optionally, in the embodiment, the storage medium can include but is not limited to a U disk, a read-only memory (ROM), a random access memory (RAM), a mobile hard disk, a magnetic disk or an optical disk and various computer program storage media.

[0156] Embodiments of the present application also provide an electronic device including a memory and a processor, the memory storing a computer program, and the processor being configured to execute the computer program to perform the steps in any of the above method embodiments.

[0157] Optionally, the electronic device can further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0158] Optionally, in the embodiment, the processor can be configured to execute the following steps through the computer program:

[0159] S1, training a first prediction model of a first vehicle model by using visual annotation data, wherein the first prediction model is used for predicting an occupancy semantic map of the first vehicle model according to visual images of the first vehicle model;

[0160] S2, obtaining original data of a second vehicle model;

[0161] S3, analyzing the data type of the original data;

[0162] S4, if the original data is target camera parameters of the second vehicle model, generating target time sequence multi-view images of the second vehicle model according to the target camera parameters and the visual annotation data, and determining the target time sequence multi-view images and a three-dimensional occupancy semantic ground truth map in the visual annotation data as first training data of the second vehicle model; if the original data is unlabeled original image data collected by the second vehicle model, generating self-supervised data by using the original image data and the first prediction model, and determining the self-supervised data as second training data of the second vehicle model;

[0163] S5, training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle model, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle model according to a visual image of the second vehicle model.

[0164] Optionally, specific examples in the embodiments can refer to the examples described in the above embodiments and optional implementation manners, and the embodiments will not be described here again.

[0165] The device embodiments described above are only schematic, wherein the units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, i.e., can be located in one place or distributed on multiple network units. Part or all of the modules can be selected according to actual needs to achieve the purpose of the embodiments.

[0166] Through the above description of the embodiments, those skilled in the art can clearly understand that the embodiments can be implemented by means of software plus a general hardware platform, and of course can also be implemented by hardware. Based on such understanding, the above technical solutions essentially or in other words make contributions to the related art, and can be embodied in the form of a software product. The computer software product can be stored in a computer readable storage medium, such as a ROM / RAM, a magnetic disk, an optical disk, etc., and includes a plurality of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute the methods described in the various embodiments or some parts of the embodiments.

[0167] It is to be understood that the terminology used herein is for the purpose of describing particular example embodiments only and is not intended to be limiting. As used herein, the singular forms "a", "an" and "the" are intended to include the plural forms as well, unless the context clearly indicates otherwise. The terms "comprises", "comprising", "includes", "including" and "has" are inclusive and therefore specify the presence of stated features, steps, operations, elements, and / or components, but do not preclude the presence or addition of one or more other features, steps, operations, elements, components, and / or groups thereof. The method steps, processes, and operations described herein are not to be construed as necessarily requiring their performance in the particular order

[0168] The above description is merely that of the specific embodiments of the application and as such is not to be taken in a limiting sense. Various modifications and alterations of the embodiments described herein will become apparent to those skilled in the art from the foregoing description, which does not limit the generality presented. It is the intention that all such modifications and alterations be considered equaliy by the spirit and scope of this application. It is therefore intended to cover in the appended claims all such changes and alterations that come within the scope of this application.

Claims

1. A prediction method of an occupancy semantic map, characterized in that, The method comprises the following steps: training a first prediction model of a first vehicle type by using visual annotation data, wherein the first prediction model is used to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type; obtaining original data of a second vehicle type; analyzing a data type of the original data; if the original data is a target camera parameter of the second vehicle type, generating a target time sequence multi-view image of the second vehicle type according to the target camera parameter and the visual annotation data, and determining a three-dimensional occupancy semantic ground truth map in the visual annotation data and the target time sequence multi-view image as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generating self-supervised data by using the original image data and the first prediction model, and determining the self-supervised data as second training data of the second vehicle type; training the first prediction model by using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is used to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

2. The method of claim 1, wherein, The method of training a first prediction model of a first vehicle type by using visual annotation data comprises the following steps: obtaining visual annotation data collected by a first vehicle type, wherein the visual annotation data comprises source time sequence multi-view images and corresponding three-dimensional occupancy semantic ground truth maps; encoding the source time sequence multi-view images in an initial model to obtain two-dimensional multi-scale features, performing visual conversion on the two-dimensional multi-scale features to obtain sample three-dimensional occupancy semantic features, and performing occupancy semantic decoding on the sample three-dimensional occupancy semantic features to obtain an occupancy semantic prediction map; calculating a first loss value between the occupancy semantic prediction map and the three-dimensional occupancy semantic ground truth map, and updating parameters of the initial model based on the first loss value to obtain a first prediction model of the first vehicle type.

3. The method of claim 1, wherein, The method of generating a target time sequence multi-view image of a second vehicle type according to a target camera parameter and visual annotation data comprises the following steps: generating a three-dimensional Gaussian rendering model by using the visual annotation data; generating a three-dimensional Gaussian display representation of the second vehicle type in each scene by using the three-dimensional Gaussian rendering model, wherein the three-dimensional Gaussian display representation comprises geometric characteristics, color and texture characteristics of the second vehicle type in each scene; analyzing a vehicle driving trajectory in the visual annotation data, and rendering a target time sequence multi-view image of the second vehicle type in the three-dimensional Gaussian display representation based on the vehicle driving trajectory and the target camera parameter.

4. The method of claim 3, wherein, The method of generating a three-dimensional Gaussian rendering model by using the visual annotation data comprises the following steps: obtaining source time sequence multi-view images in the visual annotation data; inputting the source time sequence multi-view images into a three-dimensional Gaussian rendering model to obtain initialized three-dimensional Gaussian point clouds, wherein each Gaussian point cloud contains a center point coordinate, a covariance matrix, color and opacity information; analyzing source camera parameters of the source time sequence multi-view images, and projecting the three-dimensional Gaussian point clouds into a two-dimensional image plane by using the source camera parameters, and rendering to obtain corresponding two-dimensional images. Calculate a second loss value between the two-dimensional image and the source time-series multi-view image, and optimize parameters of the three-dimensional Gaussian point cloud through the second loss value to obtain an optimized three-dimensional Gaussian rendering model.

5. The method of claim 1, wherein, Generating self-supervised data by using the original image data and the first prediction model comprises: Automatically labeling semantics and estimating depth of the original image data by using a visual language large model to correspondingly generate a pseudo-label semantic map and a pseudo-label depth map; Generating predicted three-dimensional occupancy semantic features of the original image data by using the first prediction model; Inputting the predicted three-dimensional occupancy semantic features into pre-constructed semantic prediction modules and depth prediction modules respectively to correspondingly obtain a predicted semantic map and a predicted depth map, and inputting the predicted three-dimensional occupancy semantic features into the first prediction model to output an occupancy semantic prediction map; Determining the pseudo-label semantic map, the pseudo-label depth map, the predicted semantic map, the predicted depth map, and the occupancy semantic prediction map as self-supervised data.

6. The method of claim 1, wherein, The second training data comprises a pseudo-label semantic map, a pseudo-label depth map, a predicted semantic map, a predicted depth map, and an occupancy semantic prediction map, and the first prediction model is trained by using the second training data to obtain a second prediction model of the second vehicle type, comprising: Adjusting the first prediction model by using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model; Adjusting the first intermediate model by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model; Adjusting the second intermediate model by using the occupancy semantic prediction map to obtain a second prediction model.

7. The method of claim 6, wherein, Adjusting the first prediction model by using the pseudo-label semantic map and the predicted semantic map to obtain a first intermediate model, comprising: Calculating a third loss value between the pseudo-label semantic map and the predicted semantic map; Optimizing semantic perception capability of the first prediction model according to the third loss value to obtain a first intermediate model.

8. The method of claim 6, wherein, Adjusting the first intermediate model by using the pseudo-label depth map and the predicted depth map to obtain a second intermediate model, comprising: Calculating a fourth loss value between the pseudo-label depth map and the predicted depth map; Optimizing depth perception capability of the first intermediate model according to the fourth loss value to obtain a second intermediate model.

9. The method of claim 6, wherein, Adjusting the second intermediate model by using the occupancy semantic prediction map to obtain a second prediction model, comprising: Reading a first semantic class with the highest confidence and a second semantic class with the second highest confidence of each grid in the occupancy semantic prediction map; Calculating a semantic difference value between the first semantic class and the second semantic class; Judging whether the semantic difference value is greater than a preset threshold value; If the semantic difference value is greater than the preset threshold value, rendering a rendering semantic map and a rendering depth map by using the occupancy semantic prediction map; Judging whether the rendering semantic map is aligned with the pseudo-label semantic map and whether the rendering depth map is aligned with the pseudo-label depth map; If the rendering semantic map is aligned with the pseudo-label semantic map and the rendering depth map is aligned with the pseudo-label depth map, the second prediction model is obtained by supervised training the second intermediate model using the original image data and the occupancy semantic prediction map.

10. A prediction device of an occupancy semantic map, characterized by, The method comprises the following steps: The first training module is configured to train a first prediction model of a first vehicle type using visual annotation data, wherein the first prediction model is configured to predict an occupancy semantic map of the first vehicle type according to a visual image of the first vehicle type. The acquisition module is configured to acquire original data of a second vehicle type. The analysis module is configured to analyze a data type of the original data. The generation module is configured to, if the original data is target camera parameters of the second vehicle type, generate target time-series multi-view images of the second vehicle type according to the target camera parameters and the visual annotation data, and determine the target time-series multi-view images and a three-dimensional occupancy semantic ground truth map in the visual annotation data as first training data of the second vehicle type; if the original data is unlabeled original image data collected by the second vehicle type, generate self-supervised data using the original image data and the first prediction model, and determine the self-supervised data as second training data of the second vehicle type. The second training module is configured to train the first prediction model using the first training data and / or the second training data to obtain a second prediction model of the second vehicle type, wherein the second prediction model is configured to predict an occupancy semantic map of the second vehicle type according to a visual image of the second vehicle type.

11. A storage medium, characterized by The storage medium stores a computer program, wherein the computer program is configured to execute the method described in any one of claims 1 to 9 when running. 12.An electronic device comprising a memory and a processor, the electronic device characterized by, The memory stores a computer program, and the processor is configured to execute the computer program to execute the method described in any one of claims 1 to 9.

Images