Automatic driving scene modeling method and computer equipment

Abstract

The invention provides a brand-new scene modeling pattern based on the Gaussian splashing technology and suitable for the industrial grade. In the new scene modeling norm, Gaussian primitives contained in a scene model do not need to store Gaussian parameters any more, a group of anchor points associated with feature vectors and a decoder are introduced into the scene model, and the decoder is used for decoding the feature vectors associated with the anchor points, so that the scene modeling norm is obtained. Gaussian parameters corresponding to Gaussian primitives in the area where the anchor points are located are dynamically generated. The performance consumption of scene model reconstruction can be reduced, the scene model reconstruction efficiency can be improved, and the Gaussian primitives contained in the scene model can be efficiently and stably edited by deleting or adding the anchor points.

Description

Technical Field

[0001] The embodiments in this specification relate to the field of artificial intelligence technology, and in particular to an autonomous driving scenario modeling method and computer device. Background Technology

[0002] With the continuous development of autonomous driving technology, the requirements for safety and reliability of autonomous driving systems are constantly increasing. To achieve efficient algorithm training and verification covering multiple scenarios, it is essential to construct scenario models that accurately reflect the complexity of the driving environment. These scenario models enable autonomous driving systems to repeatedly test the vehicle's perception, planning, and control modules in virtual driving environments, thereby reducing the risks and costs of actual road testing and improving development efficiency.

[0003] For a long time, synthesis engines and NeRF (Neural Radiance Fields) technology have been widely used for sensor-level simulation and scene reconstruction, but they have significant shortcomings in terms of data authenticity, editing flexibility and rendering efficiency, making it difficult to meet the closed-loop simulation requirements of the entire link from perception to control in autonomous driving systems. Summary of the Invention

[0004] According to a first aspect of this specification, a modeling method for an autonomous driving scene model is proposed. The scene model includes a three-dimensional scene composed of several Gaussian primitives, a predefined set of anchor points in the three-dimensional scene, and a neural network-based decoder. The anchor points in the set are associated with feature vectors. The decoder is used to decode the feature vectors associated with the anchor points to generate Gaussian parameters corresponding to the Gaussian primitives contained within the region where the anchor point is located. The method includes: Sensing data samples are obtained from a set of sensing data samples corresponding to the driving environment collected by sensors on the vehicle; wherein, the sensing data samples include sensing data corresponding to the driving environment and pose data corresponding to the vehicle. Based on the pose data, anchor points corresponding to the sensing data are selected from the set of anchor points; and the feature vector associated with the anchor point is input to the decoder so that the decoder can decode the feature vector to generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where the anchor point is located. Based on the Gaussian parameters, differentiable rendering is performed to generate simulated sensing data samples, and the training loss of the simulated sensing data samples and the sensing data is calculated. The learnable parameters contained in the decoder and the feature vector are adjusted based on the training loss.

[0005] Optionally, the vehicle is equipped with a variety of sensors; The various sensors include cameras and radar; the sensor data samples include image data samples acquired by the camera and point cloud data samples acquired by the radar; the scene model includes a first scene model obtained by modeling based on the image data samples and a second scene model obtained by modeling based on the point cloud data samples.

[0006] Optionally, the neural network-based decoder includes an MLP.

[0007] Optionally, the image corresponding to the 3D scene is divided into multiple overlapping image regions; wherein, different image regions are allocated different GPU hardware; the decoder is a decoder shared by each anchor point in the anchor point set; different GPU hardwares share the learnable parameters contained in the decoder; and, different GPU hardwares maintain the learnable parameters contained in the feature vector associated with the anchor point located in their corresponding image region. Based on the pose data, anchor points corresponding to the sensing data are selected from the set of anchor points; and the feature vector associated with the anchor point is input to the decoder, so that the decoder decodes the feature vector to generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where the anchor point is located, including: The multiple GPUs, based on the pose data, in parallel select anchor points corresponding to the sensing data from the anchor points contained in the corresponding image regions; and input the feature vector associated with the anchor point to the decoder, so that the decoder decodes the feature vector to generate Gaussian parameters corresponding to the Gaussian primitives contained in the corresponding image regions. Based on the Gaussian parameters, differentiable rendering is performed to generate simulated sensing data samples. The training loss of the simulated sensing data samples and the sensing data is calculated, and the learnable parameters contained in the decoder and the feature vector are adjusted based on the training loss, including: The multiple GPUs perform differentiable rendering of their corresponding image regions in parallel based on the Gaussian parameters, generating multiple local simulation sensing data that correspond one-to-one with the multiple image regions, and the multiple GPUs calculate the training loss of the sensing data and the multiple local simulation sensing data in parallel. The multiple GPUs compute the shared gradient corresponding to the learnable parameters contained in the decoder in parallel based on the training loss, and adjust the learnable parameters contained in the decoder based on the computed shared gradient; and the multiple GPUs compute the gradient corresponding to the learnable parameters contained in the feature vector associated with the anchor points in the corresponding image region in parallel based on the training loss, and adjust the maintained learnable parameters contained in the feature vector associated with the anchor points in the corresponding image region in parallel based on the computed gradient.

[0008] Optionally, different GPU hardware can be allocated to different image regions, including: The rendering time of the image regions allocated to each GPU in the previous iteration for modeling the scene model is calculated; or, the average rendering time of the image regions allocated to each GPU in all completed iterations for modeling the scene model is calculated. The area of ​​the image region allocated to each GPU is dynamically adjusted based on the rendering time or the average rendering time.

[0009] Optionally, the method further includes: Acquire the vehicle's pose data for generating simulation sensing data; Based on the pose data, at least one anchor point corresponding to the simulated sensing data is selected from the set of anchor points, and the feature vector associated with the at least one anchor point is input into the decoder included in the trained scene model, so that the decoder decodes the feature vector of the at least one anchor point to generate Gaussian parameters corresponding to a number of Gaussian primitives contained in the image region where the at least one anchor point is located. Based on the Gaussian parameters, differentiable rendering is used to generate simulated sensor data. Optionally, the vehicle is equipped with multiple sensors; these sensors have a unified projection model; the projection model is used to define the light emitted from the multiple sensors. Based on the Gaussian parameters, differentiable rendering is performed to generate simulated sensing data, including: Calculate at least one target Gaussian primitive among a plurality of Gaussian primitives contained in the image region containing the at least one anchor point, which intersects with the ray defined for the multiple sensors by the unified projection model; Simulation sensing data is generated by differentiable rendering based on the Gaussian parameters corresponding to the at least one target Gaussian primitive.

[0010] Optionally, the unified projection model parameterizes the light emitted from the various sensors using the intersection of two orthogonal planes.

[0011] Optionally, calculating the Gaussian primitives intersecting with the ray includes: Whether a Gaussian primitive intersects with a ray is determined by solving the following equation on the Gaussian plane: ; and These represent the two orthogonal planes respectively; Represents the coordinates of a two-dimensional Gaussian primitive defined on a local tangent plane.

[0012] Optionally, if the sensor is a vacuum camera, when given image coordinates ( , When ), the two orthogonal planes are respectively represented as: ; If the sensor is a fisheye camera, when given image coordinates ( , When ), the two orthogonal planes are respectively represented as: ; in, Indicates the angle of elevation. ; and Indicates camera intrinsic parameters; If the sensor is a radar, when a given azimuth angle and elevation angle The two orthogonal planes are respectively represented as: , .

[0013] Optionally, the method further includes: Obtain extended pose data; wherein the extended pose data is different from the pose data contained in the sensor data samples in the sensor data sample set; The extended pose data and the sensor data samples are input into a pre-trained diffusion model so that the diffusion model generates extended sensor data samples with different pose data from the sensor data samples based on the extended pose data. The extended sensor data sample is added to the sensor data sample set.

[0014] Optionally, the method further includes: In response to the deletion operation of foreground objects in the 3D scene contained in the trained scene model, a target anchor point corresponding to the target image region where the foreground object is located is determined, and the feature vector associated with the target anchor point is deleted.

[0015] Optionally, the method further includes: Calculate the visual similarity between the target image region and other image regions contained in the 3D scene; An anchor point is identified in the other image regions that has the highest visual similarity to the target image region. The feature vector associated with the target anchor point is replaced with the feature vector associated with the anchor point to perform visual restoration on the target image region.

[0016] Optionally, the preset object model set includes several object models; the object model is a model trained using the same training method as the scene model; the object model includes a three-dimensional scene composed of several Gaussian primitives, a predefined set of anchor points in the three-dimensional scene, and a decoder based on a neural network; The method further includes: In response to the operation of adding a target object model from a preset set of object models to the scene model, an initial parameter matrix specified for the target object model is obtained, and the target object model is added to the scene model as a foreground object; wherein, the initial parameter matrix includes a matrix composed of a variety of parameters specified for initializing the target object model.

[0017] Optionally, the method further includes: Based on the initial parameter matrix of at least one object model added as a foreground object in the scene model, annotation information corresponding to the at least one object model is generated; The annotation information corresponding to the at least one object model is used as the supervision data corresponding to the simulated sensing data to construct training samples, and the perception model in the autonomous driving system is trained based on the training samples; or, the annotation information corresponding to the at least one object model is used as the supervision data corresponding to the simulated sensing data to construct test samples, and the test samples are further input into the perception model in the autonomous driving system, so that the perception model performs perception calculations on the simulated sensing data, generates annotation information corresponding to the foreground objects contained in the image corresponding to the simulated sensing data, compares the generated annotation information with the annotation information used as supervision data, and performs performance testing on the perception model based on the comparison results.

[0018] Optionally, the method further includes: The simulated sensing data is further input into the end-to-end autonomous driving algorithm in the autonomous driving system, so that the autonomous driving algorithm can calculate the vehicle's pose data at the next moment based on the simulated sensing data. Obtain the vehicle's pose data at the next moment calculated by the autonomous driving algorithm; Based on the pose data, at least one anchor point corresponding to the simulated sensing data is selected from the set of anchor points, and the feature vector associated with the at least one anchor point is input into the decoder included in the trained scene model, so that the decoder decodes the feature vector of the at least one anchor point to generate Gaussian parameters corresponding to a number of Gaussian primitives contained in the image region where the at least one anchor point is located. Based on the Gaussian parameters, differentiable rendering is performed to generate simulated sensing data of the vehicle at the next moment. The simulated sensing data of the vehicle at the next moment is then input into the end-to-end autonomous driving algorithm in the autonomous driving system to complete the closed-loop test of the autonomous driving algorithm.

[0019] Secondly, this specification also provides a computer device including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the steps of any of the methods described in the first aspect above.

[0020] Thirdly, this specification also provides a computer program product, including a computer program that, when executed by a processor, implements the steps of any of the methods described in the first aspect above.

[0021] In the above embodiments, by introducing a set of anchor points associated with feature vectors and a decoder into the scene model, and using the decoder to decode the feature vectors associated with the anchor points, Gaussian parameters corresponding to each Gaussian primitive are dynamically generated. This eliminates the need to store Gaussian parameters in the Gaussian primitives contained in the scene model. Furthermore, during the reconstruction training of the scene model, only the learnable parameters contained in the decoder and the feature vectors associated with the anchor points need to be adjusted, rather than adjusting the Gaussian parameters stored in the Gaussian primitives. This not only reduces the performance overhead of reconstructing the scene model but also improves its efficiency, making it suitable for large-scale industrial-grade scene model reconstruction. Moreover, by introducing a set of anchor points associated with feature vectors into the scene model, not only can the loosely linked Gaussian primitives in the scene model be managed in a structured manner, but the Gaussian primitives in the scene model can also be efficiently and stably edited by deleting or adding anchor points, improving the accuracy of scene model editing. Attached Figure Description

[0022] To more clearly illustrate the technical solutions of the embodiments in this specification, the drawings used in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this specification. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0023] Figure 1 This is a schematic diagram of the architecture of an Internet service system shown in one embodiment of this specification; Figure 2 This is a flowchart illustrating an autonomous driving scenario modeling method in one embodiment of this specification; Figure 3 This is a schematic structural diagram of an electronic device shown in one embodiment of this specification; Figure 4 This is a block diagram of an autonomous driving scene modeling device shown in one embodiment of this specification. Detailed Implementation

[0024] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numerals in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with one or more embodiments of this specification. Rather, they are merely examples of apparatuses and methods consistent with some aspects of one or more embodiments of this specification as detailed in the appended claims.

[0025] It should be noted that the steps of the corresponding methods are not necessarily performed in the order shown and described in this specification in other embodiments. In some other embodiments, the methods may include more or fewer steps than described in this specification. Furthermore, a single step described in this specification may be broken down into multiple steps in other embodiments; and multiple steps described in this specification may be combined into a single step in other embodiments.

[0026] The user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, data stored, data displayed, etc.) involved in this manual are all information and data authorized by the user or fully authorized by all parties. The collection, use and processing of related data shall comply with the relevant laws, regulations and standards of the relevant countries and regions, and corresponding operation portals shall be provided for users to choose to authorize or refuse.

[0027] GS (Gaussian Splatting) technology is a 3D scene rendering technology based on Gaussian functions. Based on this technology, the geometric structure, optical properties and spatial relationships of scene elements in a 3D scene can be explicitly modeled into Gaussian primitives (also known as Gaussian points or Gaussian spots) and visualized in an intuitive, high-fidelity and interactive way, thereby achieving efficient and realistic dynamic environment modeling.

[0028] Because GS technology has the characteristics of explicit expression, efficient real-time rendering and easy scene editing and expansion, it is very suitable for building autonomous driving simulation scenarios.

[0029] However, when modeling scenes using traditional Gaussian Geometry (GS) techniques, each Gaussian primitive obtained from the model needs to independently store Gaussian parameters (also known as attributes). For example, the Gaussian parameters stored in a Gaussian primitive may include the center position of the Gaussian plane, rotation components (defining the orientation of the Gaussian plane), scaling components (defining the size of the Gaussian in the tangential direction), appearance features, opacity / transparency, and so on. When optimizing and adjusting the scene model through training, it is usually necessary to perform fine-grained adjustments to the Gaussian parameters stored in all the Gaussian primitives within the scene model.

[0030] However, in industrial-grade autonomous driving scenarios, modeling large-scale urban scenes based on traditional GS technology requires adjusting the Gaussian parameters stored in millions or even billions of Gaussian primitives, resulting in huge memory consumption and difficulty in expansion.

[0031] Moreover, while explicitly modeling scene elements in the scene model as Gaussian primitives based on GS technology facilitates editing of scene elements in the scene model—for example, scene elements in the scene model can be quickly deleted or inserted into the scene model by deleting or adding new Gaussian primitives—the Gaussian primitives contained in the scene model are loose and unstructured. Therefore, when editing scene elements in the scene model, it is necessary to accurately select all Gaussian primitives belonging to that scene element. This process is not only complex but also very error-prone.

[0032] It is evident that traditional GS technology is not only unsuitable for modeling large-scale industrial-grade scene models, but also still suffers from the weakness of difficulty in efficiently and stably editing scene models.

[0033] Based on this, this specification proposes a new scene modeling paradigm based on GS technology suitable for industrial applications.

[0034] In this new scene modeling paradigm, the Gaussian primitives contained in the scene model no longer need to store Gaussian parameters. Instead, a set of anchors associated with feature vectors and a decoder are introduced into the scene model. The decoder is used to decode the feature vectors associated with the anchors to dynamically generate Gaussian parameters corresponding to the Gaussian primitives in the region where the anchors are located.

[0035] By adopting this novel scene modeling paradigm, on the one hand, during the reconstruction and training of the scene model, only the learnable parameters contained in the decoder and the feature vector associated with the anchor point need to be adjusted, instead of adjusting the Gaussian parameters stored in the Gaussian primitive. This not only reduces the performance consumption of reconstructing the scene model, but also improves the efficiency of reconstructing the scene model. This allows the scene modeling paradigm to overcome the shortcomings of traditional GS technology, which cannot be applied to industrial-grade large-scale scene models. On the other hand, by introducing a set of anchor points associated with feature vectors into the scene model, not only can the loose Gaussian primitives contained in the scene model be managed in a structured way through the anchor points, but the Gaussian primitives contained in the scene model can also be edited efficiently and stably by deleting or adding anchor points. This can overcome the shortcomings of traditional GS technology in that it is difficult to edit the scene model efficiently and stably.

[0036] Figure 1 This is a schematic diagram of the architecture of an Internet service system provided in an exemplary embodiment. For example... Figure 1 As shown, the system may include a server 11, a network 12, and several electronic devices, such as a PC (Personal Computer) 13, a mobile phone 14, etc.

[0037] Server 11 can be a physical server containing an independent host, or server 11 can be a virtual server hosted by a host cluster. During operation, server 11 can run server-side programs of a certain service program (such as an autonomous driving service program); for example, when server 11 runs the server-side program of the service program, it can act as the server of the service program.

[0038] PC13 and mobile phone 14 are just some of the types of electronic devices that users can use. In reality, users can obviously also use electronic devices such as tablets, laptops, PDAs (Personal Digital Assistants), wearable devices (such as smart glasses, smartwatches, etc.), etc., and one or more embodiments in this specification do not limit this. During operation, the electronic device can run a client-side program of a service program to implement the relevant functions of the application. For example, when the electronic device runs a user-side program of the service program, it can act as a client of the service program.

[0039] The client-side program of the aforementioned service program can be launched and run on an electronic device. This client-side program can be a native application installed on the electronic device, or it can be a mini-program, quick app, or other similar form.

[0040] Of course, when using web technologies such as HTML5 or similar, the relevant functions can be achieved through the page displayed by the browser. The browser here can be a standalone browser application or a browser module embedded in some applications.

[0041] As for the network 12 that enables interaction between electronic devices such as PC13 and mobile phone 14 and server 11, communication can be achieved using either wired or wireless networks, depending on the communication methods supported by the respective electronic devices. This specification does not impose any restrictions on this. For example, PC13 can support both wired and wireless communication, so it can use either wired or wireless networks as needed. Mobile phone 14 typically only supports wireless communication, so it can use a wireless network for communication.

[0042] It should be noted that the services provided by the above-mentioned service program can include any service implemented on the Internet; for example, the services provided by the above-mentioned service program may specifically include autonomous driving services provided to users.

[0043] The technical solution of this specification will be described in detail below with reference to the accompanying drawings.

[0044] Please see Figure 2 , Figure 2 The flowchart of an autonomous driving scenario modeling method shown in this specification includes the following execution process: Step 202: Obtain sensor data samples from the sensor data sample set corresponding to the driving environment collected by the sensors on the vehicle; wherein, the sensor data samples include sensor data corresponding to the driving environment and pose data corresponding to the vehicle. The aforementioned scenario model may specifically include a scenario model obtained by modeling the driving environment of autonomous driving based on GS technology.

[0045] Unlike traditional Gaussian Geometry (GS) technology, this specification proposes a novel scene modeling paradigm based on GS technology suitable for industrial applications. In this paradigm, the scene model can include a 3D scene composed of several Gaussian primitives, and can also predefine a set of anchor points in the 3D scene and introduce a decoder based on a neural network.

[0046] In autonomous driving scenarios, the aforementioned 3D scene specifically refers to a simulation scene corresponding to the vehicle's driving environment. Traditional Gaussian Geometry (GS) technology is a reconstruction technique based on 3D Gaussian primitives. The Gaussian primitives contained in the 3D scene are typically in the form of 3D ellipsoids. When using Gaussian primitives in the form of 3D ellipsoids to simulate the intersection of light rays with the surface of an object, there are inherent errors in the 3D projection calculation, making it impossible to accurately calculate the intersection point of the light rays and the Gaussian primitives.

[0047] In some embodiments, Gaussian primitives contained in a 3D scene can be transformed into 2D Gaussian primitives. 2D Gaussian primitives are typically defined on local tangent planes, which will be closer to the object surface.

[0048] In this way, the calculation of the intersection of a ray and a Gaussian primitive is simplified into a more accurate and stable problem of the intersection of a ray and a plane, thereby improving the accuracy of the calculation. For example, in one case, the aforementioned two-dimensional Gaussian primitive can specifically be a two-dimensional ellipsoid projected onto a local tangent plane.

[0049] The anchor points in the above set of anchor points can be used to manage the sparse Gaussian primitives contained in a region of a 3D scene in a structured way. Different anchor points can correspond to different regions in the 3D scene.

[0050] In practical applications, each anchor point in the above anchor point set can be associated with a feature vector; the above decoder can be used to decode the feature vectors associated with the anchor points in the above anchor point set in order to dynamically generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where the anchor point is located.

[0051] In this way, since the Gaussian parameters of the Gaussian primitives are dynamically generated based on the decoder's decoding of the aforementioned feature vectors, the Gaussian primitives contained in the aforementioned 3D scene no longer need to have their Gaussian parameters stored.

[0052] Both the aforementioned feature vector and the aforementioned decoder can contain several learnable parameters. In practical applications, the aforementioned scene model can be optimized and adjusted by iterative training of the aforementioned feature vector and the learnable parameters contained in the aforementioned decoder, thereby completing the modeling process of the aforementioned scene model.

[0053] The aforementioned decoder may specifically include any form of decoder based on a neural network structure, and will not be specifically limited in this specification. For example, in some embodiments, the aforementioned decoder may specifically be an MLP (Multilayer Perceptron) shared by each anchor point in the aforementioned anchor point set. Of course, in practical applications, a more complex multilayer neural network structure can also be designed as the aforementioned decoder.

[0054] It should be noted that the specific method of dividing the above 3D scene into regions and setting anchor points in the divided regions can be flexibly designed based on actual needs in practical applications.

[0055] In one embodiment shown, for the reconstruction of large-scale industrial scenes, the aforementioned 3D scene can be divided into several square regions of the same size in the XOY plane. In order to have sufficient contextual information when visualizing and rendering these square regions, these square regions can have a certain degree of overlap.

[0056] For example, in one scenario, a 3D scene can be divided into several 30m × 30m square regions, with adjacent square regions maintaining a 25% overlap.

[0057] After dividing the 3D scene into several square regions of equal size, an anchor point can be set for each square region. This anchor point can then be used to manage the discrete Gaussian primitives contained within that square region in a structured manner. For example, in one scenario, an anchor point can be set at one vertex or the center point of each square region.

[0058] Of course, in practical applications, in addition to dividing the above-mentioned 3D scene into several square regions of the same size, other region division methods can also be used.

[0059] For example, based on the geometric structure (i.e. the shape of the scene elements) of the three-dimensional scene, the three-dimensional scene can be adaptively divided into several regions of different sizes, and then some key feature points can be selected in each region to set anchor points.

[0060] In practical applications, when optimizing and adjusting the above scenario model through iterative training, the real sensor dataset (i.e., real driving data) collected by the vehicle's sensors corresponding to the driving environment can be obtained as training samples to iteratively train the above scenario model.

[0061] Among them, the sensor data samples in the above-mentioned sensor data sample set can usually include sensor data corresponding to the driving environment and pose data corresponding to the vehicle. For example, in practical applications, the sensors mounted on a vehicle can typically include multiple sensors. These multiple sensors may specifically include cameras (vacuum cameras, fisheye cameras, etc.) and radar. Correspondingly, if the sensor is a camera, the aforementioned sensor data sample set could specifically be a video sequence captured by the vehicle's camera, in which case the aforementioned sensor data sample could specifically be image data samples (image frames) captured by the camera; if the sensor is radar, the aforementioned sensor data sample set could specifically be a point cloud sequence captured by the vehicle's radar, in which case the aforementioned sensor data sample could specifically be point cloud data samples captured by the radar.

[0062] The aforementioned pose data may specifically include vehicle pose information calculated based on calibration parameters (intrinsic and extrinsic parameters) corresponding to the sensors.

[0063] For example, in practical applications, the sensor data samples collected by the sensor may include sensor data corresponding to the driving environment and calibration parameters corresponding to the sensor. In this case, the precise pose of the sensor (i.e., the sensor that collected the sensor data) in the world coordinate system can be calculated based on the calibration parameters, serving as the pose data corresponding to the vehicle. Of course, in practical applications, the sensor can also pre-calculate the pose data corresponding to the vehicle based on the calibration parameters and directly include the vehicle-corresponding pose data in the collected sensor data samples.

[0064] It is important to emphasize that when a vehicle is equipped with multiple sensors, the aforementioned scene model may specifically include multiple scene models. For example, if the aforementioned multiple sensors may specifically include cameras and radar, then the aforementioned sensing model may specifically include a first scene model obtained by modeling based on image data samples and a second scene model obtained by modeling based on point cloud data samples.

[0065] After obtaining the real sensor dataset corresponding to the driving environment collected by the sensors on the vehicle, the sensor data samples in the sensor dataset can be used as training samples to iteratively train the above-mentioned scenario model.

[0066] In one iteration, sensor data samples can be obtained from the sensor data sample set, and the scene model can be iteratively optimized based on these sensor data samples. For example, in practical applications, the sensor data sample set can be divided into N patches, so that in each round, a batch of sensor data samples can be obtained from the sensor data sample set, and the scene model can be trained based on this batch of sensor data samples using matrix operations.

[0067] Step 204: Based on the pose data, filter anchor points from the anchor point set that correspond to the sensing data; and input the feature vector associated with the anchor point to the decoder so that the decoder can decode the feature vector and generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where the anchor point is located. In one iteration, firstly, based on the pose data contained in the acquired sensor data sample, anchor points corresponding to the sensor data contained in the sensor data sample can be selected from the set of anchor points in the 3D scene. The process of selecting anchor points from the set of anchor points in the 3D scene that correspond to the sensor data contained in the sensor data sample is the process of selecting anchor points from the set of anchor points that are visually affected by the sensor data contained in the sensor data sample.

[0068] For example, in practical applications, a view frustum can be constructed using the vehicle's pose data and sensor intrinsic parameters (such as the field of view, FOV). Then, a spatial query can be performed on the anchor points throughout the entire 3D scene to quickly find all anchor points located within this view frustum. These queried anchor points correspond to the areas that can be seen from the current viewpoint.

[0069] Furthermore, after selecting anchor points from the set of anchor points in the 3D scene that correspond to the sensing data contained in the sensing data sample, the feature vectors associated with these anchor points can be input to the decoder, so that the decoder can decode the feature vectors and generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where these anchor points are located.

[0070] In practical applications, the process of the decoder decoding the feature vectors can specifically include the decoder learning the Gaussian parameters corresponding to the Gaussian primitives contained in the regions where these anchor points are located based on the feature vectors. The specific learning process will not be detailed in this specification.

[0071] For example, in practical applications, the aforementioned decoder can be a multi-layered end-to-end neural network. By training this neural network, it can learn the Gaussian parameters corresponding to the Gaussian primitives contained in the regions where these anchor points are located, based on the input feature vectors.

[0072] Step 206: Generate simulated sensing data samples by performing differentiable rendering based on the Gaussian parameters, and calculate the training loss of the simulated sensing data samples and the sensing data. Once the decoder decodes the Gaussian parameters corresponding to the Gaussian primitives contained in the regions where these anchor points are located, differentiable rendering can then be performed based on these Gaussian parameters to generate simulated sensor data samples.

[0073] Differentiable rendering is a special rendering mechanism used in GS technology. The rendering process of differentiable rendering will not be described in detail in this specification. Those skilled in the art can refer to the records in related technologies.

[0074] Furthermore, after generating simulated sensing data through differentiable rendering based on the aforementioned Gaussian parameters, the training loss between the simulated sensing data sample and the sensing data (i.e., the real sensing data) contained in the aforementioned sensing data sample can be calculated.

[0075] The training loss described above is typically used to describe the pixel-level differences between rendered simulated sensor data samples and real sensor images. For example, for simulated images, the pixel-level differences between simulated and real images can be calculated. Similarly, the pixel-level differences between simulated and real point clouds can be calculated.

[0076] It should be noted that the loss terms included in the above training loss can be flexibly designed and added or removed based on requirements in practical applications, and will not be specifically limited in this specification. For example, in practical applications, the loss terms included in the above training loss may specifically include injected photometric differences, geometric differences, or other types of pixel-level differences, etc.

[0077] Step 208: Adjust the learnable parameters contained in the decoder and the feature vector based on the training loss.

[0078] After calculating the training loss for one iteration, the learnable parameters contained in the above scenario model can be updated based on the backpropagation algorithm.

[0079] For example, these learnable parameters may specifically include the weight values ​​contained in the decoder and the feature vectors described above.

[0080] Unlike traditional GS techniques that update and record the Gaussian parameters in the Gaussian image in each iteration, this specification only updates the learnable parameters contained in the feature vectors of the decoder and the selected anchor points.

[0081] After adjusting the learnable parameters contained in the feature vectors of the decoder and the selected anchor points based on the training loss, one iteration of the scene model is complete. In subsequent iterations, steps 202-208 can be repeated until the scene model is reconstructed.

[0082] For example, convergence conditions for iterative optimization of the above scenario model may include reaching a preset number of iterations or the quality indicators of the rendered simulation sensor data stabilizing (e.g., key indicators no longer significantly improve), etc. In this case, when the preset number of iterations is reached or the quality indicators of the rendered simulation sensor data stabilize, iteration can be stopped, and the modeling of the scenario model is complete.

[0083] In industrial-scale scene modeling scenarios, in order to improve the speed of rendering simulation sensor data, a strategy of parallel training using multiple GPUs can be adopted to train large-scale scene models.

[0084] In some implementations, the image corresponding to the aforementioned 3D scene can be divided into multiple image regions; for example, these image regions can be square regions of the same size. To provide sufficient contextual information when visually rendering these square regions, they can have a certain degree of overlap. For instance, in one example, the 3D scene can be divided into several 30m × 30m square regions, with adjacent square regions maintaining a 25% overlap. Each square region can be assigned an anchor point, which is used to structurally manage the discrete Gaussian primitives contained within that square region. Specifically, the decoder can be a decoder shared by the anchor points corresponding to these square regions.

[0085] In addition, different GPU hardware can be allocated to different image regions; that is, the scene model can be trained in parallel using multiple GPUs that correspond one-to-one with all image regions. Furthermore, different GPUs can share the learnable parameters contained in the decoder; and each GPU can also share the learnable parameters contained in the feature vectors associated with anchor points within its corresponding image region. These parameters are those that the GPU needs to continuously optimize and adjust during iterative training.

[0086] In this case, during an iteration, the multiple GPUs can execute the iterative process described in steps 202-208 in parallel to train the above-mentioned scene model.

[0087] Specifically, after acquiring sensor data samples from the sensor data sample set, the multiple GPUs can, based on the pose data contained in the sensor data samples, select anchor points corresponding to the sensor data contained in the corresponding image regions in parallel. The feature vectors associated with these selected anchor points are then input to the decoder, which decodes the feature vectors to generate Gaussian parameters corresponding to the Gaussian primitives contained in the corresponding image regions.

[0088] Furthermore, after the decoder decodes the Gaussian parameters, the multiple GPUs can perform differentiable rendering on the corresponding image regions in parallel based on the Gaussian parameters, generating multiple local simulation sensing data samples that correspond one-to-one with the multiple image regions, and the multiple GPUs can calculate the training loss between the sensing data and the multiple local simulation sensing data samples in parallel.

[0089] After the training loss is calculated, on the one hand, the multiple GPUs can also calculate the shared gradient corresponding to the learnable parameters contained in the decoder in parallel based on the training loss, and adjust the shared learnable parameters contained in the decoder based on the calculated shared gradient. Here, the shared gradient refers to the gradient value generated by multiple GPUs in the process of training a unified scene model in parallel, exchanging their calculated gradients synchronously through a communication protocol, and aggregating (usually taking the average value) them globally to update the learnable parameters contained in the locally maintained decoder.

[0090] On the other hand, these multiple GPUs can also calculate, in parallel, the gradients corresponding to the learnable parameters contained in the feature vectors associated with anchor points in their respective image regions based on the aforementioned training loss, and adjust the learnable parameters contained in the maintained feature vectors associated with anchor points in their respective image regions in parallel based on the calculated gradients. Note that these gradients are not shared gradients and can only be used to adjust the learnable parameters contained in the feature vectors associated with anchor points in their respective image regions maintained by each GPU.

[0091] By employing a strategy of parallel training using multiple GPUs, the speed of rendering simulation sensor data can be significantly improved, making the novel scene modeling paradigm based on GS technology proposed in this specification more suitable for modeling large-scale industrial scene models.

[0092] In one embodiment shown, an adaptive strategy based on rendering duration can also be introduced to allocate continuous image regions contained in the scene model across multiple GPUs.

[0093] Under this strategy, the rendering time of the image regions allocated to each GPU in the previous iteration can be calculated; or, the average rendering time of the image regions allocated to each GPU in all completed iterations can be calculated; and then the area of ​​the image regions allocated to each GPU can be dynamically adjusted based on the rendering time or the average rendering time.

[0094] For example, in some embodiments, a dynamic scheduling algorithm for image region area can be designed based on this adaptive principle. This algorithm dynamically schedules the area of ​​image regions allocated to each GPU based on a metric such as rendering time, aiming to ensure that the rendering time for each GPU to render its corresponding image region is as similar as possible. For instance, if the rendering time of an image region exceeds a threshold, the area of ​​that image region is appropriately reduced; conversely, the area of ​​that image region is appropriately increased.

[0095] In some embodiments, based on this adaptive principle, a dynamic scheduling algorithm for the computing resources of each GPU can be designed. This algorithm schedules GPU resources based on an average rendering time metric and dynamically allocates the area of ​​the image region to each GPU, aiming to ensure that the rendering time for all GPUs on their corresponding image regions is as similar as possible. For example, if the average rendering time of an image region falls below a threshold, the 3D scene can be re-divided according to the actual computing resources of each GPU.

[0096] In practical applications, since the above-mentioned sensor data sample set is real data collected by the sensors mounted on the vehicle, and the usable perspectives contained in the real data are usually very limited, this may result in limited simulation effects when performing sensor simulation based on the finally trained scene model.

[0097] In some embodiments, a pre-trained diffusion model can be introduced to generate new perspective sensor data samples, thereby expanding the aforementioned sensor data sample set.

[0098] In this case, several extended pose data can be obtained; wherein, the extended pose data may be different from the pose data contained in the sensor data samples in the sensor data sample set.

[0099] Furthermore, a sensor data sample can be read from the sensor data sample set as a parameter, and the extended pose data and the sensor data sample mentioned above can be input into a pre-trained diffusion model. The diffusion model then generates an extended sensor data sample with a different pose data from the original sensor data sample based on the extended pose data. Specifically, this extended sensor data sample can be a new sensor data sample that is completely identical to the reference sensor data sample in all details except for the viewpoint. The extended sensor data sample generated by the extension model can also be added to the aforementioned sensor data sample set.

[0100] In this way, the diffusion model's ability to generate images can be utilized to train the scene model by adding new sensor data samples from different perspectives on the basis of limited real sensor data samples. This can improve the simulation effect of the trained scene model when performing sensor simulations.

[0101] Once the above scenario model has been trained, it can be used for sensor simulation.

[0102] In some embodiments, firstly, the vehicle's pose data for generating simulation sensing data can be acquired; specifically, this vehicle pose data may be the vehicle's position data sent to the scene model by the downstream autonomous driving system based on specific sensor simulation requirements. Typically, this vehicle pose data is a test item generated by the downstream autonomous driving system at a certain moment during testing.

[0103] In this case, after obtaining the vehicle's pose data for generating simulation sensing data, at least one anchor point corresponding to the final simulation sensing data can be selected from the aforementioned set of anchor points based on the pose data; the specific implementation details of selecting anchor points will not be elaborated here.

[0104] Then, the feature vector associated with the at least one anchor point can be input into the decoder contained in the trained scene model, so that the decoder can decode the feature vector of the at least one anchor point to generate Gaussian parameters corresponding to a number of Gaussian primitives contained in the image region where the at least one anchor point is located.

[0105] Once the Gaussian parameters are decoded, differentiable rendering can be performed based on these parameters to generate the final simulation sensing data.

[0106] In practical applications, some advanced downstream autonomous driving systems require hardware support from a variety of heterogeneous sensors. Radar and cameras, as the two most commonly used external perception sensors in vehicles, are crucial for vehicles to perceive their surroundings.

[0107] Because different types of sensors typically employ independent projection models, several sensor-specific problems arise when reconstructing scene models or simulating sensors based on these sensors. For example, there is the ray-drop pattern problem in point cloud frames acquired by radar, and the lens distortion problem in image frames acquired by fisheye cameras.

[0108] In some embodiments, a unified projection model can be defined for the various sensors mounted on the vehicle.

[0109] The projection model is usually a mathematical function or a set of rules that maps the sensor's observations to a virtual ray in the three-dimensional world based on specific sensor physical characteristics (such as lens geometry and scanning method), thereby defining the ray emitted by the sensor and simulating the ray propagation path corresponding to the sensor's observations.

[0110] In this scenario, during the sensor simulation phase using the trained scene model, after decoding the Gaussian parameters corresponding to several Gaussian primitives within the image region containing the at least one anchor point, the unified projection model can be used to define the light rays for the various sensors. Then, at least one target Gaussian primitive intersects with the light ray among the several Gaussian primitives within the image region containing the at least one anchor point. Finally, differentiable rendering can be performed based on the Gaussian parameters corresponding to the at least one target Gaussian primitive to generate simulated sensing data.

[0111] The aforementioned unified projection model can be specifically defined based on the physical characteristics of various sensors.

[0112] In one embodiment shown, taking the Gaussian primitives contained in the above-mentioned three-dimensional scene as two-dimensional Gaussian primitives as an example, in this case, the above-mentioned unified projection model can specifically use the intersection of two orthogonal planes to parameterize the light emitted from the above-mentioned multiple sensors.

[0113] For example, when parameterizing light rays emitted from the aforementioned multiple sensors using the intersection of two orthogonal planes, the Gaussian primitive that intersects with the light rays emitted by these multiple sensors can be used to determine whether the Gaussian primitive intersects with the light ray by solving the following equation on the Gaussian plane: ; in, and These represent the two orthogonal planes mentioned above; Represents the coordinates of a two-dimensional Gaussian primitive defined on a local tangent plane.

[0114] For different types of sensors, it is only necessary to construct the two orthogonal planes mentioned above based on their respective sensor characteristics to unify the specific projection scheme.

[0115] For example, in some embodiments, for a vacuum camera, when given image coordinates ( , When ), the two orthogonal planes mentioned above can be represented as: ; For fisheye cameras, when given image coordinates ( , When ), the two orthogonal planes mentioned above can be represented as: ; in, Indicates the angle of elevation. ; and Indicates camera intrinsic parameters; For radar, when given an azimuth angle and elevation angle The two orthogonal planes mentioned above can be represented as follows: , .

[0116] By defining a unified projection model for the various sensors, the problem of post-processing and stitching of multi-sensor data in traditional simulations, which makes it difficult to ensure spatiotemporal alignment, is solved. In this way, it can be ensured from the root that the simulation data of different modalities are geometrically derived from the same three-dimensional world model, greatly reducing the gap between simulation and reality.

[0117] In practical applications, simulated sensor data obtained from sensor simulations based on trained scene models can typically be used as test data to test downstream autonomous driving systems. However, during the testing of downstream autonomous driving systems, there is often a need to edit the trained scene models.

[0118] For example, based on the original scene model, a new test sample needs to be generated in a customized way for the autonomous driving system by deleting some scene elements (such as foreground objects) or adding some scene elements.

[0119] In some embodiments, downstream autonomous driving system personnel may delete foreground objects in the 3D scene contained in the trained scene model based on specific testing requirements.

[0120] In this case, in response to the deletion operation, the target image region where the foreground object is located can be determined first, then the target anchor point corresponding to the target image region can be determined, and then the Gaussian primitives corresponding to the foreground object contained in the target image region can be deleted in a structured batch by deleting the feature vector associated with the target anchor point.

[0121] Since the Gaussian parameters of the Gaussian primitives corresponding to the foreground objects contained in the target image region can only be obtained by decoding the feature vector associated with the anchor point in the target image region through the aforementioned decoder, deleting the feature vector associated with the anchor point is equivalent to directly deleting all Gaussian primitives contained in the region, thereby achieving the effect of clearing all Gaussian primitives with one click.

[0122] In some embodiments, after deleting the feature vector associated with the target anchor point corresponding to the target image region, the visual similarity between the target image region and other image regions included in the three-dimensional scene can be calculated; wherein, the visual similarity can be flexibly defined based on specific needs; then, the anchor point corresponding to the image region with the highest visual similarity to the target image region can be determined, and the feature vector associated with the target anchor point can be replaced based on the feature vector associated with the anchor point to perform visual restoration of the target image region.

[0123] In this way, the problem of significant differences in visual effect between the target image region and other image regions after deleting the feature vector associated with the target anchor point can be avoided, thus ensuring the consistency of visual effect across all regions.

[0124] In some embodiments, several object models can be trained in the same way as the scenario model described above, and an object model set can be constructed based on these object models.

[0125] These objects can be basic objects that serve as foreground objects in a scene model corresponding to a driving environment. For example, they can include objects that can appear in a driving environment, such as people, fences, vehicles, and manhole covers. Since the same training method is used, the object model can also specifically include a 3D scene composed of several Gaussian primitives. An anchor point set can be predefined in this 3D scene, and a decoder based on a neural network can also be introduced.

[0126] Downstream autonomous driving system personnel can also add target object models from the object model set to a specified image region in the scene model based on specific testing requirements, thereby expanding the scene model so that simulation sensing data that meets new testing requirements can be generated based on the expanded scene model.

[0127] In this case, in response to the addition operation, the initial parameter matrix specified by the staff for the target object model can be obtained, and the target object model can be added as a foreground object to the above scene model.

[0128] For example, in practical applications, a 3D interactive editing tool for the aforementioned object model set can be provided to staff. This tool allows staff to quickly add object models from the aforementioned object model set to the aforementioned scene model through methods such as dragging and dropping within the visual interface provided by the interactive editing tool, and to manually specify the initial parameter matrix for the object model.

[0129] Specifically, the initial parameter matrix mentioned above can include a matrix composed of various parameters specified by the staff for initializing the target object model. For example, these parameters may include the initial position of the object model in the aforementioned 3D scene, its initial orientation in the aforementioned 3D scene, the initial size of the object model, and so on. For dynamic objects, these parameters may also include the motion trajectory of the object in the 3D scene specified by the staff.

[0130] In some embodiments, the simulation sensing data obtained by sensor simulation based on the trained scene model can also be used as training data to train or test the perception model carried by the downstream autonomous driving system, so as to achieve data augmentation of the perception model.

[0131] In this scenario, after the worker deletes the original foreground objects in the aforementioned scene model and adds object models from the aforementioned object model set to the scene model, each foreground object in the scene model will have an initial parameter matrix specified by the worker. At this point, annotation information corresponding to at least one of the aforementioned object models can be generated based on the initial parameter matrix of the at least one object model added to the scene model as a foreground object. For example, a 3D bounding box corresponding to the at least one object model can be automatically generated, and some parameters can be annotated within the 3D bounding box.

[0132] On the one hand, after generating simulated sensing data based on the scene model, the generated annotation information corresponding to at least one of the above object models can be used as the supervision data corresponding to the simulated sensing data to construct training samples. Then, the training samples are further transmitted to the perception model carried by the downstream autonomous driving system, and the perception model is trained based on the training samples.

[0133] On the other hand, after generating simulated sensing data based on the scene model, the generated annotation information corresponding to at least one of the above-mentioned object models can be used as supervision data corresponding to the simulated sensing data to construct test samples. The test samples are then further input into the perception model mounted on the autonomous driving system, so that the perception model can perform perception calculations on the simulated sensing data to generate annotation information corresponding to the foreground objects contained in the image corresponding to the simulated sensing data. Then, the generated annotation information and the annotation information used as supervision data can be compared, and the performance of the perception model can be tested based on the comparison results.

[0134] In some embodiments, the simulation sensing data obtained by sensor simulation based on the trained scene model can also be used as test data to perform closed-loop testing on the end-to-end autonomous driving algorithm carried by the downstream autonomous driving system.

[0135] In this case, after the staff deletes the original foreground objects contained in the above scene model and adds the object models from the above object model set to the scene model, the simulation sensing data obtained by sensor simulation based on the scene model can be further input into the end-to-end autonomous driving algorithm in the downstream autonomous driving system, so that the autonomous driving algorithm can calculate the vehicle's pose data at the next moment based on the simulation sensing data. At this time, the vehicle's pose data at the next moment can be obtained by the autonomous driving algorithm, and at least one anchor point corresponding to the simulation sensing data can be selected from the set of anchor points in the three-dimensional scene based on the pose data. The feature vector associated with the at least one anchor point is input to the decoder included in the trained scene model. The decoder decodes the feature vector of the at least one anchor point to generate Gaussian parameters corresponding to a number of Gaussian primitives contained in the image region where the at least one anchor point is located. Then, based on the Gaussian parameters, differentiable rendering can be performed to generate simulated sensing data of the vehicle at the next moment. This simulated sensing data of the vehicle at the next moment can then be input into the end-to-end autonomous driving algorithm in the autonomous driving system to complete the closed-loop test of the autonomous driving algorithm.

[0136] In this way, it is possible to test whether the iteration of the end-to-end algorithm can regress the logical problems in past real driving segments in a safe simulation environment and make effective vehicle action decisions.

[0137] In the above technical solution, a set of anchor points associated with feature vectors and a decoder are introduced into the scene model. The decoder is used to decode the feature vectors associated with the anchor points to dynamically generate Gaussian parameters corresponding to each Gaussian primitive. This eliminates the need to store Gaussian parameters in the Gaussian primitives contained in the scene model. Furthermore, during the reconstruction training of the scene model, only the learnable parameters contained in the decoder and the feature vectors associated with the anchor points need to be adjusted, instead of adjusting the Gaussian parameters stored in the Gaussian primitives. This not only reduces the performance consumption of reconstructing the scene model but also improves the efficiency of reconstructing the scene model, thus making it suitable for industrial-scale scene model reconstruction.

[0138] Furthermore, by introducing a set of anchor points associated with feature vectors into the scene model, not only can the loose Gaussian primitives contained in the scene model be managed in a structured manner, but the Gaussian primitives contained in the scene model can also be edited efficiently and stably by deleting or adding anchor points, thereby improving the accuracy of editing the scene model.

[0139] Corresponding to the embodiments of the methods described above, this specification also provides embodiments of apparatus, electronic devices, and storage media.

[0140] Figure 3 This is a schematic structural diagram of an electronic device provided in an exemplary embodiment. Please refer to... Figure 3 At the hardware level, the device includes a processor 302, an internal bus 304, a network interface 306, memory 308, and non-volatile memory 310, and may also include other necessary hardware. One or more embodiments of this specification can be implemented in software, for example, the processor 302 reads the corresponding computer program from the non-volatile memory 410 into memory 308 and then runs it. Of course, in addition to software implementation, one or more embodiments of this specification do not exclude other implementation methods, such as logic devices or a combination of hardware and software, etc. That is to say, the execution subject of the following processing flow is not limited to each logic unit, but can also be hardware or logic devices.

[0141] like Figure 4 As shown, Figure 4 This is a block diagram illustrating a modeling apparatus for an autonomous driving scene model according to an exemplary embodiment of this specification. This apparatus can be applied to, for example... Figure 3The electronic device shown implements the technical solution of this specification. The scene model includes a three-dimensional scene composed of several Gaussian primitives, a predefined set of anchor points in the three-dimensional scene, and a neural network-based decoder; wherein the anchor points in the anchor point set are associated with feature vectors; the decoder is used to decode the feature vectors associated with the anchor points to generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where the anchor point is located; the device 400 includes: The acquisition module 401 acquires sensor data samples from a set of sensor data samples corresponding to the driving environment collected by the sensors mounted on the vehicle; wherein, the sensor data samples include sensor data corresponding to the driving environment and pose data corresponding to the vehicle. The generation module 402 filters anchor points corresponding to the sensing data from the anchor point set based on the pose data; and inputs the feature vector associated with the anchor point to the decoder so that the decoder decodes the feature vector to generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where the anchor point is located. The calculation module 403 generates simulated sensing data samples by performing differentiable rendering based on the Gaussian parameters, and calculates the training loss of the simulated sensing data samples and the sensing data. The adjustment module 404 adjusts the learnable parameters contained in the decoder and the feature vector based on the training loss.

[0142] Accordingly, this specification also provides an electronic device including a processor; a memory for storing processor-executable instructions; wherein the processor is configured to implement all the steps in the previously described method flow.

[0143] Accordingly, this specification also provides a computer-readable storage medium having stored thereon executable computer program instructions; wherein, when executed by a processor, the instructions implement all the steps in the previously described method flow.

[0144] Accordingly, this specification also provides a computer program product having executable computer program instructions stored thereon; wherein, when the computer program instructions are executed by a processor, they implement all the steps in the previously described method flow.

[0145] The systems, devices, modules, or units described in the above embodiments can be implemented by computer chips or physical entities, or by products with certain functions. A typical implementation device is a server system. Of course, it is not excluded that with the future development of computer technology, the computer implementing the functions of the above embodiments may be, for example, a personal computer, a laptop computer, an in-vehicle human-machine interaction device, a cellular phone, a camera phone, a smartphone, a personal digital assistant, a media player, a navigation device, an email device, a game console, a tablet computer, a wearable device, or any combination of these devices.

[0146] While one or more embodiments of this specification provide the operational steps of the methods described in the embodiments or flowcharts, more or fewer operational steps may be included based on conventional or non-inventive means. The order of steps listed in the embodiments is merely one possible order of execution among many steps and does not represent the only possible order. In actual device or end product execution, the methods shown in the embodiments or drawings may be executed sequentially or in parallel (e.g., in a parallel processor or multi-threaded processing environment, or even a distributed data processing environment). The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, product, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, product, or apparatus. Without further limitations, it is not excluded that the process, method, product, or apparatus that includes the elements may also have other identical or equivalent elements. For example, the use of terms such as "first," "second," etc., is used to indicate names and does not indicate any particular order.

[0147] For ease of description, the above devices are described in terms of function, divided into various modules. Of course, when implementing one or more of these specifications, the functions of each module can be implemented in one or more software and / or hardware components, or a module that performs the same function can be implemented by a combination of multiple sub-modules or sub-units. The device embodiments described above are merely illustrative. For example, the division of units is only a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, indirect coupling or communication connection between devices or units, and may be electrical, mechanical, or other forms.

[0148] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0149] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0150] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0151] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0152] Memory may include non-persistent storage in computer-readable media, such as random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of computer-readable media.

[0153] Computer-readable media, including both permanent and non-permanent, removable and non-removable media, can store information using any method or technology. Information can be computer-readable instructions, data structures, program modules, or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technologies, CD-ROM, digital versatile optical disc (DVD) or other optical storage, magnetic tape, magnetic magnetic disk storage, graphene storage or other magnetic storage devices, or any other non-transferable medium that can be used to store information accessible by a computing device. As defined herein, computer-readable media does not include transient computer-readable media, such as modulated data signals and carrier waves.

[0154] Those skilled in the art will understand that one or more embodiments of this specification can be provided as a method, system, or computer program product. Therefore, one or more embodiments of this specification may take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, one or more embodiments of this specification may take the form of a computer program product implemented on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0155] One or more embodiments of this specification can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a particular task or implement a particular abstract data type. One or more embodiments of this specification can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In a distributed computing environment, program modules can reside in local and remote computer storage media, including storage devices.

[0156] The various embodiments in this specification are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, system embodiments are basically similar to method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions in the method embodiments. In the description of this specification, the terms "one embodiment," "some embodiments," "example," "specific example," or "some examples," etc., refer to specific features, structures, materials, or characteristics described in connection with that embodiment or example, which are included in at least one embodiment or example of this specification. In this specification, the illustrative expressions of the above terms do not necessarily refer to the same embodiment or example. Furthermore, the specific features, structures, materials, or characteristics described can be combined in any suitable manner in one or more embodiments or examples. Moreover, without contradiction, those skilled in the art can combine and integrate the different embodiments or examples described in this specification and the features of different embodiments or examples.

[0157] The above description is merely an embodiment of one or more embodiments of this specification and is not intended to limit the scope of this specification. Various modifications and variations can be made to the one or more embodiments of this specification by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this specification should be included within the scope of the claims.

Claims

1. A method for modeling an autonomous driving scene model, wherein the scene model comprises a three-dimensional scene composed of several Gaussian primitives, a predefined set of anchor points in the three-dimensional scene, and a decoder based on a neural network; wherein, The anchor points in the set of anchor points are associated with feature vectors; the decoder is used to decode the feature vectors associated with the anchor points to generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where the anchor point is located; the method includes: Sensing data samples are obtained from a set of sensing data samples corresponding to the driving environment collected by sensors on the vehicle; wherein, the sensing data samples include sensing data corresponding to the driving environment and pose data corresponding to the vehicle. Based on the pose data, anchor points corresponding to the sensing data are selected from the set of anchor points; and the feature vector associated with the anchor point is input to the decoder so that the decoder can decode the feature vector to generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where the anchor point is located. Based on the Gaussian parameters, differentiable rendering is performed to generate simulated sensing data samples, and the training loss of the simulated sensing data samples and the sensing data is calculated. The learnable parameters contained in the decoder and the feature vector are adjusted based on the training loss.

2. The method as described in claim 1, wherein the image corresponding to the three-dimensional scene is divided into multiple overlapping image regions; wherein, Different image regions are assigned to different GPU hardware; the decoder is a shared decoder for each anchor point in the anchor point set; different GPU hardwares share the learnable parameters contained in the decoder; and different GPU hardwares maintain the learnable parameters contained in the feature vectors associated with the anchor points located in their corresponding image regions. Based on the pose data, anchor points corresponding to the sensing data are selected from the set of anchor points; And, the feature vector associated with the anchor point is input to the decoder, so that the decoder decodes the feature vector to generate Gaussian parameters corresponding to the Gaussian primitives contained in the region where the anchor point is located, including: The multiple GPUs, based on the pose data, in parallel select anchor points corresponding to the sensing data from the anchor points contained in the corresponding image regions; and input the feature vector associated with the anchor point to the decoder, so that the decoder decodes the feature vector to generate Gaussian parameters corresponding to the Gaussian primitives contained in the corresponding image regions. Based on the Gaussian parameters, differentiable rendering is performed to generate simulated sensing data samples. The training loss of the simulated sensing data samples and the sensing data is calculated, and the learnable parameters contained in the decoder and the feature vector are adjusted based on the training loss, including: The multiple GPUs perform differentiable rendering of their corresponding image regions in parallel based on the Gaussian parameters, generating multiple local simulation sensing data samples that correspond one-to-one with the multiple image regions, and the multiple GPUs calculate the training loss of the sensing data and the multiple local simulation sensing data samples in parallel. The multiple GPUs compute the shared gradient corresponding to the learnable parameters contained in the decoder in parallel based on the training loss, and adjust the learnable parameters contained in the decoder based on the computed shared gradient; and the multiple GPUs compute the gradient corresponding to the learnable parameters contained in the feature vector associated with the anchor points in the corresponding image region in parallel based on the training loss, and adjust the maintained learnable parameters contained in the feature vector associated with the anchor points in the corresponding image region in parallel based on the computed gradient.

3. The method of claim 2, further comprising: Acquire the vehicle's pose data for generating simulation sensing data; Based on the pose data, at least one anchor point corresponding to the simulated sensing data is selected from the set of anchor points, and the feature vector associated with the at least one anchor point is input into the decoder included in the trained scene model, so that the decoder decodes the feature vector of the at least one anchor point to generate Gaussian parameters corresponding to a number of Gaussian primitives contained in the image region where the at least one anchor point is located. Based on the Gaussian parameters, differentiable rendering is performed to generate simulation sensing data.

4. The method of claim 3, wherein the vehicle is equipped with multiple sensors; the multiple sensors have a unified projection model; the projection model is used to define the light emitted from the multiple sensors; Based on the Gaussian parameters, differentiable rendering is performed to generate simulated sensing data, including: Calculate at least one target Gaussian primitive among a plurality of Gaussian primitives contained in the image region containing the at least one anchor point, which intersects with the ray defined for the multiple sensors by the unified projection model; Simulation sensing data is generated by differentiable rendering based on the Gaussian parameters corresponding to the at least one target Gaussian primitive.

5. The method of claim 4, wherein the Gaussian primitive includes a two-dimensional Gaussian primitive; the unified projection model parameterizes the light emitted from the multiple sensors using the intersection of two orthogonal planes; calculating the Gaussian primitive intersecting the light rays includes: Whether a Gaussian primitive intersects with a ray is determined by solving the following equation on the Gaussian plane: ； and These represent the two orthogonal planes respectively; Represents the coordinates of a two-dimensional Gaussian primitive defined on a local tangent plane.

6. The method as described in claim 5, wherein if the sensor is a vacuum camera, when given image coordinates ( , When ), the two orthogonal planes are respectively represented as: ； If the sensor is a fisheye camera, when given image coordinates ( , When ), the two orthogonal planes are respectively represented as: ； in, Indicates the angle of elevation. ; and Indicates camera intrinsic parameters; If the sensor is a radar, when a given azimuth angle and elevation angle The two orthogonal planes are respectively represented as: ， 。 7. The method of claim 3, further comprising: Obtain extended pose data; wherein the extended pose data is different from the pose data contained in the sensor data samples in the sensor data sample set; The extended pose data and the sensor data samples are input into a pre-trained diffusion model so that the diffusion model generates extended sensor data samples with different pose data from the sensor data samples based on the extended pose data. The extended sensor data sample is added to the sensor data sample set.

8. The method of claim 7, further comprising: In response to the deletion operation of foreground objects in the 3D scene contained in the trained scene model, a target anchor point corresponding to the target image region where the foreground object is located is determined, and the feature vector associated with the target anchor point is deleted.

9. The method as described in claim 3, wherein the preset object model set includes several object models; the object model is a model trained using the same training method as the scene model; the object model includes a three-dimensional scene composed of several Gaussian primitives, a predefined set of anchor points in the three-dimensional scene, and a decoder based on a neural network; The method further includes: In response to the operation of adding a target object model from a preset set of object models to the scene model, an initial parameter matrix specified for the target object model is obtained, and the target object model is added to the scene model as a foreground object; wherein, the initial parameter matrix includes a matrix composed of a variety of parameters specified for initializing the target object model.

10. The method of claim 9, The method further includes: Based on the initial parameter matrix of at least one object model added as a foreground object in the scene model, annotation information corresponding to the at least one object model is generated; The annotation information corresponding to the at least one object model is used as the supervision data corresponding to the simulated sensing data to construct training samples, and the perception model in the autonomous driving system is trained based on the training samples; or, the annotation information corresponding to the at least one object model is used as the supervision data corresponding to the simulated sensing data to construct test samples, and the test samples are further input into the perception model in the autonomous driving system, so that the perception model performs perception calculations on the simulated sensing data, generates annotation information corresponding to the foreground objects contained in the image corresponding to the simulated sensing data, compares the generated annotation information with the annotation information used as supervision data, and performs performance testing on the perception model based on the comparison results.

11. The method of claim 9, further comprising: The simulated sensing data is further input into the end-to-end autonomous driving algorithm in the autonomous driving system, so that the autonomous driving algorithm can calculate the vehicle's pose data at the next moment based on the simulated sensing data. Obtain the vehicle's pose data at the next moment calculated by the autonomous driving algorithm; Based on the pose data, at least one anchor point corresponding to the simulated sensing data is selected from the set of anchor points, and the feature vector associated with the at least one anchor point is input into the decoder included in the trained scene model, so that the decoder decodes the feature vector of the at least one anchor point to generate Gaussian parameters corresponding to a number of Gaussian primitives contained in the image region where the at least one anchor point is located. Based on the Gaussian parameters, differentiable rendering is performed to generate simulated sensing data of the vehicle at the next moment. The simulated sensing data of the vehicle at the next moment is then input into the end-to-end autonomous driving algorithm in the autonomous driving system to complete the closed-loop test of the autonomous driving algorithm.

12. A computer device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.

Images