Scene reconstruction method for automatic driving simulation test, electronic equipment and product

Abstract

The invention provides a scene reconstruction method for an automatic driving simulation test, electronic equipment and a product, and relates to the technical field of computer vision. According to the specific implementation scheme, semantic segmentation is carried out on original image data, and long-shot image information and close-shot image information are determined; wherein the original image data comprises multiple frames of continuous images; constructing a texture map based on the long-shot image information, and sampling in the texture map to obtain a long-shot image; generating a close-shot spot cloud corresponding to the close-shot image information, and performing three-dimensional Gaussian modeling on each point in the close-shot spot cloud to obtain a close-shot three-dimensional Gaussian ellipsoid set; and scene reconstruction is carried out based on the close-shot three-dimensional Gaussian ellipsoid set and the long-shot image to obtain a multi-frame reconstruction scene graph in sequential arrangement. According to the technical scheme of the invention, the method can improve the estimation accuracy of the close-range depth value, achieves the more accurate reconstruction of a scene, and improves the simulation training effect of an automatic driving simulation system.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a scene reconstruction method, electronic equipment and product for autonomous driving simulation testing. Background Art

[0002] In the development and testing of autonomous driving technology, real-world testing is costly and poses safety risks. Through 3D reconstruction technology, realistic virtual test environments can be created that not only include roads, buildings, pedestrians and other vehicles, but also capture details in the environment, such as road signs, traffic lights and obstacles. These environments can simulate various driving scenarios and complex traffic conditions, allowing developers to conduct extensive testing and verification under safe and controllable conditions, thereby improving the reliability and safety of autonomous driving systems. Summary of the invention

[0003] The embodiments of the present application provide a scene reconstruction method, electronic device, computer-readable storage medium and computer program product for autonomous driving simulation testing to alleviate or solve one or more technical problems existing in the prior art.

[0004] In a first aspect, an embodiment of the present application provides a scene reconstruction method for autonomous driving simulation testing, comprising:

[0005] Performing semantic segmentation on the original image data to determine distant image information and near image information; wherein the original image data includes multiple frames of continuous images;

[0006] Construct a texture map based on the distant image information, and sample the distant image in the texture map;

[0007] Generate a near-field point cloud corresponding to the near-field image information, and perform three-dimensional Gaussian modeling on each point in the near-field point cloud to obtain a near-field three-dimensional Gaussian ellipsoid set;

[0008] The scene is reconstructed based on the near-view three-dimensional Gaussian ellipsoid set and the distant view image to obtain a multi-frame reconstructed scene graph arranged in a time sequence.

[0009] In a second aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, and the processor implements any method of the embodiment of the present application when executing the computer program.

[0010] In a third aspect, an embodiment of the present application provides a computer-readable storage medium, in which a computer program is stored. When the computer program is executed by a processor, any method of the embodiment of the present application is implemented.

[0011] In a fourth aspect, an embodiment of the present application provides a computer program product, including a computer program, which implements any method of the embodiments of the present application when executed by a processor.

[0012] Based on the method of the embodiment of the present application, the original image data in the autonomous driving simulation scene can be distinguished and rendered in different ways, thereby improving the accuracy of the estimation of the near-field depth value, and the near-field image is obtained by processing in a 3DGS rendering method, so that the scene can be reconstructed more accurately, thereby improving the simulation training effect of the autonomous driving simulation system.

[0013] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0014] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present application.

[0015] Figure 1 is a flow chart of a scene reconstruction method for autonomous driving simulation testing according to an embodiment of the present application;

[0016] Figure 2 is a schematic diagram of a scene reconstruction framework according to an embodiment of the present application;

[0017] Figure 3 A schematic diagram of a scene reconstruction device for autonomous driving simulation testing according to an embodiment of the present application;

[0018] Figure 4 It is a block diagram of an electronic device used to implement the embodiment of the present application. DETAILED DESCRIPTION

[0019] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0020] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0021] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0022] 1. Closed-loop simulation system: A comprehensive testing environment that integrates key components of an autonomous driving system (such as perception, decision-making, planning, and control) to form a feedback loop. In this environment, the output of the system (such as vehicle control instructions) will directly affect the simulation environment and generate new input data, thereby simulating the interaction between the vehicle and the environment in the real world. It can fully test the performance of the autonomous driving system in various complex scenarios to ensure its high safety and reliability before actual deployment.

[0023] 2. 3D Gaussian Splatting (3DGS): An advanced computer graphics method that represents a 3D scene by decomposing it into a large number of Gaussian distributions. Each Gaussian distribution represents a tiny volume element in the scene. The parameters of these Gaussian distributions (such as position, shape, and color) are adjusted during the optimization process to match the actual observed data, thereby achieving efficient rendering of complex scenes and realistic visual effects.

[0024] 3. Long-tail scenarios: In autonomous driving systems, there are a large number of rare but critical extreme situations or abnormal events. These situations have a low probability of occurring in actual driving, but are crucial in terms of safety and reliability. Such events are called long-tail scenarios or long-tail effects.

[0025] 4. Large Language Model (LLM): An AI technology that trains models on large amounts of text data to enable them to understand and generate natural language. These models typically have more than a billion parameters and are capable of performing complex language tasks. LLMs learn the patterns and structures of language through deep learning algorithms, thereby enabling advanced language understanding capabilities in a variety of application scenarios.

[0026] 5. AI Agent: An autonomous software entity that can perform specific tasks or services, usually without direct human intervention. They use machine learning and artificial intelligence techniques to make decisions, solve problems, and interact with the environment or other agents.

[0027] Lack of realism is one of the main challenges currently faced by closed-loop simulation systems for autonomous driving. Despite continuous technological advances, simulators still have limitations in simulating complex lighting, changeable weather, and precise physical behavior. The accuracy of sensor simulation and the richness of environmental details directly affect the perception and decision-making test results of the autonomous driving system, and current simulators often find it difficult to fully replicate the complexity and nuances of the real world.

[0028] In addition, another problem with current self-driving simulation technology is that data is limited, so scene editing is also a key factor for autonomous driving simulation. Through scene editing, limited scene data can be modified and edited, thereby expanding the number of sample scenarios including long-tail scenarios. However, many current simulator parameter settings are complex, making it difficult for users to quickly create and modify test scenarios.

[0029] Based on this, an embodiment of the present application proposes a scene reconstruction method for autonomous driving simulation testing, which can use three-dimensional Gaussian splash technology to render the close-up image in the original image, and further divide the close-up into a dynamic foreground including pedestrians and vehicles, and a static background including roads, buildings, trees and traffic facilities, and perform three-dimensional Gaussian modeling on them respectively, thereby improving the simulation quality of dynamic simulation scenes.

[0030] Figure 1 1 is a flow chart of a scene reconstruction method for autonomous driving simulation testing according to an embodiment of the present application, comprising:

[0031] Step S110, performing semantic segmentation on the original image data to determine distant image information and near image information; wherein the original image data includes a plurality of frames of continuous images.

[0032] Here, the original image data can be semantically segmented, and the semantic labels assigned to each pixel in the original image may include near-view labels and far-view labels, thereby determining far-view image information and near-view image information in the original image data, wherein the objects covered in the far-view image information include but are not limited to the sky, the horizon, and mountains, etc., and the objects covered in the near-view image information in the autonomous driving scenario include both static objects such as roads, buildings, and trees, as well as dynamic objects such as pedestrians and vehicles.

[0033] Step S120 , constructing a texture map based on the distant image information, and sampling the texture map to obtain the distant image.

[0034] Here, the original image data can be input into image processing tools such as OpenGL to load the image, and texture coordinates can be assigned to each vertex in the loaded image based on the previously segmented distant image information, so that after binding the texture, the geometry with texture coordinates can be rendered and drawn, that is, the texture map geometry is constructed, and sampling is performed from the texture map to extract distant images representing the sky, horizon, etc. in the original image data.

[0035] Due to the large uncertainty in the depth estimation of the distant point cloud, compared with the existing solution of performing three-dimensional Gaussian modeling on both the distant point cloud and the near point cloud after generating the point cloud structure corresponding to the original image and rendering the image after modeling, the embodiment of the present application adopts the form of constructing a texture map and sampling in the texture map to extract the distant image, which can avoid the interference of the predicted distant depth value on the near depth value in the three-dimensional Gaussian modeling process, thereby improving the accuracy of the prediction of the near depth value. For the autonomous driving simulation scenario, the simulation training process depends on the accuracy of the near object recognition, and thus can also improve the simulation training effect of the autonomous driving simulation system.

[0036] Step S130 , generating a near-view point cloud corresponding to the near-view image information, and performing three-dimensional Gaussian modeling on each point in the near-view point cloud to obtain a near-view three-dimensional Gaussian ellipsoid set.

[0037] Here, the SfM (Structure from Motion) method can be used to estimate the camera motion and the three-dimensional point cloud of objects in the scene based on the matching of feature points in a series of two-dimensional observation image sequences at different perspectives, thereby obtaining the point cloud of the entire scene corresponding to the close-up image information in the original image data (the spatiotemporal point cloud corresponding to multiple frames of continuous images) and the camera observation position corresponding to each frame of the image.

[0038] Among them, the near-view three-dimensional Gaussian ellipsoid set contains the three-dimensional Gaussian ellipsoid corresponding to each point, and the Gaussian function representation of each ellipsoid can be defined as:

[0039]

[0040] Among them, G(x) represents the probability density value of the Gaussian distribution at point x, which represents the probability of point x given the mean and covariance; μ represents the mean vector of the Gaussian distribution, which represents the center of the distribution. In 3DGS, it can represent the expected position of a certain geometric point or the center point of the surface. Σ represents the covariance matrix, which represents the shape and size of the ellipsoid and indicates the discreteness of the data and the correlation between dimensions.

[0041] Step S140 , reconstructing the scene based on the near-view three-dimensional Gaussian ellipsoid set and the distant view image to obtain a multi-frame reconstructed scene graph arranged in a time series.

[0042] In addition, the 3D Gaussian ellipsoid also has opacity α and color c. In the process of 3DGS rendering the image, by sorting the Gaussian points by depth, the properties of each 2D Gaussian rendering image can be queried and the color of each pixel can be determined based on voxel rendering.

[0043]

[0044] Where c represents the final color vector, which represents the cumulative color of the rendering results of all voxels or points. N represents the number of 3DGS on the ray. i Indicates the opacity of the i-th 3DGS. stands for cumulative transparency, which represents the product of the transparency of all voxels before the first i points. That is, the attenuation effect of the first (i-1) points on the light, which ensures the remaining contribution after the light passes through these points.

[0046] Here, the spatial position coordinates of the near-field cloud can be transferred from the world coordinate system to the camera coordinate system according to the camera pose of each frame of the original image previously determined, and the near-field image can be obtained by rendering the color and opacity of each pixel. The near-field image can then be spliced ​​with the distant image to obtain a multi-frame reconstructed scene graph arranged in a time series, and each frame of the reconstructed scene graph corresponds to a frame of the original image.

[0047] By adopting the method of this embodiment, for the original image data in the autonomous driving simulation scene, different methods can be used to render the distant image and the near image. The near image is obtained by 3DGS rendering and the distant image is obtained by texture map construction and sampling. The accuracy of the near depth value estimation is improved, so that the scene can be reconstructed more accurately, thereby improving the simulation training effect of the autonomous driving simulation system.

[0048] Exemplarily, the near-view three-dimensional Gaussian ellipsoid set obtained in step S130 includes a dynamic foreground ellipsoid set and a static background ellipsoid set, and the near-view three-dimensional Gaussian ellipsoid set is obtained by performing three-dimensional Gaussian modeling on each point in the near-view point cloud, including:

[0049] The target detection model is used to divide the dynamic foreground point cloud and the static background point cloud in the near-field point cloud. Three-dimensional Gaussian modeling is performed on each point in the dynamic foreground point cloud and each point in the static background point cloud, respectively, to obtain a dynamic foreground ellipsoid set corresponding to the dynamic foreground point cloud and a static background ellipsoid set corresponding to the static background point cloud. The spherical harmonic function of each three-dimensional Gaussian ellipsoid in the dynamic foreground ellipsoid set introduces a time parameter based on Fourier transform.

[0050] Here, the near point cloud can be further divided into dynamic foreground point cloud and static background point cloud for modeling. For each object in the dynamic foreground, the time dimension can be introduced on the basis of 3D Gaussian modeling to capture dynamic scene changes. Specifically, the spherical harmonic function z is used in the original 3D Gaussian ellipsoid. m,l To represent the color of the point cloud, in the process of dynamic foreground modeling, in order to simulate the appearance of dynamic objects, each spherical harmonic coefficient z m,l is replaced by a set of Fourier transform coefficients f i . Given a time step t, z m,l The Fourier coefficients can be recovered by taking an inverse discrete Fourier transform. Extending the traditional spherical harmonics z with time t as an extra dimension m,l , thus simulating the appearance of dynamic objects changing over time:

[0051]

[0052] Where k is the number of Fourier coefficients; N t is the number of time steps. T is the total time period. f i represents the i-th Fourier coefficient.

[0053] Based on this, each dynamic object in the dynamic foreground can be transformed into a set of 3DGS models with an optimizable dynamic appearance, which can get rid of the reliance on inter-frame motion estimation in the existing 3DGS solution in which a series of continuous frames in the scene are captured in time series reconstruction and the displacement information between frames is used to track dynamic objects, and the dynamic scene is processed directly from the perspective of 3DGS underlying modeling. Therefore, dynamic scene blur can be effectively consistent and the simulation quality can be improved.

[0054] Specifically, the target detection model is used to divide the dynamic foreground point cloud and the static background point cloud in the near point cloud, including: using the target detection model to detect dynamic objects in the near point cloud, and determining the three-dimensional bounding box corresponding to each dynamic object; marking the point cloud within the three-dimensional bounding box as a dynamic foreground point cloud, and marking other point clouds in the near point cloud as static background point clouds.

[0055] Here, the target detection model can be a pre-trained detection model for objects such as people, cars, and lane lines in autonomous driving scenarios. The target detection model is used to estimate the three-dimensional bounding box of the foreground object. Each three-dimensional bounding box includes a three-dimensional center point (x, y, z), a three-dimensional scale (l, w, h), a yaw angle (θ), and a category confidence c. Then, based on the three-dimensional bounding box, point clouds in different areas in the near point cloud can be labeled to distinguish dynamic foreground point clouds from static background point clouds.

[0056] Exemplarily, in step S130, three-dimensional Gaussian modeling is performed on each point in the near-view point cloud to obtain a near-view three-dimensional Gaussian ellipsoid set, including:

[0057] Based on three-dimensional Gaussian modeling of each point in the near-field cloud, a candidate ellipsoid set is obtained; Gaussian splash rendering is performed using the candidate ellipsoid set to obtain multiple frames of rendered images corresponding to multiple frames of continuous images in the original image data, and a normal vector map corresponding to each frame of the rendered image is obtained based on the multiple frames of rendered images; based on the normal vector map and viewing angle constraints, the candidate ellipsoid set is updated, and the updated ellipsoid set is used as the near-field three-dimensional Gaussian ellipsoid set.

[0058] Here, since the 3DGS rendering is overfitted to the observation perspective, when the perspective of the vehicle in the simulation system changes, a more serious observation degradation problem may occur. In order to solve the observation degradation problem, it is sometimes necessary to use multi-perspective original image data taken by multiple cameras to reconstruct the scene. However, such original image data is limited. Based on this, in order to solve the observation degradation problem of the 3DGS rendered image when each frame of the original image has only a single perspective, the present application sets a perspective constraint to guide the three-dimensional Gaussian modeling process for the near-point cloud.

[0059] Specifically, the viewing angle constraint condition includes that the normal vector difference of the same area in the rendered images of adjacent frames is less than a preset threshold, the candidate ellipsoid set is updated based on the normal vector map and the viewing angle constraint condition, and the updated ellipsoid set is used as the near-view three-dimensional Gaussian ellipsoid set, including:

[0060] Based on the normal vector map, determine the normal vector difference between the normal vector of each pixel area in each frame of the rendered image and the average normal vector of the corresponding area in at least one frame of the rendered image before the frame of the rendered image; if any pixel area in any rendered image does not meet the viewing angle constraint, re-perform three-dimensional Gaussian modeling on the pixel area that does not meet the viewing angle constraint to update the candidate ellipsoid set until each pixel area in each frame of the rendered image rendered by the updated ellipsoid set meets the viewing angle constraint, and the updated ellipsoid set is used as the foreground three-dimensional Gaussian ellipsoid set.

[0061] It can be understood that although the camera pose of each frame in the original image may be different, and thus the normal vectors in the corresponding pixel areas of each frame will be different, for one or several adjacent frames, the change in the normal vectors in the same pixel area is small. Based on this, a viewing constraint can be set. For example, the normal vector difference between the normal vectors of each spatial position of the current frame and the average of the normal vectors of the corresponding spatial positions of the previous frames is compared. If any position does not meet the viewing constraint, the three-dimensional Gaussian ellipsoid corresponding to the position can be eliminated from the candidate ellipsoid set, and the point at the position can be re-modeled with a three-dimensional Gaussian model, thereby updating the candidate ellipsoid set until each pixel area in each frame rendered image obtained by rendering the updated ellipsoid set meets the viewing constraint, and the ellipsoid set that meets the condition is used as the foreground three-dimensional Gaussian ellipsoid set.

[0062] By adopting the method of this embodiment, by adding perspective constraints to guide the three-dimensional Gaussian modeling process, the amplitude of the normal vector change at the same spatial position of adjacent rendered images after three-dimensional Gaussian modeling can be controlled, effectively reducing the rendering degradation problem caused by perspective changes, and improving the quality and reliability of rendered images under different observation conditions.

[0063] Exemplarily, after obtaining the multi-frame reconstruction scene graph arranged in time sequence, the method further includes:

[0064] In response to receiving a culling instruction for a target foreground object in a reconstructed scene graph, upsampling is performed on the region where the target foreground object is located using the trained network model, and based on the upsampling result, a three-dimensional Gaussian model is re-performed on the region where the target foreground object is located to update the foreground three-dimensional Gaussian ellipsoid set, and the scene is reconstructed again using the updated foreground three-dimensional Gaussian ellipsoid set.

[0065] It is understandable that, based on the use of 3DGS rendering, if the reconstructed scene graph needs to be edited, if the 3D Gaussian model corresponding to a certain object is directly deleted, a blurred area may appear in the rendered image. In addition to the target close-up object, the area where the target close-up object to be deleted is located may also contain other close-up objects. Therefore, this embodiment uses the trained network model to upsample the area, further extracts the features of the area, and restores the area after deleting the 3D Gaussian model of the target close-up object, so as to restore the clarity and fidelity of these areas without sacrificing details. This process not only improves the overall quality of the rendered image, but also ensures the smoothness of scene transitions and visual coherence.

[0066] Exemplarily, after obtaining the multi-frame reconstruction scene graph arranged in time sequence, the method further includes:

[0067] In response to receiving an instruction to add a new target object to the reconstructed scene graph, a user's description of the target object is obtained; an object model corresponding to the target object is matched in the material library using the trained large language model, and the target object is added to the reconstructed scene graph based on the object model.

[0068] Here, the large language model can be a GPT (Generative Pre-trained Transformer) series, such as GPT-3, GPT-4 and other models, a BERT (Bidirectional Encoder Representations from Transformers) series model, an ERNIE (a language model optimized for Chinese and other Asian languages) model, or a T5 (a unified text-to-text pre-trained model that can perform a variety of NLP tasks) model, etc.

[0069] For example, the user can enter the description "red car", and the big language model can quickly match at least one object model that meets the user's description in the material library based on the description. The user can select the model from the results output by the big model and enable the simulation system to use tools such as Unity and Blender to add objects.

[0070] By introducing a large language model into the autonomous driving simulation system to drive the artificial intelligence agent to assist users in adding objects in the reconstructed scene, users only need to provide a description of the target object, and the large language model can intelligently encode and quickly match and output the object model closest to the user's description in the material library, greatly simplifying the object addition process and improving the efficiency of the operation. Through the intelligent assistance of the large language model, the interactivity and user experience of the simulation system have been significantly improved.

[0071] Figure 2 It is a schematic diagram of a scene reconstruction framework according to an embodiment of the present application.

[0072] like Figure 2 As shown in the figure, after the original image is input into the simulation system, the system can perform semantic segmentation on the original image data to divide the near-view image information and the far-view image information. For the far-view image information, the far-view image can be extracted based on the far-view pixel vector in the form of texture map construction and sampling; for the near-view image information, the SFM point cloud estimation method can be used to obtain the spatiotemporal point cloud of the near-view part of the entire original image scene, and the target detection model can be used to distinguish the dynamic foreground information (people and vehicles, etc.) and the static background information (trees and buildings, etc.) in the point cloud, so as to perform 3DGS encoding modeling on the dynamic foreground and the static background respectively, and introduce the perspective consistency constraint in the modeling process to overcome the rendering degradation problem caused by the perspective change, so as to use the 3DGS of the far-view image and the near-view image information to reconstruct the scene to obtain the 3D reconstructed scene corresponding to the original image data.

[0073] Furthermore, after obtaining the three-dimensional reconstructed scene, the simulation system also optimizes the editing function of the reconstructed scene. After receiving the object removal instruction, it can use the trained network model to upsample the area where the object to be deleted is located, so as to facilitate the restoration of the original area after the object is deleted; after receiving the object addition instruction, it can use the large language model based on the user description to quickly match the object model closest to the user description to support the simulation system to use tools such as Unity and Blender to realize object addition.

[0074] The above describes the specific settings and implementation methods of the embodiments of the present application from different perspectives. The method provided by the above embodiments can not only improve the accuracy of the near-field depth value estimation, but also effectively reduce the rendering degradation problem caused by the change of viewing angle, and improve the quality and reliability of the rendered image under different observation conditions, so that the scene can be reconstructed more accurately, and the simulation training effect of the autonomous driving simulation system can be improved.

[0075] In addition, the editing function after scene reconstruction optimized in the embodiment of the present application can enrich the limited original simulation data, expand the number of sample scenes including long-tail scenes, ensure the accuracy of the edited scene, and improve the interactivity and user experience of the simulation system.

[0076] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of the relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0077] As an implementation of the above methods, the present application also provides a scene reconstruction device for autonomous driving simulation testing, which can be applied to an autonomous driving simulation system, such as Figure 3 As shown, the device may include:

[0078] The semantic segmentation module 310 is used to perform semantic segmentation on the original image data to determine the distant image information and the near image information; wherein the original image data includes multiple frames of continuous images;

[0079] A distant view processing module 320, configured to construct a texture map based on the distant view image information, and sample the distant view image in the texture map;

[0080] A near-view processing module 330 is used to generate a near-view point cloud corresponding to the near-view image information, and perform three-dimensional Gaussian modeling on each point in the near-view point cloud to obtain a near-view three-dimensional Gaussian ellipsoid set;

[0081] The scene reconstruction module 340 is used to reconstruct the scene based on the near-view three-dimensional Gaussian ellipsoid set and the distant view image to obtain a multi-frame reconstructed scene graph arranged in a time sequence.

[0082] Exemplarily, the near view three-dimensional Gaussian ellipsoid set includes a dynamic foreground ellipsoid set and a static background ellipsoid set, and the near view processing module 330 is further used for:

[0083] Using the target detection model to divide the near-field point cloud into a dynamic foreground point cloud and a static background point cloud;

[0084] Three-dimensional Gaussian modeling is performed on each point in the dynamic foreground point cloud and each point in the static background point cloud respectively to obtain a dynamic foreground ellipsoid set corresponding to the dynamic foreground point cloud and a static background ellipsoid set corresponding to the static background point cloud; wherein the spherical harmonic function of each three-dimensional Gaussian ellipsoid in the dynamic foreground ellipsoid set introduces a time parameter based on Fourier transform.

[0085] Exemplarily, the foreground processing module 330 is also used to: use the target detection model to detect dynamic objects in the foreground point cloud, and determine the three-dimensional bounding box corresponding to each of the dynamic objects; mark the point cloud within the three-dimensional bounding box as a dynamic foreground point cloud, and mark other point clouds in the foreground point cloud as static background point clouds.

[0086] Exemplarily, the foreground processing module 330 is also used to: obtain a candidate ellipsoid set based on three-dimensional Gaussian modeling of each point in the foreground point cloud; use the candidate ellipsoid set to perform Gaussian splash rendering to obtain multiple frames of rendered images corresponding to multiple frames of continuous images in the original image data, and obtain a normal vector map corresponding to each frame of the rendered image based on the multiple frames of rendered images; update the candidate ellipsoid set based on the normal vector map and viewing angle constraints, and use the updated ellipsoid set as the foreground three-dimensional Gaussian ellipsoid set.

[0087] Exemplarily, the viewing angle constraint condition includes that the normal vector difference of the same area in adjacent frame rendered images is less than a preset threshold value, and the foreground processing module 330 is also used to: determine, based on the normal vector map, the normal vector difference between the normal vector of each pixel area in each frame of the rendered image and the average normal vector of the corresponding area in at least one frame of rendered image before the frame of the rendered image; if any pixel area in any of the rendered images does not meet the viewing angle constraint condition, re-perform three-dimensional Gaussian modeling on the pixel area that does not meet the viewing angle constraint condition to update the candidate ellipsoid set until each pixel area in each frame of the rendered image rendered by the updated ellipsoid set meets the viewing angle constraint condition, and use the updated ellipsoid set as the foreground three-dimensional Gaussian ellipsoid set.

[0088] Exemplarily, the device further includes an editing module, and the editing module is used to:

[0089] In response to receiving a culling instruction for a target foreground object in the reconstructed scene graph, upsampling the region where the target foreground object is located is performed using a trained network model, and based on the upsampling result, three-dimensional Gaussian modeling is re-performed on the region where the target foreground object is located to update the foreground three-dimensional Gaussian ellipsoid set, and the scene is reconstructed again using the updated foreground three-dimensional Gaussian ellipsoid set.

[0090] The editing module is also used to: in response to receiving an instruction to add a new target object in the reconstructed scene graph, obtain a user's description of the target object; use the trained large language model to match the object model corresponding to the target object in the material library, and add the target object to the reconstructed scene graph based on the object model.

[0091] The functions of each unit, module or sub-module in each device in the embodiments of the present application can refer to the corresponding description in the above method embodiments, and have corresponding beneficial effects, which will not be repeated here.

[0092] According to an embodiment of the present application, the present application also provides an electronic device, a computer-readable storage medium and a computer program product.

[0093] Figure 4 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Figure 4 As shown, the electronic device includes: a memory 401 and a processor 402. The memory 401 stores a computer program that can be run on the processor 402. When the processor 402 executes the computer program, the method in the above embodiment is implemented. The number of the memory 401 and the processor 402 can be one or more.

[0094] The electronic device also includes:

[0095] The communication interface 403 is used to communicate with external devices and perform data exchange transmission.

[0096] If the memory 401, the processor 402 and the communication interface 403 are implemented independently, the memory 401, the processor 402 and the communication interface 403 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0097] Optionally, in a specific implementation, if the memory 401, the processor 402 and the communication interface 403 are integrated on a chip, the memory 401, the processor 402 and the communication interface 403 can communicate with each other through an internal interface.

[0098] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0099] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the present application.

[0100] It should be understood that the processor may be a CPU, or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), FPGAs or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0101] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0102] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network or other programmable device. The computer instructions can be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0103] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example" or "some examples" etc. means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0104] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0105] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0106] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0107] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0108] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0109] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A scene reconstruction method for autonomous driving simulation testing, comprising: Performing semantic segmentation on the original image data to determine distant image information and near image information; wherein the original image data includes multiple frames of continuous images; Constructing a texture map based on the distant image information, and sampling the distant image in the texture map; Generate a near-view point cloud corresponding to the near-view image information, and perform three-dimensional Gaussian modeling on each point in the near-view point cloud to obtain a near-view three-dimensional Gaussian ellipsoid set; The scene is reconstructed based on the near-view three-dimensional Gaussian ellipsoid set and the distant view image to obtain a multi-frame reconstructed scene graph arranged in a time sequence.

2. The method according to claim 1, wherein: The near-view three-dimensional Gaussian ellipsoid set includes a dynamic foreground ellipsoid set and a static background ellipsoid set. The near-view three-dimensional Gaussian ellipsoid set is obtained by performing three-dimensional Gaussian modeling on each point in the near-view point cloud, including: Using the target detection model to divide the near-field point cloud into a dynamic foreground point cloud and a static background point cloud; Three-dimensional Gaussian modeling is performed on each point in the dynamic foreground point cloud and each point in the static background point cloud respectively to obtain a dynamic foreground ellipsoid set corresponding to the dynamic foreground point cloud and a static background ellipsoid set corresponding to the static background point cloud; wherein the spherical harmonic function of each three-dimensional Gaussian ellipsoid in the dynamic foreground ellipsoid set introduces a time parameter based on Fourier transform.

3. The method according to claim 2, wherein: The target detection model is used to divide the dynamic foreground point cloud and the static background point cloud from the near point cloud, including: Detecting dynamic objects in the near-sight cloud using the target detection model, and determining a three-dimensional bounding box corresponding to each of the dynamic objects; The point cloud within the three-dimensional bounding box is marked as a dynamic foreground point cloud, and the other point clouds in the near point cloud are marked as static background point clouds.

4. The method according to claim 1, wherein: A three-dimensional Gaussian model is performed on each point in the near-view point cloud to obtain a near-view three-dimensional Gaussian ellipsoid set, including: Based on three-dimensional Gaussian modeling of each point in the near point cloud, a candidate ellipsoid set is obtained; Performing Gaussian splash rendering using the candidate ellipsoid set to obtain multiple frames of rendered images corresponding to multiple frames of continuous images in the original image data, and obtaining a normal vector map corresponding to each frame of the rendered image based on the multiple frames of rendered images; The candidate ellipsoid set is updated based on the normal vector map and the viewing angle constraint condition, and the updated ellipsoid set is used as the near-view three-dimensional Gaussian ellipsoid set.

5. The method according to claim 4, wherein: The viewing angle constraint condition includes that the normal vector difference of the same area in the rendered images of adjacent frames is less than a preset threshold, the candidate ellipsoid set is updated based on the normal vector map and the viewing angle constraint condition, and the updated ellipsoid set is used as the near-view three-dimensional Gaussian ellipsoid set, including: Based on the normal vector map, determining a normal vector difference between a normal vector of each pixel region in the rendered image of each frame and an average normal vector of a corresponding region in at least one rendered image before the rendered image of the frame; If any pixel area in any of the rendered images does not satisfy the viewing angle constraint, three-dimensional Gaussian modeling is performed again on the pixel area that does not satisfy the viewing angle constraint to update the candidate ellipsoid set, until each pixel area in each frame of the rendered image rendered by the updated ellipsoid set satisfies the viewing angle constraint, and the updated ellipsoid set is used as the foreground three-dimensional Gaussian ellipsoid set.

6. The method according to claim 1, further comprising: In response to receiving a culling instruction for a target foreground object in the reconstructed scene graph, upsampling the region where the target foreground object is located is performed using a trained network model, and based on the upsampling result, three-dimensional Gaussian modeling is re-performed on the region where the target foreground object is located to update the foreground three-dimensional Gaussian ellipsoid set, and the scene is reconstructed again using the updated foreground three-dimensional Gaussian ellipsoid set.

7. The method according to claim 1, further comprising: In response to receiving an instruction to add a target object to the reconstructed scene graph, obtaining a user's description of the target object; The trained large language model is used to match the object model corresponding to the target object in the material library, and the target object is added to the reconstructed scene graph based on the object model.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 7.

Images