Scene rendering method and related device

Abstract

The invention discloses a scene rendering method and a related device, and relates to the technical field of rendering, and the scene rendering method comprises the steps: obtaining a three-dimensional space point set of a target scene; based on a Gaussian prediction model obtained by pre-training, predicting whether a Gaussian ellipsoid exists at each three-dimensional space point in the three-dimensional space point set, so as to obtain a target point set composed of the three-dimensional space points with the Gaussian ellipsoid; predicting attribute information of Gaussian ellipsoids existing at the three-dimensional space points in the target point set based on a Gaussian attribute prediction model obtained through pre-training; and rendering is carried out according to the attribute information of the Gaussian ellipsoids existing at the three-dimensional space points in the target point set, and a rendered graph of the target scene is obtained. The scene rendering method disclosed by the invention has relatively high rendering efficiency, relatively low memory overhead and relatively wide application range.

Description

Technical Field

[0001] The present application relates to the field of rendering technology, and in particular to a scene rendering method and related devices. Background Art

[0002] In some application fields (such as robotics, intelligent driving, etc.), it is necessary to render an image of a certain scene.

[0003] At present, a mainstream scene rendering solution is: obtain a three-dimensional space point set of the scene, predict the color information and transparency of each three-dimensional space point in the three-dimensional space point set of the scene, and render an image of the scene according to the color information and transparency of each three-dimensional space point in the three-dimensional space point set of the scene.

[0004] Although the above scene rendering scheme can realize scene rendering, its rendering efficiency is low. Summary of the invention

[0005] In view of this, the present application provides a scene rendering method and related devices, which are used to solve the problem of low rendering efficiency of existing scene rendering solutions. The technical solution is as follows:

[0006] The first aspect of the present application provides a scene rendering method, comprising:

[0007] Obtain the three-dimensional space point set of the target scene;

[0008] Based on a pre-trained Gaussian prediction model, predict whether a Gaussian ellipsoid exists at each three-dimensional space point in the three-dimensional space point set, so as to obtain a target point set consisting of three-dimensional space points where a Gaussian ellipsoid exists, wherein the Gaussian prediction model is trained using a number of labeled first training samples, any first training sample is a three-dimensional space point in the target scene, and the label of any first training sample is used to indicate whether a Gaussian ellipsoid exists at the training sample;

[0009] Based on the pre-trained Gaussian attribute prediction model, predict the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set, wherein the Gaussian attribute prediction model is trained using a number of labeled second training samples, any second training sample is a three-dimensional space point in the target scene where a Gaussian ellipsoid exists, and the label of any second training sample is the attribute information of the Gaussian ellipsoid existing at the second training sample;

[0010] Rendering is performed according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain a rendering image of the target scene.

[0011] In a possible implementation, the process of acquiring the plurality of labeled first training samples and the plurality of labeled second training samples includes:

[0012] Using a three-dimensional Gaussian splashing method, the target scene is represented as a plurality of Gaussian ellipsoids, each Gaussian ellipsoid having attribute information;

[0013] The plurality of labeled first training samples and the plurality of labeled second training samples are constructed according to the three-dimensional space point set of the target scene and the attribute information of a plurality of Gaussian ellipsoids representing the target scene.

[0014] In a possible implementation, constructing the plurality of labeled first training samples and the plurality of labeled second training samples according to the three-dimensional space point set of the target scene and the attribute information of the plurality of Gaussian ellipsoids representing the target scene includes:

[0015] For each 3D space point in the 3D space point set of the target scene:

[0016] According to the attribute information of the multiple Gaussian ellipsoids representing the target scene, it is judged whether there is a Gaussian ellipsoid at the three-dimensional space point, and a Gaussian judgment result of the three-dimensional space point is obtained;

[0017] The three-dimensional space point is used as a first training sample, and the Gaussian discriminant result of the three-dimensional space point is used as a label of the first training sample to obtain a first training sample with a label;

[0018] If a Gaussian ellipsoid exists at the three-dimensional space point, the three-dimensional space point is used as a second training sample, and the attribute information of the Gaussian ellipsoid existing at the three-dimensional space point is used as a label of the second training sample to obtain a labeled second training sample.

[0019] In a possible implementation, the target scene is represented as a plurality of Gaussian ellipsoids by using a three-dimensional Gaussian splashing method, including:

[0020] Acquire two-dimensional images of multiple viewing angles for the target scene;

[0021] According to the two-dimensional images of the multiple viewing angles, a three-dimensional Gaussian splashing method is used to perform three-dimensional modeling on the target scene to obtain multiple Gaussian ellipsoids representing the target scene.

[0022] In a possible implementation, the rendering is performed according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain a rendering of the target scene, including:

[0023] According to the specified viewing angle and the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set, a volume rendering method based on three-dimensional Gaussian splashing is adopted to render an image of the specified viewing angle of the target scene.

[0024] In a possible implementation, the training process of the Gaussian prediction model includes:

[0025] Predicting whether there is a Gaussian ellipsoid at the first training sample based on the Gaussian prediction model, and obtaining a Gaussian prediction result corresponding to the first training sample;

[0026] Determine the prediction loss of the Gaussian prediction model according to the Gaussian prediction result corresponding to the first training sample and the label of the first training sample;

[0027] According to the prediction loss of the Gaussian prediction model, the parameters of the Gaussian prediction model are updated.

[0028] In a possible implementation, the training process of the Gaussian attribute prediction model includes:

[0029] Predicting attribute information of the Gaussian ellipsoid existing at the second training sample based on the Gaussian attribute prediction model, and obtaining a Gaussian attribute prediction result corresponding to the second training sample;

[0030] Determine the prediction loss of the Gaussian attribute prediction model according to the Gaussian attribute prediction result corresponding to the second training sample and the label of the second training sample;

[0031] According to the prediction loss of the Gaussian attribute prediction model, the parameters of the Gaussian attribute prediction model are updated.

[0032] A second aspect of the present application provides a scene rendering device, comprising: a scene data acquisition module, a first prediction module, a second prediction module and a scene rendering module;

[0033] The scene data acquisition module is used to acquire a three-dimensional space point set of a target scene;

[0034] The first prediction module is used to predict whether there is a Gaussian ellipsoid at each three-dimensional space point in the three-dimensional space point set based on a pre-trained Gaussian prediction model, so as to obtain a target point set consisting of three-dimensional space points where a Gaussian ellipsoid exists, wherein the Gaussian prediction model is trained using a number of labeled first training samples, any first training sample is a three-dimensional space point in the target scene, and the label of any first training sample is used to indicate whether there is a Gaussian ellipsoid at the training sample;

[0035] The second prediction module is used to predict the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set based on the pre-trained Gaussian attribute prediction model, wherein the Gaussian attribute prediction model is trained using a number of labeled second training samples, any second training sample is a three-dimensional space point in the target scene where a Gaussian ellipsoid exists, and the label of any second training sample is the attribute information of the Gaussian ellipsoid existing at the second training sample;

[0036] The scene rendering module is used to render according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain a rendering image of the target scene.

[0037] A third aspect of the present application provides an electronic device, comprising at least one processor and a memory connected to the processor, wherein:

[0038] The memory is used to store computer programs;

[0039] The processor is used to execute the computer program so that the electronic device can implement the steps of any one of the scene rendering methods described above.

[0040] A fourth aspect of the present application provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of any one of the scene rendering methods described above.

[0041] By means of the above technical scheme, the scene rendering method provided by the present application first obtains the three-dimensional space point set of the target scene, and then predicts whether there is a Gaussian ellipsoid at each three-dimensional space point in the three-dimensional space point set of the target scene based on a pre-trained Gaussian prediction model, so as to obtain a target point set composed of three-dimensional space points where a Gaussian ellipsoid exists, and then predicts the attribute information of the Gaussian ellipsoid at each three-dimensional space point in the target point set based on a pre-trained Gaussian attribute prediction model, and finally renders according to the attribute information of the Gaussian ellipsoid at each three-dimensional space point in the target point set, thereby obtaining a rendering of the target scene. In the scene rendering method provided by the present application, the amount of computation for predicting whether there is a Gaussian ellipsoid at a three-dimensional space point based on the Gaussian prediction model is relatively small. Although the amount of computation for predicting the attribute information of the Gaussian ellipsoid at the three-dimensional space point where the Gaussian ellipsoid exists based on the Gaussian attribute prediction model is slightly larger, since the distribution of three-dimensional Gaussians in the entire space is relatively sparse, that is, the number of three-dimensional space points where there are Gaussian ellipsoids in the target scene is very small, it is only necessary to predict the attribute information of the Gaussian ellipsoid for fewer three-dimensional space points based on the Gaussian attribute prediction model. It can be seen that the overall amount of computation for the scene rendering method provided by the present application is not large, and furthermore, the scene rendering method provided by the present application has a higher rendering efficiency. BRIEF DESCRIPTION OF THE DRAWINGS

[0042] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying creative work.

[0043] Figure 1 A schematic diagram of a system architecture involved in this application;

[0044] Figure 2 A schematic diagram of a hardware structure of a terminal provided in an embodiment of the present application;

[0045] Figure 3 A schematic diagram of a hardware structure of a server provided in an embodiment of the present application;

[0046] Figure 4 A schematic diagram of a flow chart of a scene rendering method provided in an embodiment of the present application;

[0047] Figure 5 A schematic diagram of a process for obtaining a plurality of labeled first training samples and a plurality of labeled second training samples provided in an embodiment of the present application;

[0048] Figure 6 A schematic diagram of representing a scene as multiple Gaussian ellipsoids provided in an embodiment of the present application;

[0049] Figure 7 A schematic diagram of the structure of a scene rendering device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following describes the embodiments of the present application in conjunction with the drawings in the embodiments of the present application. The terms used in the implementation method section of the present application are only used to explain the specific embodiments of the present application, and are not intended to limit the present application.

[0051] The embodiments of the present application are described below in conjunction with the accompanying drawings. Those skilled in the art will appreciate that, with the development of technology and the emergence of new scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0052] The terms "first", "second", etc. in the specification and claims of the present application and the above-mentioned drawings are used to distinguish similar objects, and need not be used to describe a specific order or sequential order. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances, which is only to describe the distinction mode adopted by the objects of the same attributes when describing in the embodiments of the present application. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, so that the process, method, system, product or equipment comprising a series of units need not be limited to those units, but may include other units that are not clearly listed or inherent to these processes, methods, products or equipment.

[0053] In one possible implementation, Figure 1 As shown, the system architecture involved in the present application may include a terminal 101 and a server 102, and the terminal 101 may interact with the server 102 via a network (wired network or wireless network). The server 102 may include one or more servers ( Figure 1 The example includes a server. The terminal 101 can obtain the three-dimensional space point set of the target scene, transmit the three-dimensional space point set of the target scene to the server 102 through the network, the server 102 uses the scene rendering method provided in the present application to render the image of the target scene, and transmits the image of the target scene to the terminal 101 through the network, and the terminal 101 can output the image of the target scene.

[0054] In another possible implementation, the system architecture involved in the present application may include a terminal. The terminal has strong data processing capabilities. The terminal may render an image of a target scene using the scene rendering method provided in the present application, and then output the image of the target scene.

[0055] Next, the product form of the above-mentioned terminal is described.

[0056] The above-mentioned terminal can be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a robot, an augmented reality (AR) / virtual reality (VR) device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), etc., and the embodiments of the present application do not impose any restrictions on this.

[0057] Figure 2 An optional hardware structure diagram of a terminal is shown.

[0058] refer to Figure 2As shown, the terminal may include a radio frequency unit 210, a memory 220, an input unit 230, a display unit 240, a camera 250 (optional), an audio circuit 260 (optional), a speaker 261 (optional), a microphone 262 (optional), an earphone jack 263 (optional), a processor 270, an external interface 280, a power supply 290 and other components. Those skilled in the art will appreciate that Figure 2 This is merely an example of a terminal and does not constitute a limitation on the terminal. The terminal may include more or fewer components than shown in the figure, or a combination of certain components, or different components.

[0059] The input unit 230 can be used to receive input digital or character information, and generate key signal input related to the user settings and function control of the terminal. Specifically, the input unit 230 may include a touch screen 231 (optional) and / or other input devices 232. The touch screen 231 can collect the user's touch operations on or near it (such as the user's operation on or near the touch screen using any suitable object such as fingers, joints, stylus, etc.), and drive the corresponding connection device according to a pre-set program. The touch screen can detect the user's touch action on the touch screen, convert the touch action into a touch signal and send it to the processor 270, and can receive and execute the command sent by the processor 270; the touch signal at least includes the contact point coordinate information. The touch screen 231 can provide an input interface and an output interface between the terminal and the user. In addition, the touch screen can be implemented using multiple types such as resistive, capacitive, infrared, and surface acoustic waves. In addition to the touch screen 231, the input unit 230 can also include other input devices. Specifically, other input devices 232 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, a joystick, and the like.

[0060] The display unit 240 may be used to display information input by a user or information provided to a user, various menus of the terminal, an interactive interface, file display, and / or playback of any multimedia file.

[0061] The memory 220 can be used to store instructions and data. The memory 220 can mainly include an instruction storage area and a data storage area. The data storage area can store various data, such as multimedia files, texts, etc. The instruction storage area can store software units such as operating systems, applications, and instructions required for at least one function, or their subsets and extensions. It can also include a non-volatile random access memory; it provides the processor 270 with hardware, software, and data resources including management of computing and processing equipment, and supports control software and applications. It is also used for the storage of multimedia files, and the storage of running programs and applications.

[0062] The processor 270 is the control center of the terminal. It uses various interfaces and lines to connect various parts of the entire terminal. By running or executing instructions stored in the memory 220 and calling data stored in the memory 220, it executes various functions of the terminal and processes data, thereby controlling the terminal as a whole. Optionally, the processor 270 may include one or more processing units; preferably, the processor 270 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface and application program, and the modem processor mainly processes wireless communication. It is understandable that the above-mentioned modem processor may not be integrated into the processor 270. In some embodiments, the processor and the memory may be implemented on a single chip, and in some embodiments, they may also be implemented separately on separate chips. The processor 270 may also be used to generate corresponding operation control signals, send them to corresponding components of the computing and processing device, read and process data in the software, especially read and process data and programs in the memory 220, so that each functional module therein performs corresponding functions, thereby controlling the corresponding components to act according to the requirements of the instructions.

[0063] Among them, the memory 220 can be used to store software codes related to the scene rendering method, the processor 270 can execute the software codes in the memory 220, and can also schedule other units (such as the above-mentioned input unit 230 and display unit 240) to implement corresponding functions.

[0064] The radio frequency unit 210 (optional) can be used for receiving and sending information or receiving and sending signals during a call, for example, after receiving the downlink information of the base station, it is sent to the processor 270 for processing; in addition, the designed uplink data is sent to the base station. Generally, the radio frequency unit 210 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LowNoiseAmplifier, LNA), a duplexer, etc. In addition, the radio frequency unit 210 can also communicate with network devices and other devices through wireless communication. The wireless communication can use any communication standard or protocol, including but not limited to Global System of Mobile communication (Global System of Mobile communication, GSM), General Packet Radio Service (General PacketRadio Service, GPRS), Code Division Multiple Access (Code Division Multiple Access, CDMA), Wideband Code Division Multiple Access (Wideband Code Division Multiple Access, WCDMA), Long Term Evolution (Long Term Evolution, LTE), email, Short Messaging Service (SMS), etc.

[0065] In the embodiment of the present application, the radio frequency unit 210 can send data to other devices, and can also receive data sent by other devices. It should be understood that the radio frequency unit 210 is optional, and can be replaced by other communication interfaces, such as a network port.

[0066] The terminal also includes a power supply 290 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 270 through a power management system, so that the power management system can manage charging, discharging, power consumption and other functions.

[0067] The terminal also includes an external interface 280, which may be a standard Micro USB interface or a multi-pin connector, and may be used to connect the terminal to communicate with other devices, or to connect a charger to charge the terminal.

[0068] Although not shown, the terminal may further include a flashlight, a wireless fidelity (WiFi) module, a Bluetooth module, sensors with different functions, etc., which will not be described in detail here.

[0069] Next, the product form of the above server is described.

[0070] Figure 3 A structural diagram of the above server is provided, such as Figure 3 As shown, the server may include a bus 301, a processor 302, a communication interface 303, and a memory 304. The processor 302, the memory 304, and the communication interface 303 communicate with each other via the bus 301.

[0071] The bus 301 may be a peripheral component interconnect (PCI) bus or an extended industry standard architecture (EISA) bus. The bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 3 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0072] The processor 302 may be any one or more of a central processing unit (CPU), a graphics processing unit (GPU), a microprocessor (MP), or a digital signal processor (DSP).

[0073] The memory 304 may include a volatile memory, such as a random access memory (RAM). The memory 304 may also include a non-volatile memory, such as a read-only memory (ROM), a flash memory, a hard disk drive (HDD), or a solid state drive (SSD).

[0074] The memory 304 may be used to store software codes related to the scene rendering method. The processor 302 may call the software codes stored in the memory 304 or schedule other units to implement corresponding functions.

[0075] The processors in the above-mentioned terminals and servers (for example, processor 270 and processor 302) may be hardware circuits (such as application specific integrated circuits (ASIC), field-programmable gate arrays (FPGA), general-purpose processors, digital signal processors (DSP), microprocessors or microcontrollers, etc.), or combinations of these hardware circuits. For example, the processor may be a hardware system with an instruction execution function, such as a CPU, DSP, etc., or a hardware system without an instruction execution function, such as an ASIC, FPGA, etc., or a combination of the above-mentioned hardware systems without an instruction execution function and hardware systems with an instruction execution function.

[0076] A current mainstream scene rendering scheme is to pre-train a neural network model that can predict color information and transparency based on the three-dimensional space points and observation perspective of the target scene. During actual rendering, each three-dimensional space point in the three-dimensional space point set of the target scene and the observation perspective are input into the pre-trained neural network model for inference to obtain the color information and transparency corresponding to each three-dimensional space point in the three-dimensional space point set of the target scene, and then render the image of the target scene according to the color information and transparency corresponding to each three-dimensional space point in the three-dimensional space point set of the target scene.

[0077] The inventors of this case studied the above-mentioned scene rendering scheme based on the neural network model and found that although the above-mentioned scheme can realize scene rendering, since the above-mentioned scene rendering scheme based on the neural network model needs to use the neural network model to perform inference on each three-dimensional spatial point of the target scene, and the amount of computation required to use the neural network model to perform inference on each three-dimensional spatial point of the target scene is large, therefore, the rendering efficiency of the above-mentioned scene rendering scheme based on the neural network model is low.

[0078] In view of the low rendering efficiency of the above-mentioned scene rendering scheme, the inventor of this case attempted to propose a scene rendering scheme with higher rendering efficiency and conducted research for this purpose. The initial idea was: pre-use the three-dimensional Gaussian splash method to represent the target scene as multiple Gaussian ellipsoids, each Gaussian ellipsoid having attribute information, and store the attribute information of the multiple Gaussian ellipsoids representing the target scene into the memory. When rendering is required, render the image of the target scene according to the stored attribute information of the multiple Gaussian ellipsoids representing the target scene.

[0079] The inventor of this case studied the above-mentioned three-dimensional Gaussian-based scene rendering scheme and found that compared with the above-mentioned neural network model-based scene rendering scheme, the above-mentioned three-dimensional Gaussian-based scene rendering scheme can effectively improve the rendering efficiency, but it has a defect, that is, when rendering a very large scene, since the number of Gaussian ellipsoids representing the scene is huge, and the attribute information of each Gaussian ellipsoid is 59-dimensional information, therefore, storing the attribute information of a large number of Gaussian ellipsoids requires a huge storage space. It can be seen that the above-mentioned three-dimensional Gaussian-based rendering scheme has a huge memory overhead, and the huge memory overhead may cause the rendering to fail.

[0080] In view of the defects of the above-mentioned 3D Gaussian rendering solution, the inventor of this case continued to study and finally proposed a scene rendering solution that can take into account both rendering efficiency and memory overhead. Next, the scene rendering method provided by this application is introduced through the following embodiments.

[0081] See also Figure 4 , shows a schematic diagram of the process of a scene rendering method provided by an embodiment of the present application, the method may include:

[0082] Step S401: Acquire a three-dimensional space point set of a target scene.

[0083] The three-dimensional space point set of the target scene includes multiple three-dimensional space points of the target scene.

[0084] The three-dimensional space points in the three-dimensional space point set of the target scene can be acquired based on any three-dimensional space point acquisition method, such as an acquisition method based on a binocular camera, an acquisition method based on a depth camera, an acquisition method based on a laser scanner, and the like.

[0085] Step S402: Based on the pre-trained Gaussian prediction model, predict whether there is a Gaussian ellipsoid at each three-dimensional space point in the three-dimensional space point set of the target scene, so as to obtain a target point set consisting of three-dimensional space points where a Gaussian ellipsoid exists.

[0086] The Gaussian prediction model is trained using a number of labeled first training samples, where any first training sample is a three-dimensional space point in the target scene, and the label of any first training sample is used to indicate whether a Gaussian ellipsoid exists at the training sample (three-dimensional space point).

[0087] The present application pre-uses the above-mentioned labeled first training sample to train a Gaussian prediction model that can predict whether a Gaussian ellipsoid exists at a three-dimensional space point of the target scene, and then uses the trained Gaussian prediction model to predict whether a Gaussian ellipsoid exists at each three-dimensional space point in the three-dimensional space point set of the target scene. It should be noted that predicting whether a Gaussian ellipsoid exists at a three-dimensional space point is essentially a simple binary classification task, and therefore, only a smaller model is needed to complete the task, that is, the Gaussian prediction model in this embodiment can use a smaller neural network.

[0088] Specifically, each three-dimensional space point (i.e., three-dimensional coordinate) in the three-dimensional space point set of the target scene is input into a pre-trained Gaussian prediction model. The Gaussian prediction model predicts whether there is a Gaussian ellipsoid at each input three-dimensional space point and outputs a Gaussian prediction result.

[0089] After obtaining the Gaussian prediction results corresponding to each 3D space point in the 3D space point set of the target scene, the 3D space points in the 3D space point set of the target scene where the Gaussian ellipsoid exists can be obtained, and the target point set is composed of the 3D space points in which the Gaussian ellipsoid exists.

[0090] Step S403: Based on the pre-trained Gaussian attribute prediction model, predict the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set.

[0091] Among them, the Gaussian attribute prediction model is trained using a number of labeled second training samples, any second training sample is a three-dimensional space point where a Gaussian ellipsoid exists in the target scene, and the label of any second training sample is the attribute information of the Gaussian ellipsoid existing at the second training sample (the three-dimensional space point where the Gaussian ellipsoid exists).

[0092] In addition to pre-training with a first labeled training sample to obtain a Gaussian prediction model capable of predicting whether a Gaussian ellipsoid exists at a three-dimensional space point of a target scene, the present application also pre-trains with a second labeled training sample to obtain a Gaussian attribute prediction model capable of predicting attribute information of a Gaussian ellipsoid at a three-dimensional space point where a Gaussian ellipsoid exists in the target scene. Furthermore, the pre-trained Gaussian attribute prediction model is used to predict the attribute information of the Gaussian ellipsoid at each three-dimensional space point in the target point set.

[0093] Specifically, each three-dimensional space point (ie, three-dimensional coordinate) in the target point set is input into a pre-trained Gaussian attribute prediction model, and the Gaussian attribute prediction model predicts the attribute information of the Gaussian ellipsoid existing at each input three-dimensional space point, and outputs the Gaussian attribute prediction result.

[0094] Among them, the attribute information of the Gaussian ellipsoid includes 59 dimensions of information, namely: the geometric center xyz of the Gaussian ellipsoid (a total of 3 dimensions), the spherical harmonic function SH expressing the anisotropic color (a total of 48 dimensions), the opacity of the Gaussian ellipsoid opacity (1 dimension), the Gaussian ellipsoid scaling vector scale (3 dimensions), and the Gaussian ellipsoid rotation quaternion rot (4 dimensions).

[0095] It should be noted that the distribution of three-dimensional Gaussians in the entire space is relatively sparse, that is, the number of three-dimensional space points in the target point set is much smaller than the number of three-dimensional space points in the three-dimensional space point set of the target scene. This means that only a small number of three-dimensional space points need to be inferred and calculated based on the Gaussian attribute prediction model.

[0096] Step S404: Rendering is performed according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain a rendering image of the target scene.

[0097] In one possible implementation, rendering is performed according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional spatial point in the target point set, and the process of obtaining a rendered image of the target scene may include: rendering an image of the target scene at a specified perspective using a volume rendering method based on three-dimensional Gaussian splattering according to the specified perspective and the attribute information of the Gaussian ellipsoid existing at each three-dimensional spatial point in the target point set.

[0098] In practical applications, it may be necessary to render an image of a certain perspective of the target scene. After obtaining the attribute information of the Gaussian ellipsoid at the three-dimensional space point where the Gaussian ellipsoid exists in the three-dimensional space point set of the target scene based on the Gaussian attribute prediction model, the image of the target scene at any perspective can be rendered according to the attribute information of the Gaussian ellipsoid at the three-dimensional space point where the Gaussian ellipsoid exists in the three-dimensional space point set of the target scene.

[0099] The scene rendering method provided in the embodiment of the present application first obtains the three-dimensional space point set of the target scene, and then predicts whether there is a Gaussian ellipsoid at each three-dimensional space point in the three-dimensional space point set of the target scene based on a pre-trained Gaussian prediction model, so as to obtain a target point set composed of three-dimensional space points where there are Gaussian ellipsoids, and then predicts the attribute information of the Gaussian ellipsoids at each three-dimensional space point in the target point set based on a pre-trained Gaussian attribute prediction model, and finally renders according to the attribute information of the Gaussian ellipsoids at each three-dimensional space point in the target point set, so as to obtain a rendering image of the target scene. In the scene rendering method provided in the embodiment of the present application, the amount of computation for predicting whether there is a Gaussian ellipsoid at a three-dimensional space point based on the Gaussian prediction model is relatively small. Although the amount of computation for predicting the attribute information of the Gaussian ellipsoid at the three-dimensional space point where the Gaussian ellipsoid exists based on the Gaussian attribute prediction model is slightly larger, since the distribution of three-dimensional Gaussians in the entire space is relatively sparse, that is, the number of three-dimensional space points where there are Gaussian ellipsoids in the target scene is very small, it is only necessary to predict the attribute information of the Gaussian ellipsoid for fewer three-dimensional space points based on the Gaussian attribute prediction model. It can be seen that the overall computational amount of the scene rendering method provided in the embodiment of the present application is not large, and furthermore, the scene rendering method provided in the embodiment of the present application has a high rendering efficiency.

[0100] In addition, the scene rendering method provided in the embodiment of the present application only needs to store the Gaussian prediction model and the Gaussian attribute prediction model in the memory, without storing the attribute information of the Gaussian ellipsoid representing the target scene (for ultra-large scenes, the storage space required to store the Gaussian prediction model and the Gaussian attribute prediction model is much smaller than the storage space required to store the attribute information of a huge number of Gaussian ellipsoids). Since predicting whether there is a Gaussian ellipsoid at a point in three-dimensional space and predicting the attribute information of the Gaussian ellipsoid at a point in three-dimensional space are not very complicated tasks, the scale of the Gaussian prediction model and the Gaussian attribute prediction model are not large. Therefore, the memory overhead of the scene rendering method provided in the embodiment of the present application is not large.

[0101] It can be seen from the above embodiments that the present application realizes scene rendering based on a pre-trained Gaussian prediction model and a pre-trained Gaussian attribute prediction model. In another embodiment of the present application, the training process of the Gaussian prediction model and the Gaussian attribute prediction model is introduced.

[0102] The above embodiment mentioned that the Gaussian prediction model is trained using a number of labeled first training samples (any first training sample is a three-dimensional space point in the target scene, and the label of any first training sample is used to indicate whether a Gaussian ellipsoid exists at the training sample), and the Gaussian attribute prediction model is trained using a number of labeled second training samples (any second training sample is a three-dimensional space point in the target scene where a Gaussian ellipsoid exists, and the label of any second training sample is the attribute information of the Gaussian ellipsoid existing at the second training sample). Before introducing the training process of the Gaussian prediction model and the Gaussian attribute prediction model, the process of obtaining a number of labeled first training samples and a number of labeled second training samples is first introduced.

[0103] See also Figure 5 , which shows a schematic diagram of a process of obtaining a plurality of labeled first training samples and a plurality of labeled second training samples, may include:

[0104] Step S501: using a three-dimensional Gaussian splashing method, the target scene is represented as a plurality of Gaussian ellipsoids, each Gaussian ellipsoid having attribute information.

[0105] See also Figure 6 , Figure 6 The second figure shows the Figure 6 The scene in the first figure is represented as a schematic diagram of multiple Gaussian ellipsoids.

[0106] In one possible implementation, the process of using a three-dimensional Gaussian splashing method to represent a target scene as multiple Gaussian ellipsoids may include: acquiring two-dimensional images of the target scene from multiple perspectives, and performing three-dimensional modeling of the target scene based on the two-dimensional images from multiple perspectives using a three-dimensional Gaussian splashing method to obtain multiple Gaussian ellipsoids representing the target scene.

[0107] It should be noted that, in order to better perform three-dimensional modeling on the target scene, when acquiring a two-dimensional image of the target scene, the two-dimensional image should be acquired from as many viewing angles as possible.

[0108] Step S502: constructing a plurality of labeled first training samples and a plurality of labeled second training samples according to the three-dimensional space point set of the target scene and the attribute information of a plurality of Gaussian ellipsoids representing the target scene.

[0109] Specifically, the process of constructing a plurality of labeled first training samples and a plurality of labeled second training samples according to the three-dimensional space point set of the target scene and the attribute information of a plurality of Gaussian ellipsoids representing the target scene may include:

[0110] Step a1: for each 3D space point in the 3D space point set of the target scene, determine whether there is a Gaussian ellipsoid at the 3D space point according to the attribute information of multiple Gaussian ellipsoids representing the target scene, and obtain the Gaussian determination result of the 3D space point.

[0111] Whether there is a Gaussian ellipsoid at the three-dimensional space point can be determined based on the geometric center and the scaling vector in the attribute information of multiple Gaussian ellipsoids representing the target scene, so as to obtain the Gaussian determination result of the three-dimensional space point.

[0112] Step a2: taking the three-dimensional space point as a first training sample, and taking the Gaussian discriminant result of the three-dimensional space point as a label of the first training sample, to obtain a first training sample with a label.

[0113] Step a3: If a Gaussian ellipsoid exists at the three-dimensional space point, the three-dimensional space point is used as a second training sample, and the attribute information of the Gaussian ellipsoid existing at the three-dimensional space point is used as a label of the second training sample to obtain a labeled second training sample.

[0114] When it is determined that a Gaussian ellipsoid exists at the three-dimensional space point, the attribute information of the Gaussian ellipsoid existing at the three-dimensional space point can be obtained, and then a second labeled training sample can be constructed.

[0115] Through the above process, a number of labeled first training samples and a number of labeled second training samples can be obtained.

[0116] After obtaining a plurality of labeled first training samples and a plurality of labeled second training samples, the plurality of labeled first training samples and the plurality of labeled second training samples may be used to train a Gaussian prediction model and a Gaussian attribute prediction model.

[0117] The training process of the Gaussian prediction model may include:

[0118] Step b1: predict whether there is a Gaussian ellipsoid at the first training sample based on the Gaussian prediction model, and obtain a Gaussian prediction result corresponding to the first training sample.

[0119] The first training sample (a three-dimensional space point of the target scene) is input into the Gaussian prediction model. The Gaussian prediction model predicts the probability of the existence of a Gaussian ellipsoid at the first training sample (a three-dimensional space point of the target scene) and the probability of the absence of a Gaussian ellipsoid, and obtains the Gaussian prediction result corresponding to the first training sample.

[0120] Step b2: Determine the prediction loss of the Gaussian prediction model according to the Gaussian prediction result corresponding to the first training sample and the label of the first training sample.

[0121] The prediction loss of the Gaussian prediction model may adopt the cross entropy loss. The calculation method of the cross entropy loss is a prior art and will not be described in detail in this embodiment.

[0122] Step b3: update the parameters of the Gaussian prediction model according to the prediction loss of the Gaussian prediction model.

[0123] The training goal of the Gaussian prediction model is to make the Gaussian prediction result corresponding to the first training sample approach the label of the first training sample.

[0124] The Gaussian prediction model is trained iteratively multiple times using multiple labeled first training samples according to the process of step b1 to step b3 until the training end condition is met (for example, the Gaussian prediction model converges, or the set number of training iterations is reached).

[0125] The training process of the Gaussian attribute prediction model may include:

[0126] Step c1: predicting the attribute information of the Gaussian ellipsoid existing at the second training sample based on the Gaussian attribute prediction model, and obtaining the Gaussian attribute prediction result corresponding to the second training sample.

[0127] The second training sample (a three-dimensional space point of the target scene where a Gaussian ellipsoid exists) is input into the Gaussian attribute prediction model. The Gaussian attribute prediction model predicts the attribute information of the Gaussian ellipsoid existing at the second training sample (a three-dimensional space point of the target scene where a Gaussian ellipsoid exists), and obtains the Gaussian attribute prediction result corresponding to the second training sample.

[0128] Step c2: determining the prediction loss of the Gaussian attribute prediction model according to the Gaussian attribute prediction result corresponding to the second training sample and the label of the second training sample.

[0129] The difference between the Gaussian attribute prediction result corresponding to the second training sample and the label of the second training sample can be calculated as the prediction loss of the Gaussian attribute prediction model.

[0130] Step c3: update the parameters of the Gaussian attribute prediction model according to the prediction loss of the Gaussian attribute prediction model.

[0131] The training goal of the Gaussian attribute prediction model is to minimize the difference between the Gaussian attribute prediction result corresponding to the second training sample and the label of the second training sample (the real Gaussian attribute information), so that the Gaussian attribute prediction result corresponding to the second training sample is close to the label of the second training sample (the real Gaussian attribute information).

[0132] The Gaussian attribute prediction model is trained multiple times by using multiple labeled second training samples according to the process of step c1 to step c3 until the training end condition is met (for example, the Gaussian attribute prediction model converges, or the set number of training iterations is reached).

[0133] After the Gaussian prediction model and the Gaussian attribute prediction model are trained, it is possible to predict whether a Gaussian ellipsoid exists at each three-dimensional space point in the three-dimensional space point set based on the pre-trained Gaussian prediction model, so as to obtain a target point set composed of three-dimensional space points where a Gaussian ellipsoid exists, and then predict the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set based on the pre-trained Gaussian attribute prediction model. Finally, rendering is performed according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain a rendering of the target scene.

[0134] The scene rendering method provided in the embodiment of the present application has a relatively small Gaussian prediction model for predicting whether there is a Gaussian ellipsoid at a three-dimensional space point of the target scene, and its computational complexity is very small and can be ignored. Although the Gaussian attribute prediction model for predicting the attribute information of the Gaussian ellipsoid at the three-dimensional space point has a larger computational complexity, since there is no need to perform calculations in the entire three-dimensional space of the target scene, but only where the Gaussian ellipsoid exists, the overall computational complexity is not large. In terms of memory overhead, it is only necessary to store the Gaussian prediction model and the Gaussian attribute prediction model, and there is no need to store the attribute information of the Gaussian ellipsoid representing the target scene. In summary, the scene rendering method provided in the embodiment of the present application has a high rendering efficiency and a low memory overhead. That is, the scene rendering method provided in the embodiment of the present application can take into account both rendering efficiency and memory overhead, and has a wider range of application (suitable for rendering of small scenes as well as rendering of large scenes).

[0135] The scene rendering method provided in the embodiment of the present application is introduced above, and the device corresponding to the above scene rendering method will be introduced below.

[0136] See also Figure 7 , Figure 7 A schematic diagram of the structure of a scene rendering device provided in an embodiment of the present application, the scene rendering device may include: a scene data acquisition module 701, a first prediction module 702, a second prediction module 703 and a scene rendering module 704.

[0137] The scene data acquisition module 701 is used to acquire a three-dimensional space point set of a target scene.

[0138] The first prediction module 702 is used to predict whether there is a Gaussian ellipsoid at each three-dimensional space point in the three-dimensional space point set based on a pre-trained Gaussian prediction model, so as to obtain a target point set consisting of three-dimensional space points where a Gaussian ellipsoid exists.

[0139] The Gaussian prediction model is trained using a number of labeled first training samples, any first training sample is a three-dimensional space point in the target scene, and the label of any first training sample is used to indicate whether a Gaussian ellipsoid exists at the training sample.

[0140] The second prediction module 703 is used to predict the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set based on the pre-trained Gaussian attribute prediction model.

[0141] The Gaussian attribute prediction model is trained using a number of labeled second training samples, any second training sample is a three-dimensional space point in the target scene where a Gaussian ellipsoid exists, and the label of any second training sample is the attribute information of the Gaussian ellipsoid existing at the second training sample.

[0142] The scene rendering module 704 is used to render according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain a rendering image of the target scene.

[0143] In a possible implementation, the scene rendering device provided in the embodiment of the present application may further include: a training data acquisition module.

[0144] The training data acquisition module may include a scene modeling module and a training data construction module.

[0145] The scene modeling module is used to adopt a three-dimensional Gaussian splashing method to represent the target scene as a plurality of Gaussian ellipsoids, each of which has attribute information.

[0146] The training data construction module is used to construct the plurality of labeled first training samples and the plurality of labeled second training samples according to the three-dimensional space point set of the target scene and the attribute information of a plurality of Gaussian ellipsoids representing the target scene.

[0147] In a possible implementation, when the training data construction module constructs the plurality of labeled first training samples and the plurality of labeled second training samples according to the three-dimensional space point set of the target scene and the attribute information of the plurality of Gaussian ellipsoids representing the target scene, it is specifically used to:

[0148] For each 3D space point in the 3D space point set of the target scene:

[0149] According to the attribute information of the multiple Gaussian ellipsoids representing the target scene, it is judged whether there is a Gaussian ellipsoid at the three-dimensional space point, and a Gaussian judgment result of the three-dimensional space point is obtained;

[0150] The three-dimensional space point is used as a first training sample, and the Gaussian discriminant result of the three-dimensional space point is used as a label of the first training sample to obtain a first training sample with a label;

[0151] If a Gaussian ellipsoid exists at the three-dimensional space point, the three-dimensional space point is used as a second training sample, and the attribute information of the Gaussian ellipsoid existing at the three-dimensional space point is used as a label of the second training sample to obtain a labeled second training sample.

[0152] In a possible implementation, when the scene modeling module uses a three-dimensional Gaussian splashing method to represent the target scene as a plurality of Gaussian ellipsoids, the scene modeling module is specifically used to:

[0153] Acquire two-dimensional images of multiple viewing angles for the target scene;

[0154] According to the two-dimensional images of the multiple viewing angles, a three-dimensional Gaussian splashing method is used to perform three-dimensional modeling on the target scene to obtain multiple Gaussian ellipsoids representing the target scene.

[0155] In a possible implementation, when the scene rendering module 704 performs rendering according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain the rendering image of the target scene, it is specifically used to:

[0156] According to the specified viewing angle and the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set, a volume rendering method based on three-dimensional Gaussian splashing is adopted to render an image of the specified viewing angle of the target scene.

[0157] In a possible implementation, the scene rendering device provided in the embodiment of the present application may also include: a Gaussian prediction model training module.

[0158] Gaussian prediction model training module, used for:

[0159] Predicting whether there is a Gaussian ellipsoid at the first training sample based on the Gaussian prediction model, and obtaining a Gaussian prediction result corresponding to the first training sample;

[0160] Determine the prediction loss of the Gaussian prediction model according to the Gaussian prediction result corresponding to the first training sample and the label of the first training sample;

[0161] According to the prediction loss of the Gaussian prediction model, the parameters of the Gaussian prediction model are updated.

[0162] In a possible implementation, the scene rendering device provided in the embodiment of the present application may also include: a Gaussian attribute prediction model training module.

[0163] Gaussian attribute prediction model training module, used for:

[0164] Predicting attribute information of the Gaussian ellipsoid existing at the second training sample based on the Gaussian attribute prediction model, and obtaining a Gaussian attribute prediction result corresponding to the second training sample;

[0165] Determine the prediction loss of the Gaussian attribute prediction model according to the Gaussian attribute prediction result corresponding to the second training sample and the label of the second training sample;

[0166] According to the prediction loss of the Gaussian attribute prediction model, the parameters of the Gaussian attribute prediction model are updated.

[0167] The scene rendering device provided in the embodiment of the present application has a relatively small Gaussian prediction model for predicting whether there is a Gaussian ellipsoid at a three-dimensional space point of the target scene, and its computational complexity is very small and can be ignored. Although the Gaussian attribute prediction model for predicting the attribute information of the Gaussian ellipsoid at the three-dimensional space point has a larger computational complexity, since there is no need to perform calculations in the entire three-dimensional space of the target scene, but only where the Gaussian ellipsoid exists, the overall computational complexity is not large. In terms of memory overhead, it is only necessary to store the Gaussian prediction model and the Gaussian attribute prediction model, and there is no need to store the attribute information of the Gaussian ellipsoid representing the target scene. In summary, the scene rendering device provided in the embodiment of the present application has a high rendering efficiency and a low memory overhead. That is, the scene rendering device provided in the embodiment of the present application can take into account both rendering efficiency and memory overhead, and has a wider range of application (suitable for rendering of small scenes as well as rendering of large scenes).

[0168] An embodiment of the present application also provides an electronic device, which may include: at least one processor, at least one communication interface, at least one memory and at least one communication bus.

[0169] In the embodiment of the present application, the number of the processor, the communication interface, the memory, and the communication bus is at least one, and the processor, the communication interface, and the memory communicate with each other through the communication bus;

[0170] The processor may be a central processing unit (CPU), or an application-specific integrated circuit (ASIC), or one or more integrated circuits configured to implement the embodiments of the present application;

[0171] The memory may include a high-speed RAM memory, and may also include a non-volatile memory, such as at least one disk memory;

[0172] The memory stores a program, and the processor can call the program stored in the memory, and the program is used to implement the steps of the scene rendering method provided in the above embodiment.

[0173] An embodiment of the present application also provides a computer storage medium, which carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the scene rendering method provided in the above embodiment.

[0174] An embodiment of the present application also provides a computer program product, including computer-readable instructions. When the computer-readable instructions are executed on an electronic device, the electronic device implements the steps of the scene rendering method provided in the above embodiment.

[0175] It should also be noted that the device embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed over multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. In addition, in the drawings of the device embodiments provided by the present application, the connection relationship between the modules indicates that there is a communication connection between them, which may be specifically implemented as one or more communication buses or signal lines.

[0176] Through the description of the above implementation mode, the technicians in the field can clearly understand that the present application can be implemented by means of software plus necessary general hardware, and of course, it can also be implemented by special hardware including special integrated circuits, special CPUs, special memories, special components, etc. In general, all functions completed by computer programs can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be various, such as analog circuits, digital circuits or special circuits. However, for the present application, software program implementation is a better implementation mode in more cases. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art can be embodied in the form of a software product, which is stored in a readable storage medium, such as a computer floppy disk, a U disk, a mobile hard disk, a ROM, a RAM, a disk or an optical disk, etc., including a number of instructions to enable a computer device (which can be a personal computer, a training device, or a network device, etc.) to execute the methods described in each embodiment of the present application.

[0177] In the above embodiments, all or part of the embodiments may be implemented by software, hardware, firmware or any combination thereof. When implemented by software, all or part of the embodiments may be implemented in the form of a computer program product.

[0178] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, a computer, a training device, or a data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, training device, or data center. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device, a data center, etc. that includes one or more available media integrations. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.

Claims

1. A scene rendering method, characterized in that: include: Obtain the three-dimensional space point set of the target scene; Based on a pre-trained Gaussian prediction model, predict whether a Gaussian ellipsoid exists at each three-dimensional space point in the three-dimensional space point set, so as to obtain a target point set consisting of three-dimensional space points where a Gaussian ellipsoid exists, wherein the Gaussian prediction model is trained using a number of labeled first training samples, any first training sample is a three-dimensional space point in the target scene, and the label of any first training sample is used to indicate whether a Gaussian ellipsoid exists at the training sample; Based on the pre-trained Gaussian attribute prediction model, predict the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set, wherein the Gaussian attribute prediction model is trained using a number of labeled second training samples, any second training sample is a three-dimensional space point in the target scene where a Gaussian ellipsoid exists, and the label of any second training sample is the attribute information of the Gaussian ellipsoid existing at the second training sample; Rendering is performed according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain a rendering image of the target scene.

2. The scene rendering method according to claim 1, characterized in that: The process of acquiring the plurality of labeled first training samples and the plurality of labeled second training samples includes: Using a three-dimensional Gaussian splashing method, the target scene is represented as a plurality of Gaussian ellipsoids, each Gaussian ellipsoid having attribute information; The plurality of labeled first training samples and the plurality of labeled second training samples are constructed according to the three-dimensional space point set of the target scene and the attribute information of a plurality of Gaussian ellipsoids representing the target scene.

3. The scene rendering method according to claim 2, characterized in that: The step of constructing the plurality of labeled first training samples and the plurality of labeled second training samples according to the three-dimensional space point set of the target scene and the attribute information of the plurality of Gaussian ellipsoids representing the target scene comprises: For each 3D space point in the 3D space point set of the target scene: According to the attribute information of the multiple Gaussian ellipsoids representing the target scene, it is judged whether there is a Gaussian ellipsoid at the three-dimensional space point, and a Gaussian judgment result of the three-dimensional space point is obtained; The three-dimensional space point is used as a first training sample, and the Gaussian discriminant result of the three-dimensional space point is used as a label of the first training sample to obtain a first training sample with a label; If a Gaussian ellipsoid exists at the three-dimensional space point, the three-dimensional space point is used as a second training sample, and the attribute information of the Gaussian ellipsoid existing at the three-dimensional space point is used as a label of the second training sample to obtain a labeled second training sample.

4. The scene rendering method according to claim 2, characterized in that: The target scene is represented as a plurality of Gaussian ellipsoids by using a three-dimensional Gaussian splashing method, including: Acquire two-dimensional images of multiple viewing angles for the target scene; According to the two-dimensional images of the multiple viewing angles, a three-dimensional Gaussian splashing method is used to perform three-dimensional modeling on the target scene to obtain multiple Gaussian ellipsoids representing the target scene.

5. The scene rendering method according to claim 1, characterized in that: The rendering is performed according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain a rendering of the target scene, including: According to the specified viewing angle and the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set, a volume rendering method based on three-dimensional Gaussian splashing is adopted to render an image of the specified viewing angle of the target scene.

6. The scene rendering method according to claim 1, characterized in that: The training process of the Gaussian prediction model includes: Predicting whether there is a Gaussian ellipsoid at the first training sample based on the Gaussian prediction model, and obtaining a Gaussian prediction result corresponding to the first training sample; Determine the prediction loss of the Gaussian prediction model according to the Gaussian prediction result corresponding to the first training sample and the label of the first training sample; According to the prediction loss of the Gaussian prediction model, the parameters of the Gaussian prediction model are updated.

7. The scene rendering method according to claim 1, characterized in that: The training process of the Gaussian attribute prediction model includes: Predicting attribute information of the Gaussian ellipsoid existing at the second training sample based on the Gaussian attribute prediction model, and obtaining a Gaussian attribute prediction result corresponding to the second training sample; Determine the prediction loss of the Gaussian attribute prediction model according to the Gaussian attribute prediction result corresponding to the second training sample and the label of the second training sample; According to the prediction loss of the Gaussian attribute prediction model, the parameters of the Gaussian attribute prediction model are updated.

8. A scene rendering device, characterized in that: include: A scene data acquisition module, a first prediction module, a second prediction module and a scene rendering module; The scene data acquisition module is used to acquire a three-dimensional space point set of the target scene; The first prediction module is used to predict whether there is a Gaussian ellipsoid at each three-dimensional space point in the three-dimensional space point set based on a pre-trained Gaussian prediction model, so as to obtain a target point set consisting of three-dimensional space points where a Gaussian ellipsoid exists, wherein the Gaussian prediction model is trained using a number of labeled first training samples, any first training sample is a three-dimensional space point in the target scene, and the label of any first training sample is used to indicate whether there is a Gaussian ellipsoid at the training sample; The second prediction module is used to predict the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set based on the pre-trained Gaussian attribute prediction model, wherein the Gaussian attribute prediction model is trained using a number of labeled second training samples, any second training sample is a three-dimensional space point in the target scene where a Gaussian ellipsoid exists, and the label of any second training sample is the attribute information of the Gaussian ellipsoid existing at the second training sample; The scene rendering module is used to render according to the attribute information of the Gaussian ellipsoid existing at each three-dimensional space point in the target point set to obtain a rendering image of the target scene.

9. An electronic device, characterized in that: The method comprises at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program so that the electronic device can implement the steps of the scene rendering method as described in any one of claims 1 to 7.

10. A computer storage medium, characterized in that: The storage medium carries one or more computer programs, and when the one or more computer programs are executed by an electronic device, the electronic device can implement the steps of the scene rendering method as described in any one of claims 1 to 7.

Images