Three-dimensional scene reconstruction method, device, equipment, storage medium and program product
By combining blurred image sequences and event sequences, constructing a loss function and optimizing Gaussian ellipsoid parameters, the problem of poor 3D scene reconstruction in high-speed motion scenes is solved, and more efficient 3D scene reconstruction is achieved.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- MOTOVIS TECH SHANGHAI CO LTD
- Filing Date
- 2025-12-05
- Publication Date
- 2026-06-02
AI Technical Summary
In high-speed motion scenes, images captured by ordinary cameras have motion blur, resulting in poor 3D scene reconstruction. Existing technologies ignore the geometric characteristics of event cameras and have failed to effectively solve this problem.
By combining blurred image sequences acquired by a traditional camera and event sequences acquired by an event camera, a 3D Gaussian ellipsoid is initialized, a loss function is constructed, and the minimum value of the loss function is used as the optimization objective. The parameter information of the Gaussian ellipsoid is adjusted, and geometric constraints are added to improve the reconstruction effect.
It improves the reconstruction effect of 3D scenes by guiding and adjusting the Gaussian ellipsoid, enhancing the geometric constraints and visual feature representation of 3D scenes, and improving the accuracy and efficiency of reconstruction.
Smart Images
Figure CN121280668B_ABST
Abstract
Description
Technical Field
[0001] This invention relates to the field of three-dimensional reconstruction technology, specifically to three-dimensional scene reconstruction methods, devices, equipment, storage media, and program products. Background Technology
[0002] Image-based 3D reconstruction technology has wide applications in computer vision, such as virtual reality, autonomous driving, and embody intelligence. However, the reconstruction effect of 3D scenes depends on the quality of the acquired images, but in high-speed moving scenes, images captured by ordinary cameras have a certain degree of motion blur, resulting in poor 3D scene reconstruction results. Summary of the Invention
[0003] This invention provides a method, apparatus, device, storage medium, and program product for three-dimensional scene reconstruction, in order to solve the problem of poor reconstruction effect of three-dimensional scenes.
[0004] In a first aspect, the present invention provides a three-dimensional scene reconstruction method, the method comprising:
[0005] Based on the acquired blurred image sequence and event sequence, the three-dimensional Gaussian ellipsoid is initialized to obtain the initial Gaussian ellipsoid, wherein the event sequence and the blurred image sequence correspond to the same motion scene;
[0006] Based on the geometric constraints of the regions without generated events corresponding to the clear image sequence, the blurred image sequence, and the event sequence obtained by rendering the blurred image sequence using the initial Gaussian ellipsoid, a loss function is constructed.
[0007] The optimization objective is to minimize the loss function. The parameters of the initial Gaussian ellipsoid are adjusted to obtain the target Gaussian ellipsoid.
[0008] The target 3D scene is reconstructed based on the target Gaussian ellipsoid.
[0009] In one optional implementation, a loss function is constructed based on the geometric constraints of the sharp image sequence obtained by rendering the blurred image sequence using the initial Gaussian ellipsoid and the corresponding ungenerated event regions in the event sequence, including:
[0010] Based on the differences between clear image sequences and their corresponding blurry images, a blurry reconstruction loss function is constructed.
[0011] Based on the differences between event sequences and corresponding clear images, an event reconstruction loss function is constructed.
[0012] Construct geometric constraints for the event sequence based on the geometric constraints of the regions where no events have been generated corresponding to the event sequence.
[0013] The loss function is obtained by adding the fuzzy reconstruction loss function, the event reconstruction loss function, and the event sequence geometric constraints.
[0014] In one optional implementation, a blur reconstruction loss function is constructed based on the difference between the clear image sequence and the corresponding blurry image, including:
[0015] By superimposing a sequence of clear images within the exposure time corresponding to the blurred image, a simulated blurred image corresponding to the blurred image is obtained.
[0016] A fuzzy reconstruction loss function is constructed based on the difference between the blurred image and the corresponding simulated blurred image.
[0017] In one optional implementation, a sequence of clear images within the exposure time corresponding to the blurred image is superimposed to obtain a simulated blurred image corresponding to the blurred image, including:
[0018] The average value of a sequence of clear images within the same exposure time as the blurred image is applied to obtain a simulated blurred image corresponding to the blurred image.
[0019] In one alternative implementation, an event reconstruction loss function is constructed based on the difference between the event sequence and the corresponding clear image, including:
[0020] Based on the integral of the event sequence over each preset time period, the first brightness change image corresponding to each preset time period is determined;
[0021] Based on the clear images of the start and end times corresponding to each preset time period, determine the second brightness change image corresponding to each preset time period.
[0022] An event reconstruction loss function is constructed based on the difference between the first brightness change image and the second brightness change image corresponding to each preset time period.
[0023] In one optional implementation, geometric constraints for the event sequence are constructed based on the geometric constraints of the regions where no events have been generated, including:
[0024] Based on the event generation location corresponding to the event sequence, determine the area where no events were generated;
[0025] Geometric constraints for the event sequence are constructed based on the zero-axis scale and orientation of the Gaussian ellipsoid corresponding to the region where no events were generated.
[0026] In a second aspect, the present invention provides a three-dimensional scene reconstruction apparatus, the apparatus comprising:
[0027] The Gaussian ellipsoid initialization module is used to initialize the three-dimensional Gaussian ellipsoid based on the acquired blurred image sequence and event sequence to obtain the initial Gaussian ellipsoid. The event sequence and the blurred image sequence correspond to the same motion scene.
[0028] The loss function construction module is used to construct the loss function based on the geometric constraints of the clear image sequence, the blurred image sequence, and the event sequence corresponding to the ungenerated event region obtained by rendering the blurred image sequence based on the initial Gaussian ellipsoid;
[0029] The Gaussian ellipsoid optimization module is used to adjust the parameter information of the initial Gaussian ellipsoid with the goal of minimizing the loss function, so as to obtain the target Gaussian ellipsoid.
[0030] The 3D scene reconstruction module is used to reconstruct the 3D scene based on the target Gaussian ellipsoid to obtain the target 3D scene.
[0031] Thirdly, the present invention provides an electronic device, comprising: a memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the computer instructions to perform the three-dimensional scene reconstruction method of the first aspect or any corresponding embodiment described above.
[0032] Fourthly, the present invention provides a computer-readable storage medium storing computer instructions for causing a computer to perform the three-dimensional scene reconstruction method of the first aspect or any corresponding embodiment described above.
[0033] Fifthly, the present invention provides a computer program product, including computer instructions for causing a computer to execute the three-dimensional scene reconstruction method of the first aspect or any corresponding embodiment described above.
[0034] The 3D scene reconstruction method provided in this invention initializes a 3D Gaussian ellipsoid based on the acquired blurred image sequence and event sequence to obtain an initial Gaussian ellipsoid. This initial Gaussian ellipsoid is then combined with the blurred image sequence acquired by a traditional camera and the event sequence acquired by an event camera for 3D reconstruction, improving the reconstruction effect of the 3D scene. Simultaneously, based on the geometric constraints of the clear image sequence obtained by rendering the blurred image sequence using the initial Gaussian ellipsoid and the corresponding ungenerated event regions in the event sequence, a loss function is constructed. The minimum value of the loss function is used as the optimization objective to adjust the parameter information of the initial Gaussian ellipsoid, obtaining a target Gaussian ellipsoid. By adding geometric constraints, the Gaussian ellipsoid is guided and adjusted, further improving the 3D scene reconstruction effect. Attached Figure Description
[0035] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.
[0036] Figure 1 This is a schematic diagram of an application scenario according to an embodiment of the present invention;
[0037] Figure 2 This is a schematic diagram of the first process of a three-dimensional scene reconstruction method according to an embodiment of the present invention;
[0038] Figure 3 This is a schematic diagram of a second process for a three-dimensional scene reconstruction method according to an embodiment of the present invention;
[0039] Figure 4 This is a structural block diagram of a three-dimensional scene reconstruction device according to an embodiment of the present invention;
[0040] Figure 5 This is a schematic diagram of the hardware structure of an electronic device according to an embodiment of the present invention. Detailed Implementation
[0041] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.
[0042] It is understood that before using the technical solutions disclosed in the various embodiments of the present invention, users should be informed of the types, scope of use, and usage scenarios of the personal information involved in the present invention and their authorization should be obtained in accordance with relevant laws and regulations through appropriate means.
[0043] The terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of technical features indicated. Thus, a feature defined as "first" or "second" may explicitly or implicitly include one or more of that feature. In the description of this invention, "a plurality of" means two or more, unless otherwise explicitly specified.
[0044] Image-based 3D reconstruction technology has wide applications in computer vision, such as virtual reality, autonomous driving, and embodied intelligence. Currently, among commonly used 3D reconstruction techniques, implicit 3D scene representation based on neural radiation fields can render images from new perspectives quite well. However, this method is time-consuming to render images, and the implicit representation lacks interpretability. 3D Gaussian jetting technology, based on a 3D Gaussian ellipsoid representing 3D space, uses spherical cofunctions to simulate the color of the ellipsoid and generates images from new perspectives using point rendering. Compared to the neural radiation field method, it is more interpretable and renders images faster. However, the reconstruction effect of a 3D scene depends on the quality of the acquired images. In high-speed moving scenes, images captured by ordinary cameras often have motion blur, resulting in poor 3D scene reconstruction. Even when using event cameras to acquire scene data to avoid motion blur, the geometric characteristics of the event camera itself are often overlooked, leading to poor 3D scene reconstruction results.
[0045] To address the aforementioned technical problems, this invention provides a 3D scene reconstruction method. Based on acquired blurred image sequences and event sequences, a 3D Gaussian ellipsoid is initialized to obtain an initial Gaussian ellipsoid. This initial ellipsoid is then combined with blurred image sequences acquired by a traditional camera and event sequences acquired by an event camera for 3D reconstruction, improving the reconstruction effect of the 3D scene. Simultaneously, based on the geometric constraints of the sharp image sequences obtained by rendering the blurred image sequences from the initial Gaussian ellipsoid and the corresponding ungenerated event regions in the event sequences, a loss function is constructed. The minimum value of the loss function is used as the optimization objective to adjust the parameter information of the initial Gaussian ellipsoid, obtaining a target Gaussian ellipsoid. Thus, by adding geometric constraints, the Gaussian ellipsoid is guided and adjusted, further improving the 3D scene reconstruction effect.
[0046] As an optional application scenario of this invention, such as Figure 1 As shown, the system may include at least one conventional camera, at least one event camera, at least one terminal device, and at least one server. Figure 1 The system, as exemplified in the diagram, includes a conventional camera 101, an event camera 102, a terminal device 103, and a server 104. The conventional camera 101, the event camera 102, and the terminal device 103 are connected to the server 104 via a network. The conventional camera 101 and the event camera 102 are used to capture scenes in real time and transmit the captured data to the server 104. The server 104 reconstructs a 3D scene based on the received captured data and transmits the reconstructed 3D scene to the terminal device 103 for display. The terminal device 103 can be an in-vehicle terminal. Correspondingly, the conventional camera 101 and the event camera 102 are installed on the vehicle to capture the scene around the vehicle, thereby realizing fusion perception during vehicle use.
[0047] In one alternative implementation, the terminal device can also be a smartphone, tablet, laptop, PDA, desktop computer, game console, smart TV, smart wearable device, VR (Virtual Reality) device, AR (Augmented Reality) device, etc. Server 104 can be a standalone physical server, a server cluster, a distributed system, or a cloud server providing cloud services. Network 110 can be a wired or wireless network, examples of which include, but are not limited to, the Internet, corporate intranet, local area network, wide area network, mobile communication network, and combinations thereof.
[0048] It should be noted that, Figure 1 This is merely an example of an application scenario and does not limit the scope of protection of this invention.
[0049] According to an embodiment of the present invention, a three-dimensional scene reconstruction method embodiment is provided. It should be noted that the steps shown in the flowchart in the accompanying drawings can be executed in a computer system such as a set of computer-executable instructions. Furthermore, although a logical order is shown in the flowchart, in some cases, the steps shown or described may be executed in a different order than that shown here.
[0050] This embodiment provides a three-dimensional scene reconstruction method, which can be used in the aforementioned server. Figure 2 This is a first flowchart of a three-dimensional scene reconstruction method according to an embodiment of the present invention, such as... Figure 2 As shown, the process includes the following steps:
[0051] Step S201: Based on the acquired blurred image sequence and event sequence, initialize the three-dimensional Gaussian ellipsoid to obtain the initial Gaussian ellipsoid.
[0052] In this embodiment of the invention, the blurred image sequence is acquired by a traditional camera and contains multiple images with motion blur arranged in order of shooting time. The event sequence is acquired by an event camera and is actually a set of discrete events arranged in time. The event sequence and the blurred image sequence correspond to the same motion scene, thus using different data formats to represent the same motion scene.
[0053] In this embodiment of the invention, SFM (Structure from Motion) technology is used to generate an initial three-dimensional point cloud based on feature extraction and matching of blurred images in a blurred image sequence. Based on the initial three-dimensional point cloud, 3D Gaussian jetting technology is used to generate an initial Gaussian ellipsoid.
[0054] Step S202: Based on the geometric constraints of the clear image sequence, the blurred image sequence, and the event sequence corresponding to the ungenerated event regions obtained by rendering the blurred image sequence using the initial Gaussian ellipsoid, a loss function is constructed.
[0055] In this embodiment of the invention, an initial Gaussian ellipsoid is used to render a sequence of blurred images to obtain a sequence of clear images. Using the clear image sequence as a benchmark, corresponding image processing is performed on the blurred image sequence and the event sequence. Based on the differences between the image processing results and the corresponding blurred image sequence and event sequence, and combined with the geometric constraints of the ungenerated event region corresponding to the event sequence, a loss function is constructed. Thus, the loss function simultaneously reflects the differences between the images rendered with a 3D Gaussian ellipsoid and the blurred image sequence and event sequence obtained from actual shooting, as well as the geometric constraints of the event camera.
[0056] Step S203: With the minimum value of the loss function as the optimization objective, the parameter information of the initial Gaussian ellipsoid is adjusted to obtain the target Gaussian ellipsoid.
[0057] In this embodiment of the invention, the parameters of the initial Gaussian ellipsoid are adjusted with the goal of minimizing the loss function. During each parameter adjustment, optimization algorithms such as gradient descent are used to calculate the gradient of the loss function for each parameter of the initial Gaussian ellipsoid, and the parameters are adjusted along the negative gradient direction. Through iterative optimization, the loss function reaches its minimum, satisfying the optimization objective, thus completing the adjustment of the parameters of the initial Gaussian ellipsoid and obtaining the target Gaussian ellipsoid.
[0058] Step S204: Reconstruct the three-dimensional scene based on the target Gaussian ellipsoid to obtain the target three-dimensional scene.
[0059] In this embodiment of the invention, a 3D rendering engine is used to render the target Gaussian ellipsoid, thereby reconstructing the 3D scene and obtaining the target 3D scene. Simultaneously, the target Gaussian ellipsoid can characterize all scene features of the target 3D scene; that is, the position and shape of the target Gaussian ellipsoid correspond to the geometric structure of the target 3D scene, and the color corresponds to the visual features of the target scene. The target 3D scene can also be analyzed based on the target Gaussian ellipsoid, such as obstacle detection.
[0060] This embodiment provides a three-dimensional scene reconstruction method, which can be used in the aforementioned server. Figure 3 This is a second flowchart of a three-dimensional scene reconstruction method according to an embodiment of the present invention, such as... Figure 3 As shown, the process includes the following steps:
[0061] Step S301: Based on the acquired blurred image sequence and event sequence, the 3D Gaussian ellipsoid is initialized to obtain the initial Gaussian ellipsoid. For details, please refer to [link to relevant documentation]. Figure 2 Step S201 of the illustrated embodiment will not be described again here.
[0062] Step S302: Based on the geometric constraints of the clear image sequence, the blurred image sequence, and the event sequence corresponding to the ungenerated event regions obtained by rendering the blurred image sequence using the initial Gaussian ellipsoid, a loss function is constructed.
[0063] Specifically, step S302 includes:
[0064] Step S3021: Based on the difference between the clear image sequence and the corresponding blurry image, construct the blurry reconstruction loss function.
[0065] In this embodiment of the invention, the blur reconstruction loss function is used to characterize the difference between the rendering and reconstruction effect of the initial Gaussian ellipsoid and the blurry image sequence captured by a traditional camera. In high-speed motion scenes, due to the high-speed movement of the camera during the camera's exposure time, the captured image will have motion blur. The blurry image can be formed by superimposing multiple clear images. Therefore, the blurry image can be restored by superimposing a sequence of clear images. Thus, the blur reconstruction loss function is constructed based on the difference between the restored blurry image and the actual blurry image.
[0066] Specifically, a sequence of clear images within the exposure time corresponding to the blurred image is superimposed to obtain a simulated blurred image corresponding to the blurred image. This can be achieved by averaging the sequence of clear images within the exposure time corresponding to the blurred image, and by averaging the pixel values of each pixel in the clear image sequence to obtain the simulated blurred image. Based on the difference between the blurred image and the corresponding simulated blurred image, a blur reconstruction loss function is constructed. The difference between the blurred image and the corresponding simulated blurred image can be calculated by subtracting the pixel values of the blurred image from those of the corresponding simulated blurred image. The blur reconstruction loss function can be represented by the following formula (1):
[0067] Formula (1)
[0068] in, For fuzzy reconstruction loss function, The weight parameters are those of the fuzzy reconstruction loss function. For blurry images, Let represent the i-th sharp image obtained from rendering, and n represent the n sharp images rendered at different times within the exposure time of the blurry image.
[0069] Step S3022: Construct an event reconstruction loss function based on the difference between the event sequence and the corresponding clear image.
[0070] In this embodiment of the invention, the event reconstruction loss function is used to characterize the difference between the rendering and reconstruction effect of the initial Gaussian ellipsoid and the event sequence acquired by the event camera. The event camera generates an event when the change in pixel brightness reaches a set threshold. Therefore, the event reconstruction loss function can be constructed based on the difference between the brightness change reflected in the time series at two different times and the clear image.
[0071] Specifically, based on the integral of the event sequence over each preset time period, the first brightness change image corresponding to each preset time period is determined; based on the clear images at the start and end times corresponding to each preset time period, the second brightness change image corresponding to each preset time period is determined, wherein the brightness of the clear image at the end time is subtracted from the brightness of the clear image at the start time to obtain the second brightness change image corresponding to the preset time period; based on the difference between the first brightness change image and the second brightness change image corresponding to each preset time period, an event reconstruction loss function is constructed. The event reconstruction loss function can be shown in the following formula (2):
[0072] Formula (2)
[0073] in, For event reconstruction loss function, The weight parameters of the event reconstruction loss function. This is the first image showing the brightness change. The image shows the second brightness variation, where n represents the n preset time periods.
[0074] Step S3023: Construct geometric constraints for the event sequence based on the geometric constraints of the ungenerated event regions corresponding to the event sequence.
[0075] In this embodiment of the invention, since the event camera is sensitive to edges, that is, events are generally triggered at the edges, the area where no events are generated is usually a relatively flat area, and the area where events are generated is usually a relatively sharp area. Based on this, the geometric constraints of the event sequence are constructed.
[0076] Specifically, based on the event generation position corresponding to the event sequence, the region where no event was generated is determined; based on the zero-axis scale and orientation of the Gaussian ellipsoid corresponding to the region where no event was generated, the geometric constraints of the event sequence are constructed. Among them, for the pixel point in the region where no event was generated, Gaussian points within a preset range before and after the depth position are sampled according to the depth of the pixel point, and the Gaussian point that contributes the most to the depth position is selected, thereby obtaining the set of Gaussian points corresponding to the region where no event was generated. For these Gaussian point sets, constraints are constructed on their zero-axis scale and orientation. Specifically, the zero-axis scale of each Gaussian point in the Gaussian point set is constrained to be close to 0, thereby constraining the part of the Gaussian ellipsoid corresponding to the region where no event was generated to be a flat region, and the orientation of each Gaussian point in the Gaussian point set is constrained to be consistent with that of the neighboring Gaussian points. Specifically, the consistency of the normal vector of the Gaussian point can be used to constrain the orientation of the part of the Gaussian ellipsoid corresponding to the region where no event was generated to be consistent. The geometric constraints of the event sequence can be shown in the following formula (3):
[0077] Formula (3)
[0078] in, p Used to characterize Gaussian points in a Gaussian point set Used to characterize Gaussian points p The nearest point, Used to characterize the Gaussian point set corresponding to the region where no events were generated. Used to characterize Gaussian points p The set of nearest neighbors; For the geometric constraints of the event sequence, The weight parameters for the Gaussian points on the 0-axis scale are... The weighting parameters for the orientation of Gaussian points. The zero-axis scale is the Gaussian point in the Gaussian ellipsoid. express p Orientation of the point Gaussian ellipsoid express p Orientation of the Gaussian ellipsoid of the point neighboring points euler () indicates a rotation matrix to Euler angles.
[0079] Step S3024: Add the fuzzy reconstruction loss function, the event reconstruction loss function, and the event sequence geometric constraints together to obtain the loss function.
[0080] In this embodiment of the invention, the fuzzy reconstruction loss function, the event reconstruction loss function, and the event sequence geometric constraints are added together to obtain the loss function, which can be expressed as shown in the following formula (4):
[0081] Formula (4)
[0082] in, For loss function, For fuzzy reconstruction loss function, For event reconstruction loss function, Geometric constraints for the event sequence.
[0083] Step S303: Using the minimum value of the loss function as the optimization objective, adjust the parameter information of the initial Gaussian ellipsoid to obtain the target Gaussian ellipsoid. For details, please refer to [link to relevant documentation]. Figure 2 Step S203 of the illustrated embodiment will not be described again here.
[0084] Step S304: Reconstruct the 3D scene based on the target Gaussian ellipsoid to obtain the target 3D scene. For details, please refer to [link to relevant documentation]. Figure 2 Step S204 of the illustrated embodiment will not be described again here.
[0085] The 3D scene reconstruction method provided in this invention initializes a 3D Gaussian ellipsoid based on the acquired blurred image sequence and event sequence to obtain an initial Gaussian ellipsoid. This initial Gaussian ellipsoid is then combined with the blurred image sequence acquired by a traditional camera and the event sequence acquired by an event camera for 3D reconstruction, improving the reconstruction effect of the 3D scene. Simultaneously, based on the geometric constraints of the clear image sequence obtained by rendering the blurred image sequence using the initial Gaussian ellipsoid and the corresponding ungenerated event regions in the event sequence, a loss function is constructed. The minimum value of the loss function is used as the optimization objective to adjust the parameter information of the initial Gaussian ellipsoid, obtaining a target Gaussian ellipsoid. By adding geometric constraints, the Gaussian ellipsoid is guided and adjusted, further improving the 3D scene reconstruction effect.
[0086] This embodiment also provides a three-dimensional scene reconstruction device for implementing the above embodiments and preferred embodiments; details already described will not be repeated. As used below, the term "module" can refer to a combination of software and / or hardware that performs a predetermined function. Although the device described in the following embodiments is preferably implemented in software, hardware implementation, or a combination of software and hardware, is also possible and contemplated.
[0087] This embodiment provides a three-dimensional scene reconstruction device, such as... Figure 4 As shown, it includes:
[0088] The Gaussian ellipsoid initialization module 401 is used to initialize the three-dimensional Gaussian ellipsoid based on the acquired blurred image sequence and event sequence to obtain the initial Gaussian ellipsoid, wherein the event sequence and the blurred image sequence correspond to the same motion scene;
[0089] The loss function construction module 402 is used to construct the loss function based on the geometric constraints of the clear image sequence, the blurred image sequence, and the event sequence corresponding to the ungenerated event region obtained by rendering the blurred image sequence based on the initial Gaussian ellipsoid;
[0090] The Gaussian ellipsoid optimization module 403 is used to adjust the parameter information of the initial Gaussian ellipsoid with the goal of minimizing the loss function, so as to obtain the target Gaussian ellipsoid.
[0091] The 3D scene reconstruction module 404 is used to reconstruct the 3D scene based on the target Gaussian ellipsoid to obtain the target 3D scene.
[0092] In one alternative implementation, the loss function construction module 402 includes:
[0093] The fuzzy reconstruction loss function construction unit is used to construct the fuzzy reconstruction loss function based on the difference between the clear image sequence and the corresponding fuzzy image;
[0094] The event reconstruction loss function building unit is used to construct the event reconstruction loss function based on the difference between the event sequence and the corresponding clear image;
[0095] The event sequence geometric constraint construction unit is used to construct event sequence geometric constraints based on the geometric constraints of the ungenerated event regions corresponding to the event sequence.
[0096] The loss function determination unit is used to add the fuzzy reconstruction loss function, the event reconstruction loss function, and the event sequence geometric constraints to obtain the loss function.
[0097] In one alternative implementation, the fuzzy reconstruction loss function building unit includes:
[0098] The simulated blurred image determination subunit is used to superimpose a sequence of clear images within the exposure time corresponding to the blurred image to obtain the simulated blurred image corresponding to the blurred image.
[0099] The fuzzy reconstruction loss function construction sub-unit is used to construct the fuzzy reconstruction loss function based on the difference between the fuzzy image and the corresponding simulated fuzzy image.
[0100] In one alternative implementation, the simulated blurred image determination subunit includes:
[0101] The clear image processing submodule is used to perform mean processing on the sequence of clear images within the exposure time corresponding to the blurry image to obtain the simulated blurry image corresponding to the blurry image.
[0102] In one alternative implementation, the event reconstruction loss function building unit includes:
[0103] The first brightness change image determination subunit is used to determine the first brightness change image corresponding to each preset time period based on the integration of the event sequence over each preset time period.
[0104] The second brightness change image determination subunit is used to determine the second brightness change image corresponding to each preset time period based on the clear images at the start and end times of each preset time period.
[0105] The event reconstruction loss function construction sub-unit is used to construct the event reconstruction loss function based on the difference between the first brightness change image and the second brightness change image corresponding to each preset time period.
[0106] In one alternative implementation, the event sequence geometric constraint building unit includes:
[0107] The ungenerated event region determination sub-unit is used to determine the ungenerated event region based on the event generation position corresponding to the event sequence;
[0108] The event sequence geometric constraint construction sub-unit is used to construct the event sequence geometric constraints based on the 0-axis scale and orientation of the Gaussian ellipsoid corresponding to the region where no events have been generated.
[0109] The three-dimensional scene reconstruction apparatus provided in this embodiment of the invention can execute the three-dimensional scene reconstruction method provided in any embodiment of the invention, and has the corresponding functional modules and beneficial effects for executing the method. Further functional descriptions of the various modules and units described above are the same as in the corresponding embodiments described above, and will not be repeated here.
[0110] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present invention.
[0111] The following is a detailed reference. Figure 5 The diagram illustrates a structural schematic suitable for implementing an electronic device according to embodiments of the present invention. The electronic device may include a processor (e.g., a central processing unit, graphics processor, etc.) 501, which can perform various appropriate actions and processes according to a program stored in read-only memory (ROM) 502 or a program loaded from memory 508 into random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device. The processor 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.
[0112] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; memory devices 508 including, for example, magnetic tapes, hard disks, etc.; and communication devices 509. Communication device 509 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown, and more or fewer devices may be implemented or have instead.
[0113] In particular, according to embodiments of the present invention, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, embodiments of the present invention include a computer program product comprising a computer program carried on a non-transitory computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via a communication device 509, or installed from a memory 508, or installed from a ROM 502. When the computer program is executed by the processor 501, it performs the functions defined in the three-dimensional scene reconstruction method of the embodiments of the present invention.
[0114] Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of use of the embodiments of the present invention.
[0115] This invention also provides a computer-readable storage medium. The methods described above according to embodiments of the invention can be implemented in hardware or firmware, or implemented as computer code that can be recorded on a storage medium, or implemented as computer code downloaded via a network and originally stored on a remote storage medium or a non-transitory machine-readable storage medium and then stored on a local storage medium. Thus, the methods described herein can be processed by software stored on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. The storage medium can be a magnetic disk, optical disk, read-only memory, random access memory, flash memory, hard disk, or solid-state drive, etc.; further, the storage medium can also include combinations of the above types of memory. It is understood that computers, processors, microprocessor controllers, or programmable hardware include storage components capable of storing or receiving software or computer code. When the software or computer code is accessed and executed by the computer, processor, or hardware, the three-dimensional scene reconstruction method shown in the above embodiments is implemented.
[0116] A portion of this invention can be applied as a computer program product, such as computer program instructions, which, when executed by a computer, can invoke or provide the methods and / or technical solutions according to the invention through the operation of the computer. Those skilled in the art will understand that the forms in which computer program instructions exist in a computer-readable medium include, but are not limited to, source files, executable files, installation package files, etc. Correspondingly, the ways in which computer program instructions are executed by a computer include, but are not limited to: the computer directly executing the instructions, or the computer compiling the instructions and then executing the corresponding compiled program, or the computer reading and executing the instructions, or the computer reading and installing the instructions and then executing the corresponding installed program. Here, the computer-readable medium can be any available computer-readable storage medium or communication medium accessible to a computer.
[0117] Although embodiments of the invention have been described in conjunction with the accompanying drawings, those skilled in the art can make various modifications and variations without departing from the spirit and scope of the invention, and all such modifications and variations fall within the scope defined by the invention.
Claims
1. A method of three-dimensional scene reconstruction, characterized by, The method includes: Based on the acquired blurred image sequence and event sequence, the three-dimensional Gaussian ellipsoid is initialized to obtain an initial Gaussian ellipsoid, wherein the event sequence and the blurred image sequence correspond to the same motion scene; Based on the initial Gaussian ellipsoid rendering of the blurred image sequence to obtain a clear image sequence, the geometric constraints of the blurred image sequence and the corresponding ungenerated event regions of the event sequence, a loss function is constructed. This construction of the loss function includes: constructing a blurred image reconstruction loss function based on the difference between the clear image sequence and the corresponding blurred image; constructing an event reconstruction loss function based on the difference between the event sequence and the corresponding clear image; and constructing an event reconstruction loss function based on the geometric constraints of the corresponding ungenerated event regions of the event sequence. The event sequence geometric constraints are applied; the fuzzy reconstruction loss function, the event reconstruction loss function, and the event sequence geometric constraints are added together to obtain the loss function; the event reconstruction loss function is constructed based on the difference between the event sequence and the corresponding clear image, including: determining the first brightness change image corresponding to each preset time period based on the integral of the event sequence in each preset time period; determining the second brightness change image corresponding to each preset time period based on the clear images at the start and end times corresponding to each preset time period; and constructing the event reconstruction loss function based on the difference between the first brightness change image and the second brightness change image corresponding to each preset time period. The parameters of the initial Gaussian ellipsoid are adjusted with the minimum value of the loss function as the optimization objective to obtain the target Gaussian ellipsoid. Based on the target Gaussian ellipsoid, a three-dimensional scene is reconstructed to obtain the target three-dimensional scene.
2. The method of claim 1, wherein, The step of constructing a blur reconstruction loss function based on the difference between the clear image sequence and the corresponding blurry image includes: By superimposing a sequence of clear images within the exposure time corresponding to the blurred image, a simulated blurred image corresponding to the blurred image is obtained. The blur reconstruction loss function is constructed based on the difference between the blurred image and the corresponding simulated blurred image.
3. The method of claim 2, wherein, The step of superimposing a sequence of clear images within the exposure time corresponding to the blurred image to obtain a simulated blurred image corresponding to the blurred image includes: The sequence of clear images within the exposure time corresponding to the blurred image is averaged to obtain the simulated blurred image corresponding to the blurred image.
4. The method of claim 1, wherein, The construction of geometric constraints for the event sequence based on the geometric constraints of the ungenerated event regions corresponding to the event sequence includes: Based on the event generation location corresponding to the event sequence, determine the region where no events were generated; Based on the zero-axis scale and orientation of the Gaussian ellipsoid corresponding to the region where no events were generated, the geometric constraints of the event sequence are constructed.
5. A three-dimensional scene reconstruction apparatus, characterized by comprising: The device includes: The Gaussian ellipsoid initialization module is used to initialize a three-dimensional Gaussian ellipsoid based on the acquired blurred image sequence and event sequence to obtain an initial Gaussian ellipsoid, wherein the event sequence and the blurred image sequence correspond to the same motion scene; The loss function construction module is used to construct a loss function based on the clear image sequence obtained by rendering the blurred image sequence with the initial Gaussian ellipsoid, the geometric constraints of the blurred image sequence and the non-generated event regions corresponding to the event sequence, and the clear image sequence obtained by rendering the blurred image sequence with the initial Gaussian ellipsoid, the geometric constraints of the non-generated event regions corresponding to the event sequence, and the geometric constraints of the non-generated event regions corresponding to the event sequence. The construction of the loss function based on the difference between the clear image sequence and the corresponding blurred image includes: constructing a blurred reconstruction loss function based on the difference between the event sequence and the corresponding clear image; constructing an event reconstruction loss function based on the difference between the event sequence and the corresponding clear image; and constructing a loss function based on the geometric constraints of the non-generated event regions corresponding to the event sequence. The event sequence geometric constraints are constructed by adding the fuzzy reconstruction loss function, the event reconstruction loss function, and the event sequence geometric constraints to obtain the loss function. The step of constructing the event reconstruction loss function based on the difference between the event sequence and the corresponding clear image includes: determining a first brightness change image corresponding to each preset time period based on the integral of the event sequence over each preset time period; determining a second brightness change image corresponding to each preset time period based on the clear images at the start and end times corresponding to each preset time period; and constructing the event reconstruction loss function based on the difference between the first brightness change image and the second brightness change image corresponding to each preset time period. The Gaussian ellipsoid optimization module is used to adjust the parameter information of the initial Gaussian ellipsoid with the minimum value of the loss function as the optimization objective, so as to obtain the target Gaussian ellipsoid. The 3D scene reconstruction module is used to reconstruct the 3D scene based on the target Gaussian ellipsoid to obtain the target 3D scene.
6. An electronic device, characterized in that, include: The system includes a memory and a processor, which are interconnected and the memory stores computer instructions. The processor executes the computer instructions to perform the three-dimensional scene reconstruction method according to any one of claims 1 to 4.
7. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer instructions for causing the computer to perform the three-dimensional scene reconstruction method according to any one of claims 1 to 4.
8. A computer program product, characterized in that, Includes computer instructions for causing a computer to perform the three-dimensional scene reconstruction method according to any one of claims 1 to 4.