Method and processor for rendering 3D objects using multi-camera image input

Abstract

Methods and processors for rendering a 3D object are disclosed. The method includes: obtaining a multi-camera image input, the multi-camera image input including a first image frame of the 3D object generated by a first camera and a second image frame of the 3D object generated by a second camera; obtaining an initial 3D Gaussian Splatting (3DGS) model, the 3DGS model having a plurality of initial parameters, the plurality of initial parameters including initial frame-wise GS parameters and initial camera-wise GS parameters; generating an adjusted 3DGS model by adjusting at least one of: the initial frame-wise GS parameters; the initial camera-wise GS parameters based on the multi-camera image input; generating a 3DGS output with the adjusted 3DGS model; rendering a 2D image of the 3D object using the 3DGS output.

Description

[0001] Cross-reference to related applications

[0002] This application claims priority to U.S. Patent Application No. 18 / 597,468, filed March 6, 2024, entitled “METHODS AND PROCESSORS FOR RENDERING A 3D OBJECT USING MULTI-CAMERA IMAGE INPUTS”, the entire contents of which are incorporated herein by reference. Technical Field

[0003] The present invention generally relates to multi-camera rendering, and more specifically, to a method and processor for rendering 3D objects using multi-camera image input. Background Technology

[0004] Image data can refer to images captured by image sensors such as cameras. Autonomous vehicles can be modified and / or equipped with camera sensors to collect image data about the environment, process this image data, and trigger one or more control actions to automatically control the vehicle's behavior in the environment. For decision-making purposes, road elements such as vehicles, pedestrians, trees, and buildings can be simulated in the environment, and the vehicle's behavior can be controlled accordingly.

[0005] Collecting image data in "real-world scenarios" refers to the process of acquiring image data from real-world scenarios, environments, or uncontrolled environments where autonomous vehicles may operate. It should be noted that image data collected in real-world scenarios may include information acquired in natural environments, rather than in controlled laboratory or studio environments. Therefore, due to varying operating and / or environmental conditions, this data may be variable, unpredictable, and otherwise flawed. For example, images captured in real-world scenarios may come from multiple cameras of different types and / or with different point-of-view (POV) perspectives, resulting in misalignment and / or different image styles between the cameras.

[0006] It should be noted that traditional simulation systems may include data-driven models for generating objects during environmental simulation. Compared to manual or "hands-on" methods, data-driven models are capable of generating 3D objects in a more efficient manner and / or with higher quality. An example of a data-driven model is the Neural Radiance Field (NeRF) model, which is constructed from multiple successive multi-layer perceptrons (MLPs) and / or large encoder-decoder modules.

[0007] However, existing solutions for rendering images based on multi-camera image input still have many shortcomings when processing autonomous driving image data captured in real time and / or in real-world scenarios. Summary of the Invention

[0008] Developers have devised methods and processors to overcome at least some of the shortcomings of existing technological solutions.

[0009] In the context of this invention, "multi-camera rendering" refers to a set of simulation techniques that use data from multiple camera sensors to render real-world scenes and / or objects within them. These techniques may use one or more of data-driven models and / or hand-designed patterns to render real-world scenes and / or objects within them.

[0010] In the paper "S-Nerf: Neural Radiance Fields For Street Views" authored by Ziyang Xie et al. and published in 2023, a neural rendering pipeline for performing neural rendering on multi-camera image data is disclosed. Broadly speaking, the neural rendering pipeline comprises three steps: (i) learning confidence maps of multi-camera inputs in a real-world scene using multimodal inputs (e.g., images, LiDAR points, optical flow, etc.); (ii) generating noisy, dense depth maps using a depth completion network to facilitate learnable aggregation of different confidence maps; and (iii) constructing a NeRF-based neural renderer using multiple MLPs to render RGB images and depth maps.

[0011] However, developers have realized that neural rendering requires multimodal input (not just camera input) to aggregate rich information from multiple camera inputs. Furthermore, the model architectures required for neural rendering are highly complex (e.g., including numerous encoder-decoder networks and / or NeRF architectures with a large number of MLPs) (resulting in slow rendering of road features), making it unsuitable and / or unable to meet the real-time rendering requirements of autonomous driving applications.

[0012] Some real-time rendering solutions may employ machine learning models that are similar in implementation to those disclosed in the paper entitled “3D Gaussian Splatting for Real-Time Radiance Field Rendering”, authored by Bernhard Kerbl et al. and published on August 8, 2023 (the entire contents of which are incorporated herein by reference). For example, a 3DGaussian Splatting (3DGS) rendering pipeline may include three steps: (i) initializing Gaussian points provided by the software application for aligning the image and / or point cloud, and constructing a 3DGS model containing a set of parameters, including Gaussian Splatting (GS) mean position parameters, GS rotation parameters, GS scale parameters, GS opacity parameters, and GS spherical harmonics (SH) parameters; (ii) projecting the 3D Gaussian onto the image plane and rendering the image using a differentiable tile rasterizer; and (iii) employing a density control mechanism to clone and split the Gaussian.

[0013] However, the developers of this invention have realized that "off-the-shelf" (sometimes referred to as "basic") 3DGS models are not suitable for handling noisy (e.g., sparse and / or incomplete) multi-camera inputs. It should be noted that image data captured in real-world scenarios may exhibit inconsistencies and / or differences across cameras. Therefore, basic 3DGS models are not suitable for simulating road features for autonomous driving applications using multi-camera inputs.

[0014] As will be described in more detail below, in some embodiments of the present invention, the developers have designed an adjusted 3DGS model that enables real-time road feature simulation using multi-camera input during autonomous driving. In at least some embodiments, the adjusted 3DGS model can be generated by adaptively adjusting the Gaussian position parameters of the initial 3DGS model using a first module to compensate for multi-camera pose alignment deviations in the multi-camera input and / or improve spatiotemporal Gaussian consistency. In at least some other embodiments, the adjusted 3DGS model can be generated by using a second module to learn camera-related information to mitigate image style inconsistencies between multiple cameras.

[0015] Developers have recognized that in various simulators used for simulating real-world scenes and / or objects therein, an adjustable 3DGS model can be used instead of the initial 3DGS model. In one example, an adaptive 3DGS model can be implemented on an online cloud computing platform. In this example, the adaptive 3DGS model can be applied to the following pipeline: (i) a user selects and / or provides input of real-world data captured by multiple cameras and selects the adjustable 3DGS model to simulate the input; (ii) the online cloud computing platform trains the adjustable 3DGS model to simulate the input; (iii) outputs simulation results to the user, and / or optimizes the adaptive 3DGS model. In a second example, the adjustable 3DGS model can be implemented using a robot simulation system. The adjustable 3DGS model can be used to support the optimization of data-driven simulators. Therefore, different simulator systems can use the adaptive 3DGS model architecture as their base model architecture to build simulator systems.

[0016] In a first broad aspect of the present invention, a method for rendering a 3D object is provided. The method is executable by a processor. The method includes: acquiring multi-camera image input, the multi-camera image input including a first image frame of the 3D object generated by a first camera and a second image frame of the 3D object generated by a second camera, the first camera being different from the second camera. The method includes: acquiring an initial 3D Gaussian Splatting (3DGS) model, the 3DGS model having a plurality of initial parameters, the plurality of initial parameters including initial frame-by-frame Gaussian Splatting (GS) parameters and initial camera-by-camera GS parameters. The method includes: generating an adjusted 3DGS model based on the multi-camera image input by adjusting at least one of the following: the initial frame-by-frame GS parameters to compensate for misalignment between the first image frame and the second image frame; the initial camera-by-camera GS parameters to compensate for cross-camera differences between the first camera and the second camera. The method includes: generating a 3DGS output based on the multi-camera image input using the adjusted 3DGS model. The method includes: using the 3DGS output to render a 2D image of the 3D object.

[0017] In some embodiments of the method, generating the adjusted 3DGS model includes: adjusting the initial frame-by-frame GS parameters and the initial camera-by-camera GS parameters.

[0018] In some embodiments of the method, the initial frame-by-frame GS parameters are initial GS position parameters, and the initial camera-by-camera GS parameters are initial GS spherical harmonics (SH) parameters.

[0019] In some embodiments of the method, the plurality of initial parameters further include GS rotation parameters, GS scale parameters, and GS opacity parameters.

[0020] In some embodiments of the method, the adjustment of the initial frame-by-frame GS parameters is performed according to the following formula: ,in These are the initial frame-by-frame GS position parameters. These are the adjusted frame-by-frame GS position parameters. It is a rotation matrix. It is a translation matrix.

[0021] In some embodiments of the method, the adjustment of the initial camera-by-camera GS parameters is performed according to the following formula: , ,in It is a scaling factor. It is the bias factor, and SH is the initial per-camera GS SH parameter. It is the 0th order frequency band of the adjusted camera-by-camera GS SH parameters. It is the higher-order frequency band of the adjusted camera-by-camera GS SH parameters.

[0022] In some embodiments of the method, the method further includes using a multi-layer perceptron (MLP) model to optimize the rotation matrix and the translation matrix.

[0023] In some embodiments of the method, the method further includes using a multi-layer perceptron (MLP) model to optimize the scaling factor and the bias factor.

[0024] In some embodiments of the method, the rendering includes: projecting the 3DGS output onto a 2D plane using a differentiable renderer.

[0025] In some embodiments of the method, the method further includes: using the multi-camera image data to generate a camera repository; and using the camera repository to generate a global camera index.

[0026] In a second, broader aspect of the invention, a non-transitory computer-readable medium is provided. The computer-readable medium stores instructions that, when executed by a processor, cause the processor to perform the method described in the above embodiments.

[0027] In a third, broader aspect of the invention, a server is provided. The server is used to perform the method described according to the above embodiments.

[0028] In a fourth broad aspect of the invention, a processor for rendering 3D objects is provided. The processing is configured to: acquire multi-camera image input, the multi-camera image input including a first image frame of the 3D object generated by a first camera and a second image frame of the 3D object generated by a second camera, the first camera being different from the second camera. The processing is configured to: acquire an initial 3D Gaussian Splatting (3DGS) model, the 3DGS model having multiple initial parameters, the multiple initial parameters including initial frame-by-frame GS parameters and initial camera-by-camera GS parameters. The processing is configured to: generate an adjusted 3DGS model by adjusting at least one of the following based on the multi-camera image input: the initial frame-by-frame GS parameters to compensate for misalignment between the first image frame and the second image frame; the initial camera-by-camera GS parameters to compensate for cross-camera differences between the first camera and the second camera. The processing is configured to: generate a 3DGS output based on the multi-camera image input using the adjusted 3DGS model. The processing is configured to: render a 2D image of the 3D object using the 3DGS output.

[0029] In some embodiments of the processor, generating the adjusted 3DGS model includes: the processor adjusting the initial frame-by-frame GS parameters and the initial camera-by-camera GS parameters.

[0030] In some embodiments of the processor, the initial frame-by-frame GS parameters are initial GS position parameters, and the initial camera-by-camera GS parameters are initial GS spherical harmonics (SH) parameters.

[0031] In some embodiments of the processor, the plurality of initial parameters also include GS rotation parameters, GS scale parameters, and GS opacity parameters.

[0032] In some embodiments of the processor, the adjustment of the initial frame-by-frame GS parameters is performed by the processor according to the following formula: ,in These are the initial frame-by-frame GS position parameters. These are the adjusted frame-by-frame GS position parameters. It is a rotation matrix. It is a translation matrix.

[0033] In some embodiments of the processor, the adjustment of the initial per-camera GS parameters is performed according to the following formula: , ,in It is a scaling factor. It is the bias factor, and SH is the initial per-camera GS SH parameter. It is the 0th order frequency band of the adjusted camera-by-camera GS SH parameters. It is the higher-order frequency band of the adjusted camera-by-camera GS SH parameters.

[0034] In some embodiments of the processor, the processor is further configured to: use an MLP model to optimize the rotation matrix and the translation matrix.

[0035] In some embodiments of the processor, the processor is further configured to: use an MLP model to optimize the scaling factor and the bias factor.

[0036] In some embodiments of the processor, the rendering includes: the processor projecting the 3DGS output onto a 2D plane using a differentiable renderer.

[0037] In some embodiments of the processor, the processor is further configured to: use the multi-camera image data to generate a camera repository; and use the camera repository to generate a global camera index.

[0038] In a fifth aspect of the present invention, a computing device is provided. The computing device includes a processor configured according to the above embodiments.

[0039] In a sixth aspect, a robot simulation system is provided. The robot simulation system includes a processor configured according to the above embodiments.

[0040] In the context of this specification, a "server" is a computer program running on suitable hardware capable of receiving (e.g., from devices) requests over a network and executing or causing such requests to be executed. The hardware may be a physical computer or a physical computer system, but is not necessarily so for the purposes of this invention. In the context of this invention, the use of the term "server" is not intended to indicate that every task (e.g., a received instruction or request) or any particular task will be received, executed, or caused to be executed by the same server (i.e., the same software and / or hardware); it is intended to indicate that receiving / sending, executing, or causing the execution of any task or request, or the consequences of any task or request, may involve any number of software elements or hardware devices; and all such software and hardware may be one server or multiple servers, all of which are included within the expression "at least one server."

[0041] In the context of this specification, "device" means any computer hardware capable of running software suitable for the relevant task at hand. Therefore, some (non-limiting) examples of devices include personal computers (desktops, laptops, netbooks, etc.), smartphones and tablets, as well as network devices such as routers, switches, and gateways. It should be noted that, in the context of this invention, the use of a device as a device does not exclude the use of servers as other devices. The use of the term "device" does not preclude the use of multiple devices to receive / send, perform, or cause the performance of any task or request, or the result of any task or request, or the steps of any method described herein.

[0042] In the context of this specification, a "database" is any structured collection of data, independent of its specific structure, database management software, or computer hardware that stores, implements, or otherwise provides the data. A database may reside on the same hardware as the processes that store or use the information stored in the database, or it may reside on separate hardware, such as a dedicated server or multiple servers. In essence, a database is a logically ordered, electronically stored collection of structured data within a computer system.

[0043] In the context of this specification, the term "information" includes information of any nature or kind that can be stored in a database. Therefore, information includes, but is not limited to, audiovisual works (images, films, recordings, presentations, etc.), data (location data, numerical data, etc.), text (opinions, comments, questions, messages, etc.), documents, spreadsheets, glossaries, etc.

[0044] In the context of this specification, the term "component" is intended to include software (appropriate to the particular hardware context) that is both necessary and sufficient for implementing the particular function referenced.

[0045] In the context of this specification, the term "computer-usable information storage medium" is intended to include media of any nature and kind, including RAM, ROM, disks (CD-ROM, DVD, floppy disk, hard disk, etc.), USB keys, solid-state drives, tape drives, etc.

[0046] In the context of this specification, the words “first,” “second,” “third,” etc., are used as adjectives only to distinguish the nouns they modify from one another, and not to describe any particular relationship between those nouns. Therefore, for example, it should be understood that the use of the terms “first server” and “third server” is not intended to imply any particular order, type, chronological sequence, hierarchy, or sorting (for example) between servers / servers, nor is their use (in itself) intended to imply that any “second server” must exist in any given situation. Furthermore, as discussed in other contexts herein, references to the elements “first” and “second” do not preclude the two elements from being the same actual real-world element. Therefore, for example, in some cases, the “first” server and the “second” server may be the same software and / or hardware, while in others they may be different software and / or hardware.

[0047] All implementations of the present invention include at least one of the above-mentioned objectives and / or aspects, but not necessarily all of them. It should be understood that some aspects of the present invention formed to achieve the above-mentioned objectives may not achieve those objectives, and / or may achieve other objectives not specifically described herein.

[0048] Additional and / or alternative features, aspects, and advantages of the present invention will be apparent from the following description, the accompanying drawings, and the appended claims. Attached Figure Description

[0049] To better understand the present invention, as well as other aspects and features thereof, reference is made to the following description used in conjunction with the accompanying drawings, in which:

[0050] Figure 1 An example of a computing device that can be used to implement any of the methods described herein is shown;

[0051] Figure 2 A first processing pipeline according to a first non-limiting embodiment of the present invention is shown for generating a first adjusted 3DGS model for rendering a 3D object from multi-camera input;

[0052] Figure 3A second processing pipeline according to a second non-limiting embodiment of the present invention is shown for generating a second adjusted 3DGS model for rendering a 3D object;

[0053] Figure 4 A third processing pipeline according to a third non-limiting embodiment of the present invention is shown for generating a third adjusted 3DGS model for rendering 3D objects;

[0054] Figure 5 A fourth processing pipeline according to a fourth non-limiting embodiment of the present invention is shown for generating a fourth adjusted 3DGS model for rendering 3D objects;

[0055] Figure 6 The invention illustrates at least some non-limiting embodiments of the technology according to the present invention. Figure 1 A block diagram illustrating the method executed by the processor of a computing device.

[0056] Appendix A is an unpublished document entitled “ADGauS: Autonomous Driving Road Asset Simulation with Adaptive Gaussian Splatting”, which describes at least some non-limiting implementations of the present invention. Detailed Implementation

[0057] The examples and conditions described herein are primarily intended to help the reader understand the principles of the invention and not to limit its scope to such specific examples and conditions. It should be understood that those skilled in the art can design various devices, which, although not explicitly described or shown herein, embody the principles of the invention and are included within its spirit and scope.

[0058] Furthermore, for ease of understanding, the following description illustrates a relatively simplified implementation of the present invention. As those skilled in the art will understand, various implementations of the present invention may involve greater complexity.

[0059] In some cases, useful examples of modifications to the technology of this invention may also be described. This is merely for ease of understanding and not for defining the scope or limits of the technology of this invention. These modifications are only some examples, and those skilled in the art can make other modifications while remaining within the scope of the technology of this invention. Furthermore, if examples of modifications are not listed, it should not be construed as meaning that modifications cannot be made and / or that the described content is the only way to implement the elements of the technology of this invention.

[0060] Furthermore, all descriptions and specific examples of the principles, aspects, and implementations of the present invention herein are intended to cover their structural and functional equivalents, whether such equivalents are currently known or will be developed in the future. Therefore, for example, those skilled in the art will understand that any block diagram herein represents a conceptual view of an illustrative circuit embodying the principles of the present invention. Similarly, it should be understood that any flowchart, diagrammatic flowchart, state transition diagram, pseudocode, etc., represents various processes that can be represented substantially in a computer-readable medium and therefore executed by a computer or processor, whether or not such computer or processor is explicitly shown.

[0061] The functionality of the various elements shown in the figure (including any functional blocks labeled "processor") can be provided by using dedicated hardware and hardware capable of executing software in association with appropriate software. When provided by a processor, these functions can be provided by a single dedicated processor, a single shared processor, or multiple independent processors (some of which may be shared). In some embodiments of the present invention, the processor can be a general-purpose processor, such as a central processing unit (CPU), or a purpose-specific processor, such as a digital signal processor (DSP). Furthermore, the explicit use of the term "processor" should not be construed as specifically referring to hardware capable of executing software, and may implicitly include, but is not limited to, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), read-only memory (ROM), random access memory (RAM), and non-volatile memory for storing software. Other conventional and / or custom hardware may also be included.

[0062] A software module, or simply a module implied as software, may be represented herein as any combination of flowchart elements or other elements indicating process steps and / or textual descriptions of performance. Such modules may be executed by hardware, whether explicitly or implicitly shown. Furthermore, it should be understood that modules may include, for example, but not limited to, computer program logic, computer program instructions, software, stacks, firmware, hardware circuitry, or combinations thereof, providing the required capabilities. For example, such software modules may be provided as part of a non-transitory computer-readable medium on which instructions are stored, which, when executed by a processor, cause the processor to perform software-defined process steps.

[0063] Based on these fundamental principles, some non-limiting examples will now be considered to illustrate various implementations of various aspects of the present invention.

[0064] Figure 1 A diagram illustrating a computing environment 100 according to an embodiment of the present invention is shown. In some embodiments, the computing environment 100 may be implemented by any conventional personal computer, a computer dedicated to running and / or monitoring data center-related systems, controllers and / or electronic devices (e.g., but not limited to mobile devices, tablet devices, servers, controller units, control devices, monitoring devices, etc.), and / or any combination of the above suitable for the relevant task at hand. In some embodiments, the computing environment 100 includes various hardware components, including one or more single-core or multi-core processors (collectively referred to as processor 110), a solid-state drive 120, random access memory 130, and input / output interfaces 150.

[0065] In some embodiments, computing environment 100 may also be a subsystem of one of the systems listed above. In some other embodiments, computing environment 100 may be an "off-the-shelf" general-purpose computer system. In some embodiments, computing environment 100 may also be distributed among multiple systems. Computing environment 100 may also be specifically designed to implement the techniques of the present invention. As those skilled in the art will appreciate, various variations regarding how computing environment 100 can be implemented without departing from the scope of the present invention.

[0066] Communication between the various components of the computing environment 100 can be achieved through one or more internal and / or external buses 160 (e.g., PCI bus, Universal Serial Bus, IEEE 1394 FireWire bus, SCSI bus, Serial ATA bus, ARINC bus, etc.), to which the various hardware components are electronically coupled.

[0067] Input / output interface 150 can implement network capabilities, such as wired or wireless access. For example, input / output interface 150 may include network interfaces, such as, but not limited to, network ports, network sockets, network interface controllers, etc. Several examples of how the network interface can be implemented will become apparent to those skilled in the art. For example, but not limited to, the network interface can implement specific physical layer and data link layer standards, such as Ethernet, Fibre Channel, Wi-Fi, or Token Ring. Specific physical and data link layers can provide the foundation for a complete network protocol stack, thereby supporting communication between small groups of computers on the same local area network (LAN) and large-scale network communication via routable protocols such as Internet Protocol (IP).

[0068] In some embodiments of the present invention, computing environment 100 may be communicatively coupled to one or more cameras (e.g., pinhole cameras, omnidirectional cameras, etc.) for receiving multi-camera input, and / or communicatively coupled to one or more user devices in a cloud platform for receiving multi-camera input provided by users of the cloud platform. For example, processor 110 may be used to acquire data from a first camera 170 and a second camera 175.

[0069] A pinhole camera is a camera that uses a small aperture (pinhole) to capture images. Light passes through this tiny opening and forms an inverted image on a photosensitive surface located inside a sealed container opposite the pinhole.

[0070] An omnidirectional camera is a device capable of capturing a complete 360-degree field of view in all directions. Unlike traditional cameras with limited field of view, omnidirectional cameras use dedicated lenses or groups of lenses to capture complete spherical or cylindrical images. These cameras are commonly used in virtual reality applications, autonomous driving, security surveillance, immersive video experiences, and the creation of panoramic photos or videos.

[0071] According to an implementation of the present invention, the solid-state drive 120 stores program instructions suitable for loading into the random access memory 130 and being executed by the processor 110, for performing data center operations based on a generated machine learning pipeline. For example, the program instructions may be part of a library or application.

[0072] In some embodiments of the present invention, the computing environment 100 can be implemented as part of a cloud computing environment. Broadly speaking, a cloud computing environment is a type of computing that relies on a network of remote servers hosted on the Internet to store, manage, and process data, rather than relying on local servers or personal computers. This type of computing allows users to access data and applications from remote locations and provides a scalable, flexible, and cost-effective solution for data storage and computing. Cloud computing environments can be divided into three main categories: Infrastructure as a Service (IaaS), Platform as a Service (PaaS), and Software as a Service (SaaS). In an IaaS environment, for example, users can rent virtual servers, storage, and other computing resources from a third-party provider. In a PaaS environment, users can access a platform for developing, running, and managing applications without managing the underlying infrastructure. In a SaaS environment, for example, users can access pre-built software applications hosted by a third-party provider. In summary, cloud computing environments offer a range of benefits, including cost savings, scalability, increased agility, and the ability to quickly deploy and manage applications.

[0073] refer to Figure 2 The figure depicts a first processing pipeline 204 for generating a first adjusted 3DGS model 250 by processor 110 of computer environment 100. More specifically, processor 110 may be used to: (i) acquire an initial 3DGS model 220; and (ii) adjust the parameters of the initial 3DGS model 220 using an Adaptive Gaussian Adjustment (ADA) module 230 and a Camera-Dependent Gaussian Modeling (CDGM) module 240 to generate the adjusted 3DGS model 250.

[0074] It can be envisioned that the initial 3DGS model 220 could be implemented in a manner similar to the 3DGS model disclosed in the paper entitled "3D Gaussian Splatting for Real-Time Radiance Field Rendering". In other words, the initial 3DGS model 220 can be considered a foundational implementation of a given 3DGS neural architecture. As for the basic 3DGS model, it consists of learnable parameters of 3D Gaussians and a differentiable tile rasterizer, where each Gaussian is defined by its position (mean), rotation, scaling, opacity, and spherical harmonic function.

[0075] In some embodiments, the initial 3DGS model 220 can be provided to the processor 110 via a cloud platform, either separately from or as a supplement to the multi-camera input 200. The multi-camera input 200 includes sequences of image frames generated by different cameras. It is conceivable that the multi-camera input 200 may include a set of image frame sequences captured by more than one camera.

[0076] Developers have recognized that, since the multi-camera input 200 is typically generated using different cameras in a real-world scene, image frames from different cameras are often misaligned. Therefore, during 3DGS learning, it may be necessary to implement the ADA module 230 to adaptively adjust the Gaussian position on the corresponding image frames, thereby compensating for multi-camera pose alignment deviations in the multi-camera input 200 captured by multiple cameras in a real-world scene, and / or improving spatiotemporal Gaussian consistency.

[0077] The developers of this invention have also recognized that, due to differences in lighting and / or sensors, image styles may vary between different cameras, making the initial 3DGS model 220 unsuitable for processing image input captured in real-world scenes. Therefore, it may be necessary to implement a CDGM module 240 to learn camera-related information, thereby compensating for the inconsistency in image styles among the multiple camera inputs 200.

[0078] In some embodiments of the present invention, processor 110 may be used to generate camera repository 210 to store information from multi-camera input 200. Processor 110 may generate camera repository 210 to store image-by-image camera information (e.g., camera extrinsic and intrinsic parameters). Processor 110 may also use camera repository 210 to construct an image index. For example, image data may be organized in the image index based on at least one of the following: (i) the camera ID of the corresponding camera used to capture the image data; (ii) parameters of the corresponding camera; (iii) timestamps associated with image frames, etc.

[0079] In some embodiments, the information stored in the camera repository 210 can be used by at least one of the initial 3DGS model 220, the AGA module 230, the CDGM module 240, and the adjusted 3DGS model 250. Additionally or alternatively, the information stored in the camera repository 210 can be used by the processor 110 to select the most recent training viewpoint during rendering of a new viewpoint. Specifically, the processor 110 can be used to construct the camera repository using a multi-camera input image sequence captured from an autonomous vehicle. This is used to store image-by-image camera information (i.e., camera extrinsic and intrinsic parameters). This camera repository... Processor 110 is used to render new perspectives During this period, the nearest training viewpoint is selected, which can be expressed by the formula as follows: , where D is used to measure the distance between two camera viewpoints (e.g., angular distance), and I is the index of the nearest viewpoint.

[0080] In some embodiments, the processor 110 may acquire an initial 3DGS model 220, which has multiple raw learnable neural network parameters, including: initial GS mean location parameters. 221, used to control the position of each Gaussian; initial GS rotation parameters. 222, used to control the rotation direction of each Gaussian; initial GS scale parameter. 223, used to control the scale of each Gaussian; initial GS opacity parameter. 224, used to control whether Gaussian fragments are visible; initial GS spherical harmonics (SH) parameter. 225 is used to control the color of Gaussian fragments, where all these parameters are defined in 3D space and therefore associated with all 2D image sources; that is, each 2D image or each frame can be viewed as a projection of the 3D Gaussian at each specific viewpoint. As will be described in more detail below, one or more initial GS parameters of the initial 3D GS model 220 can be selectively adjusted to learn the 3D information contained in the multi-camera inputs 200.

[0081] In some embodiments of the present invention, it can be stated that PyTorch can be used to train one or more machine learning algorithms. Broadly speaking, PyTorch is a full-featured framework for building deep learning models; deep learning is a type of machine learning that can be used for applications such as image recognition and language processing. Within the PyTorch framework, one or more neural network parameters can be learnable tensors.

[0082] In some embodiments, the processor 110 may be used to perform a learning process to adjust the underlying 3DGS model. Broadly speaking, this learning process includes the following set of steps: (i) acquiring multi-camera input; (ii) learning a Gaussian in 3D space using the underlying 3DGS model; (iii) thinning / adjusting the Gaussian; (iv) rendering the image using a rasterizer and optimizing the model (adjusted model and / or underlying 3DGS model) using a loss; and (v) repeating steps (i) through (iv) until one or more stopping conditions are met.

[0083] It can be said that the initial GS mean position parameter 221 is the GS mean position parameter of the initial 3D GS model 220. Specifically, for multi-camera image inputs, they are used to learn in 3D space in the initial 3DGS model. This controls the position of each Gaussian, ensuring that the projection of the 3D Gaussian onto each image plane is consistent with each input image.

[0084] It can be said that the initial GS SH parameter 225 is the GS spherical harmonic function parameter of the initial 3D GS model 220. .

[0085] In some embodiments, the AGA module 230 can be trained and / or used to generate adjusted GS mean position parameters 251 for a given image frame based on the initial GS mean position parameters 221 of the initial 3DGS model 220 and information stored in the camera repository 210. Specifically, the camera repository is first constructed using a multi-camera input image sequence captured from an autonomous vehicle. This camera repository stores image-by-image camera information (i.e., camera extrinsic and intrinsic parameters). It is used to render new perspectives. During this period, the nearest training view is selected, which can be expressed by the formula as follows: , where D is used to measure the distance between two camera viewpoints (e.g., angular distance), and I is the index of the nearest viewpoint. Then, based on Use the corresponding first indivual and The transformation is performed. It can be envisioned that processor 110 could be used to optimize the Gaussian position rotation matrix for each input frame. Translation matrix To adaptively adjust the Gaussian according to the following formula :

[0086] (1)

[0087] in It is the Gaussian position per frame. It is the adjusted frame-by-frame Gaussian position. It is a rotation matrix. It is a translation matrix. In other words, These are adaptively adjusted Gaussian position parameters. and This can be achieved using learnable neural network parameters in the form of learnable tensors, consistent with the initial 3DGS model 220. Adjusting the modeling parameters frame-by-frame involves using the position and rotation matrices for each frame. Translation matrix They represent for each Additional rotations and translations are needed to keep it consistent across multiple cameras.

[0088] In some embodiments, a given camera repository can store camera extrinsic and intrinsic parameters. Therefore, for a given rendering camera viewpoint, the given camera repository can be used to calculate the distance between the rendering viewpoint and the training viewpoints in the given camera repository to determine which viewpoint in the camera repository is the closest viewpoint. Furthermore, the processor 110 can use the closest viewpoint to retrieve... and To perform the transformation.

[0089] In some embodiments, processor 110 can be used to generate a global camera index using camera repository 210. CDGM module 210 can then use the global camera index, utilizing a scaling factor according to the following formula. and bias factor To optimize camera-related Gaussian :

[0090] (2)

[0091] (3)

[0092] in It is a scaling factor. It is the bias factor, SH is the camera-by-camera SH. It is the 0th order frequency band of the adjusted camera-by-camera SH parameters. These are the higher-order frequency bands of the adjusted per-camera SH parameters. It is conceivable that the processor 110 can be used to adjust the camera correlation factors separately for different frequency bands, as these bands have different uses and learning rates. It should be noted that each SH can have multiple orders, where each order corresponds to a specific frequency band in the rendered color result. For example, the 0th-order frequency band of the SH corresponds to low-frequency features, while other frequency bands of the SH correspond to high-frequency features. Additionally, it is conceivable that the scaling factor... and bias factor This can be achieved using learnable neural network parameters in the form of learnable tensors, consistent with the basic 3DGS model 220. The camera-by-camera modeling parameters are the scaling factors for each camera. and bias factor These factors represent the effect of each Additional scaling and biasing are applied to make it specific to each camera, thereby learning camera-related information.

[0093] It can be envisioned that the adjusted 3DGS model 250 can have the initial GS rotation parameters 222, initial GS scale parameters 223, and initial GS opacity parameters 224 of the initial 3DGS model 220. In other words, when generating the adjusted 3DGS model, the subset of initial GS parameters of the initial 3DGS model can remain unchanged.

[0094] During use, the adjusted 3DGS model 250 can be input with information stored in the camera repository 210, and a 3D output in Gaussian form can be generated. Specifically, given a camera viewpoint, the nearest camera viewpoint is searched in the camera repository using that viewpoint to retrieve an index. Then, based on Use the corresponding parameter number indivual and Perform the transformation. Similarly, based on This allows us to infer the camera index (i.e., which camera the viewpoint originates from) so that the corresponding parameters can also be used. and Perform the transformation.

[0095] In some embodiments, processor 110 may be used to project a 3D Gaussian onto a 2D image plane using a differentiable renderer to perform image rendering operation 206. Here, a differentiable renderer refers to a fast differentiable rasterizer for Gaussians described in the literature entitled "3D Gaussian Splatting for Real-Time Radiance Field Rendering". The main function of a differentiable renderer is to render a 2D image by projecting a given 3D Gaussian in 3D space onto a 2D image plane with a given camera viewpoint.

[0096] It should be noted that for 3DGS models, a differentiable renderer may be required because the image is rendered based on the output of the 3DGS model. For example, in practice, a camera view matrix is ​​provided, and the image is rendered using that camera view matrix and the learned 3DGS model; this operation may require a differentiable rasterizer.

[0097] In some embodiments, the processor 110 can generate outputs 208 from the multi-camera image input 200, which correspond to one or more rendered 2D images of 3D objects captured in image frames.

[0098] refer to Figure 3 The figure depicts a second processing pipeline for generating a modified 3DGS model 350 by processor 110 of computer environment 100. In this embodiment, processor 110 may be used to replace ADA module 230 with a second ADA module 330 and CDGM module 240 with a second CDGM module 340. Figure 2 In contrast, processor 110 can be used to implement the second ADA module 330 and the second CDGM module 340 using one or more MLP-based learning models, instead of using learnable neural network parameters to implement the ADA module 230 and CDGM module 240.

[0099] In one embodiment, a given MLP-based learning model may include a linear layer, an activation layer, another linear layer, and a learnable scaling factor. Alternatively, without departing from the scope of the invention, an MLP may include multiple sequential linear layers, activation layers, and normalization layers.

[0100] In the context of other ADA modules 330, the input to the MLP-based learning model 335 can be camera information obtained by the processor 110 from the camera repository 210, and the output of the MLP-based learning model can be used to optimize the new Gaussian position. matrix and To optimize the model, a training objective incorporating image loss was formulated. The loss is defined as Where L1 is the L1 loss between the ground truth image and the rendered image. This is the D-SSIM term, and λ is the weighting parameter. Using this training objective loss and gradient backpropagation, an adaptive 3DGS model is trained from scratch based on multi-camera input images to optimize a specific 3DGS model that can be used to render images of objects for different camera perspectives.

[0101] In some embodiments, an optimized model can be trained for simulation purposes of autonomous vehicles. Multi-camera image inputs of the autonomous vehicle can be provided, and the optimized model can then be trained from scratch during initialization without using any pre-trained model. Therefore, the training dataset may only include training images of the autonomous vehicle and not training images of other autonomous vehicles. It is conceivable that the base 3DGS model can be trained from scratch using one or more modules disclosed herein, thus eliminating the need for pre-training.

[0102] In the context of another CDGM module 340, the input to the MLA-based learning model 345 can be camera information obtained by the processor 110 from the camera repository 210, while the output can be used to optimize new Gaussian colors. scaling factor and bias factor For example, the parameters of the CDGM module 340 can be trained from scratch using a defined training objective and given multi-camera input images.

[0103] In an additional embodiment of the present invention, the adaptive Gaussian parameters may use only the scale factor. Or bias factor Update without using the scale factor at the same time. Or bias factor In other words, formulas (2) and (3) can be defined as and ,or and .

[0104] refer to Figure 4The figure depicts a third processing pipeline for generating an adjusted 3DGS model 450 by processor 110 of computer environment 100. In this embodiment, processor 110 may be used to generate ADA module 230 (or second ADA module 330). However, unlike the first and second processing pipelines, the third processing pipeline does not include a CDGM module. In this embodiment, processor 110 may use ADA module 230 to adjust the Gaussian mean position. However, the adjusted 3DGS model 450 may retain the initial SH parameters 225 from the initial 3DGS model 220.

[0105] In this embodiment, the processor 110 can be used to adaptively adjust the Gaussian position to compensate for multi-camera pose alignment misalignment and improve spatiotemporal Gaussian consistency. Developers have recognized that implementing a given ADA module can help mitigate misalignment issues caused by image data captured by multiple cameras in a real-world scene, thus outperforming the initial 3DGS model in rendering more accurate 3D output results.

[0106] refer to Figure 5 The figure depicts a fourth processing pipeline for generating a modified 3DGS model 550 by processor 110 of computer environment 100. In this embodiment, processor 110 may be used to generate CDGM module 240 (or second CDGM module 340). However, unlike the first and second processing pipelines, the fourth processing pipeline does not include an ADA module.

[0107] In this embodiment, processor 110 can use CDGM module 230 to adjust the GS SH parameters. However, the adjusted 3DGS model 550 can retain the original Gaussian mean position from the initial 3DGS model 220. In this embodiment, processor 110 can be used to learn camera-related information to attempt to mitigate image style inconsistencies across multiple cameras. Developers have recognized that cross-camera style differences caused by lighting and / or sensor variations can be mitigated, thereby improving the initial 3DGS model and enabling it to adaptively learn to model diverse input image data from different cameras.

[0108] In some embodiments of the present invention, the processor 110 can be used to execute... Figure 6 The computer-implemented method 600 is shown. The steps of method 600 will now be described in more detail.

[0109] Step 602: Acquire multi-camera image input, which includes a first image frame of the 3D object generated by the first camera and a second image frame of the 3D object generated by the second camera. The first camera and the second camera are different.

[0110] Method 600 begins with: processor 110 acquiring multi-camera image input 300, the multi-camera image input including a first image frame of a 3D object generated by a first camera and a second image frame of a 3D object generated by a second camera. The first camera and the second camera are different. For example, the first camera may be camera 170, while the second camera may be camera 175.

[0111] The multi-camera input 200 includes a sequence of image frames generated by different cameras. It is conceivable that the multi-camera input 200 may include a set of image frame sequences captured by more than one camera.

[0112] The developers have recognized that, since multi-camera input 200 is typically generated using different cameras in a real-world scene, image frames from different cameras are often misaligned. Therefore, during 3DGS learning, it may be necessary to implement an ADA module 230 to adaptively adjust the Gaussian position on the corresponding image frames, thereby compensating for multi-camera pose alignment deviations in the multi-camera input 200 captured by multiple cameras in a real-world scene, and / or improving spatiotemporal Gaussian consistency. The developers of this invention have also recognized that, due to differences in lighting and / or sensors, image styles may differ between different cameras, making the initial 3DGS model 220 unsuitable for processing image inputs captured in real-world scenes. Therefore, it may be necessary to implement a CDGM module 240 to learn camera-related information, thereby compensating for the image style inconsistency problem in the multi-camera input 200.

[0113] Step 604: Obtain the initial 3D Gaussian Splatting (3DGS) model, which has multiple initial parameters, including initial frame-by-frame GS parameters and initial camera-by-camera GS parameters.

[0114] Method 600 continues to step 604: Processor 110 is used to acquire an initial 3DGS model 220, which has multiple initial parameters. These multiple initial parameters include initial frame-by-frame GS parameters and initial camera-by-camera GS parameters.

[0115] The initial frame-by-frame GS parameters can be described as the initial GS mean position parameters 221 of the initial 3D GS model 220. The initial camera-by-camera GS parameters can also be described as the initial GS SH parameters 225 of the initial 3D GS model 220. In some embodiments, the initial frame-by-frame GS parameters are initial GS position parameters, and the initial camera-by-camera GS parameters are initial GS SH parameters. In some embodiments, these multiple initial parameters also include GS rotation parameters, GS scale parameters, and GS opacity parameters.

[0116] Step 606: Generate the adjusted 3DGS model by adjusting at least one of the following based on the multi-camera image input: initial frame-by-frame GS parameters to compensate for misalignment between the first and second image frames; initial camera-by-camera GS parameters to compensate for cross-camera differences between the first and second cameras.

[0117] Method 600 continues with step 606: Processor 110 generates an adjusted 3DGS model by adjusting at least one of the following based on multi-camera image input: initial frame-by-frame GS parameters to compensate for misalignment between the first image frame and the second image frame; initial camera-by-camera GS parameters to compensate for cross-camera differences between the first camera and the second camera.

[0118] In some embodiments, such as Figure 2 or Figure 3 As shown, processor 110 can be used to adjust the initial frame-by-frame GS parameters and the initial camera-by-camera GS parameters to generate the adjusted 3D GS model 250. Figure 2 ) and / or generate an adjusted 3DGS model 350 ( Figure 3 ).

[0119] In some embodiments, processor 110 may be used to perform an adjustment of the initial frame-by-frame GS parameters according to formula (1). In other embodiments, processor 110 may be used to adjust the initial camera-by-camera GS parameters according to formulas (2) and (3).

[0120] In some embodiments of the present invention, such as Figure 3 As shown, processor 110 can use MLP-based machine learning model 335 to optimize the rotation and translation matrices. In other embodiments, such as Figure 3 As shown, processor 110 can use MLP-based machine learning model 345 to optimize scaling factor and bias factor.

[0121] Step 608: Generate 3DGS output based on multi-camera image input using the adjusted 3DGS model.

[0122] Method 600 proceeds to step 608: The processor generates 3DGS output using the adjusted 3DGS model. It should be noted that the output of the adjusted 3DGS model can take the form of one or more 3D Gaussians, which are calculated based on the multi-camera input 200 using the adjusted 3DGS model.

[0123] Step 610: Use 3DGS output to render a 2D image of the 3D object.

[0124] Method 600 continues with step 610: Processor 110 is used to render a 2D image of a 3D object using the 3DGS output. For this purpose, processor 110 may employ a differentiable renderer. In some embodiments, processor 110 may be used to project a 3D Gaussian onto a 2D image plane using a differentiable renderer to perform image rendering operation 206. In one embodiment, the differentiable renderer may be a differentiable rasterizer for Gaussians described in the literature entitled "3D Gaussian Splatting for Real-Time Radiance Field Rendering".

[0125] Modifications and improvements to the above-described implementations of the present invention will be readily apparent to those skilled in the art. The above description is intended to be illustrative rather than limiting. Therefore, the scope of the present invention is intended to be limited only by the scope of the appended claims.

Claims

1. A method for rendering 3D objects, characterized in that, The method can be executed by a processor, and the method includes: Acquire multi-camera image input, the multi-camera image input including a first image frame of the 3D object generated by a first camera and a second image frame of the 3D object generated by a second camera, wherein the first camera and the second camera are different; Obtain an initial 3D Gaussian Splatting (3DGS) model, which has multiple initial parameters, including initial frame-by-frame Gaussian Splatting (GS) parameters and initial camera-by-camera GS parameters. The adjusted 3DGS model is generated by adjusting at least one of the following based on the multi-camera image input: (i) The initial frame-by-frame GS parameters are used to compensate for the misalignment between the first image frame and the second image frame; (ii) the initial per-camera GS parameters, to compensate for cross-camera differences between the first camera and the second camera; The adjusted 3DGS model is used to generate 3DGS output based on the multi-camera image input; The 3DGS output is used to render a 2D image of the 3D object.

2. The method according to claim 1, characterized in that, Generating the adjusted 3DGS model includes: adjusting the initial frame-by-frame GS parameters and the initial camera-by-camera GS parameters.

3. The method according to claim 1 or 2, characterized in that, The initial frame-by-frame GS parameters are the initial GS position parameters, and the initial camera-by-camera GS parameters are the initial GS spherical harmonics (SH) parameters.

4. The method according to any one of claims 1 to 3, characterized in that, The initial parameters also include GS rotation parameters, GS scale parameters, and GS opacity parameters.

5. The method according to any one of claims 1 to 4, characterized in that, The initial frame-by-frame GS parameters are adjusted according to the following formula: in These are the initial frame-by-frame GS position parameters. These are the adjusted frame-by-frame GS position parameters. It is a rotation matrix. It is a translation matrix.

6. The method according to any one of claims 1 to 5, characterized in that, The initial camera-by-camera GS parameters are adjusted according to the following formula: ， ， in It is a scaling factor. It is the bias factor, and SH is the initial per-camera GS SH parameter. It is the 0th order frequency band of the adjusted camera-by-camera GS SH parameters. It is the higher-order frequency band of the adjusted camera-by-camera GS SH parameters.

7. The method according to claim 5, characterized in that, The method further includes: The rotation matrix and the translation matrix are optimized using a multi-layer perceptron (MLP) model.

8. The method according to claim 6, characterized in that, The method further includes: The scaling factor and the bias factor are optimized using a multi-layer perceptron (MLP) model.

9. The method according to any one of claims 1 to 8, characterized in that, The rendering includes: projecting the 3DGS output onto a 2D plane using a differentiable renderer.

10. The method according to any one of claims 1 to 9, characterized in that, The method further includes: The multi-camera image data is used to generate a camera repository; Use the camera repository to generate a global camera index.

11. A processor for rendering 3D objects, characterized in that, The processing is used to perform the method according to any one of claims 1 to 10.

12. One or more non-transitory computer-readable storage media, characterized in that, It includes computer-executable instructions that, when executed, cause one or more processors to perform the method according to any one of claims 1 to 10.

13. A computing system, characterized in that, include: At least one processor; A memory coupled to the at least one processor, the memory storing computer-executable instructions that, when executed by the at least one processor, cause the processor to perform the method according to any one of claims 1 to 10.

14. An apparatus, characterized in that, include: One or more processors; A memory storing instructions that, when executed by the one or more processors, cause the apparatus to perform the method according to any one of claims 1 to 10.

15. A non-transitory computer-readable medium for storing instructions, characterized in that, When executed by a processor, the instructions configure the processor to perform the method according to any one of claims 1 to 10.

16. A device, characterized in that, Used to perform the method according to any one of claims 1 to 10.

17. A processor, characterized in that, Used to execute instructions to cause the device to perform the method according to any one of claims 1 to 10.

18. An integrated circuit, characterized in that, Used to perform the method according to any one of claims 1 to 10.

19. A module, characterized in that, It includes one or more circuits for performing the method according to any one of claims 1 to 10.

20. An apparatus, characterized in that, It includes one or more processors, said one or more processors being functionally connected to memory to perform the method according to any one of claims 1 to 10.

21. An apparatus, characterized in that, Used to perform the method according to any one of claims 1 to 10.

22. A non-transitory computer-readable storage medium, characterized in that, A storage instruction, when executed, causes at least one processing unit, at least one processor, or at least one circuit to perform the method according to any one of claims 1 to 10.

23. A computer program product, characterized in that, The device includes instructions that, when executed by the device, enable the device to perform the method according to any one of claims 1 to 10.

24. A computing system, characterized in that, Includes nodes for performing the method according to any one of claims 1 to 10.

Images