Image processing device, image processing method, and image processing program

Abstract

The present invention generates a two-dimensional image of an arbitrary viewpoint by dividing a three-dimensional primitive into a plurality of chunks set by a predetermined algorithm. An image processing device, wherein at least one processor acquires a plurality of three-dimensional primitives arranged in a three-dimensional space, associates the plurality of three-dimensional primitives with a plurality of chunks set in the three-dimensional space by a predetermined algorithm, sets a viewpoint in the three-dimensional space, and draws, for each chunk, a three-dimensional primitive associated with the chunk on the basis of the viewpoint.

Description

Image processing device, image processing method, and image processing program

[0001] The present disclosure relates to an image processing device, an image processing method, and an image processing program.

[0002] A three-dimensional Gaussian splatting technique is known in which a two-dimensional image is generated by projecting a three-dimensional Gaussian (an example of a three-dimensional primitive) generated from a captured image onto a two-dimensional image plane based on the capturing viewpoint, and the Gaussian parameters are adjusted so that the generated two-dimensional image matches the original captured image.

[0003] According to this technology, a two-dimensional image of an arbitrary viewpoint can be generated by using an adjusted three-dimensional Gaussian whose Gaussian parameters have been adjusted.

[0004] Here, when generating a two-dimensional image from an arbitrary viewpoint using the adjusted three-dimensional Gaussians, there may be an upper limit on the number of three-dimensional Gaussians that can be processed at once. In such a case, in order to prevent the number of three-dimensional Gaussians from exceeding the upper limit, a large number of three-dimensional Gaussians may be divided into multiple chunks, and the three-dimensional Gaussians may be processed for each chunk.

[0005] Bernhard Kerbl, Georgios Kopanas, Thomas Limkuhler, George Drettakis, “3D Gaussian Splatting for Real-Time Radiance Field Rendering”, arXiv:2308.04079 [cs.GR], 8 Aug 2023, [Retrieved 2024.7.17], Internet, <URL:https: / / arxiv.org / abs / 2308.04079>

[0006] The present disclosure provides a technique that can be used to divide a three-dimensional primitive into multiple chunks and generate a two-dimensional image from an arbitrary viewpoint.

[0007] An image processing device according to one aspect of the present disclosure has, for example, the following configuration: That is, the image processing device has at least one memory and at least one processor, wherein the at least one processor acquires a plurality of three-dimensional primitives arranged in a three-dimensional space, links the plurality of three-dimensional primitives to a plurality of chunks set in the three-dimensional space, sets a viewpoint in the three-dimensional space, and renders, for each chunk, each of the three-dimensional primitives linked to a chunk based on the viewpoint, wherein the linking of the three-dimensional primitives to the chunks includes linking a three-dimensional primitive spanning two or more chunks to each of the two or more chunks, and the rendering of the three-dimensional primitives for each chunk based on the viewpoint includes identifying, in each of the two or more chunks, a portion of the two or more chunks that corresponds to the interior of the chunk of the spanning three-dimensional primitive, and rendering the identified portion.

[0008] FIG. 1A is a first diagram illustrating an example of the system configuration of an image processing system. FIG. 1B is a second diagram illustrating an example of the system configuration of an image processing system. FIG. 1C is a third diagram illustrating an example of the system configuration of an image processing system. FIG. 2 is a diagram illustrating an example of the hardware configuration of an image processing device. FIG. 3 is an example of a flowchart illustrating the flow of image processing by the image processing system. FIG. 4 is a diagram illustrating an example of the functional configuration of a Gaussian generation unit. FIG. 5 is a diagram illustrating a specific example of processing by a Gaussian adjustment unit. FIG. 6 is a diagram illustrating a specific example of processing by a drawing unit. FIG. 7 is a diagram illustrating an example of the functional configuration of an image processing unit. FIG. 8 is a diagram for explaining an overview of linking processing. FIG. 9 is an example of a flowchart illustrating the flow of linking processing. FIG. 10 is a diagram for explaining an overview of sorting processing. FIG. 11 is an example of a flowchart illustrating the flow of sorting processing. FIG. 12 is an example of a flowchart illustrating the flow of arbitrary viewpoint 2D image generation processing. FIG. 13 is another example of a flowchart illustrating the flow of image processing by the image processing system. FIG. 14 is a first diagram for explaining an overview of division and linking processing. Fig. 15 is a second diagram for explaining the outline of the division and linking process, and Fig. 16 is a flowchart showing an example of the flow of the division and linking process.

[0009] Hereinafter, each embodiment will be described with reference to the accompanying drawings. In this specification and drawings, components having substantially the same functional configurations are designated by the same reference numerals, and redundant description will be omitted.

[0010] First Embodiment <System Configuration of Image Processing System> First, three system configurations will be described as system configurations of an image processing system including an image processing apparatus according to a first embodiment.

[0011] (1) System Configuration 1 of Image Processing System Fig. 1A is a first diagram showing an example of the system configuration of an image processing system. As shown in Fig. 1A, the image processing system 100A includes an image capturing device 110, an information processing device 120A, an image processing device 130A, and a server device 140A.

[0012] The image capturing device 110 captures an image from each of a plurality of image capturing viewpoints. The plurality of image capturing images captured from the plurality of image capturing viewpoints are transmitted to the information processing device 120A. As an example, the image capturing device 110 may transmit the plurality of image capturing images captured from the plurality of image capturing viewpoints to the information processing device 120A.

[0013] The information processing device 120A transmits multiple captured images captured by the image capture device 110 to the server device 140A via the network 150 and requests the server device 140A to generate adjusted three-dimensional primitives. A three-dimensional primitive is a basic element for representing a luminance field in three-dimensional space and is a distribution that represents a predetermined geometric shape. In this embodiment, a three-dimensional Gaussian corresponding to an ellipsoidal shape is used as the three-dimensional primitive. However, the three-dimensional primitive used is not limited to a shape represented by a distribution, such as a three-dimensional Gaussian, a two-dimensional Gaussian corresponding to a disk shape, or a distribution corresponding to a polygonal shape, as long as it is a primitive to which differentiable rendering can be applied. The three-dimensional Gaussian has the following Gaussian parameters: Position, Covariance, Color, and Alpha (opacity). A three-dimensional Gaussian with the above Gaussian parameters appropriately adjusted is referred to as an "adjusted three-dimensional Gaussian." Therefore, in the following description, the adjusted three-dimensional primitive will be referred to as an adjusted three-dimensional Gaussian. Depending on the context, the three-dimensional Gaussian may be referred to as a three-dimensional Gaussian whose Gaussian parameters are being adjusted or a three-dimensional Gaussian whose Gaussian parameters have been adjusted.

[0014] Among the Gaussian parameters, Covariance is expressed to include Rotation, which indicates the rotation of the three-dimensional Gaussian, and Scale, which indicates the size of the three-dimensional Gaussian in the x-axis, y-axis, and z-axis directions. However, the method of expressing Covariance is not limited to this, and Covariance may be expressed explicitly using Rotation and Scale. Covariance represents the shape (size (scale) and orientation (angle)) of the three-dimensional Gaussian. When a three-dimensional primitive other than a three-dimensional Gaussian is adopted, a parameter representing the shape of the three-dimensional primitive may be used instead of Covariance.

[0015] Furthermore, among the Gaussian parameters, Color may be expressed to include a diffuse reflection component corresponding to a color that does not change depending on the viewing direction, a specular reflection component corresponding to a color that changes depending on the viewing direction, etc. For example, if Color represents a color that includes a diffuse reflection component and a specular reflection component of a three-dimensional Gaussian, and if a spherical harmonic function, for example, is used to express the color of the specular reflection component, Color may be expressed using: RGB (red, green, blue) component colors; and Coefficients of the spherical harmonic function. Even when a three-dimensional primitive other than a three-dimensional Gaussian is used, parameters that include a diffuse reflection component, a specular reflection component, etc. may be used.

[0016] A Gaussian generation program is installed in the server device 140A, and the server device 140A functions as a Gaussian generation unit 141 by executing the program.

[0017] The Gaussian generator 141 receives multiple captured images from the information processing device 120A and, upon receiving a request to generate adjusted 3D Gaussians, generates multiple 3D Gaussians based on the received multiple captured images. The Gaussian generator 141 then projects the generated 3D Gaussians onto a 2D image plane based on the capture viewpoint to generate a 2D image. The Gaussian generator 141 also adjusts the Gaussian parameters so that the generated 2D image approximates the original captured image. The 3D Gaussians are drawn and the Gaussian parameters are adjusted using an appropriate method. For example, the method disclosed in Non-Patent Document 1 may be used, and the same applies to the following description. In this way, the Gaussian generator 141 generates multiple adjusted 3D Gaussians that can generate a 2D image from any viewpoint for the scene in the original captured image.

[0018] The adjusted three-dimensional Gaussians generated by the Gaussian generator 141 are transmitted via the network 150 to the image processing device 130A.

[0019] An image processing program is installed in the image processing device 130A, and the image processing device 130A functions as an image processing unit 131 by executing the program.

[0020] The image processing unit 131 acquires the plurality of adjusted 3D Gaussians transmitted from the server device 140A. The image processing unit 131 acquires an arbitrary viewpoint designated by the user and generates an arbitrary viewpoint 2D image of the scene in the original captured image by using the acquired plurality of adjusted 3D Gaussians. This allows the image processing unit 131 to display the generated arbitrary viewpoint 2D image to the user.

[0021] (2) System Configuration 2 of Image Processing System Fig. 1B is a second diagram showing an example of the system configuration of an image processing system. As shown in Fig. 1B, an image processing system 100B includes an image capturing device 110, an information processing device 120A, and a server device 140B.

[0022] The image capturing device 110 captures an image from each of a plurality of image capturing viewpoints. The plurality of image capturing images captured from the plurality of image capturing viewpoints are transmitted to the information processing device 120A. As an example, the image capturing device 110 may transmit the plurality of image capturing images captured from the plurality of image capturing viewpoints to the information processing device 120A.

[0023] The information processing device 120A transmits a plurality of images captured by the image capturing device 110 to the server device 140B via the network 150, and requests the server device 140B to generate an adjusted three-dimensional Gaussian.

[0024] Furthermore, the information processing device 120A acquires an arbitrary viewpoint specified by the user, transmits the acquired arbitrary viewpoint to the server device 140B via the network 150, and requests the server device 140B to generate a two-dimensional image of the arbitrary viewpoint. In response to the request to generate a two-dimensional image of the arbitrary viewpoint, the information processing device 120A receives the arbitrary viewpoint two-dimensional image from the server device 140B, and displays the received arbitrary viewpoint two-dimensional image to the user.

[0025] A Gaussian generation program and an image processing program are installed in the server device 140B, and by executing these programs, the server device 140B functions as a Gaussian generation unit 141 and an image processing unit 142.

[0026] The Gaussian generator 141 receives multiple captured images from the information processing device 120A and, upon receiving a request to generate adjusted 3D Gaussians, generates multiple 3D Gaussians based on the received multiple captured images. The Gaussian generator 141 then projects the generated multiple 3D Gaussians onto a 2D image plane based on the capturing viewpoint to generate a 2D image. The Gaussian generator 141 also adjusts each Gaussian parameter so that the generated 2D image approximates the original captured image. In this way, the Gaussian generator 141 generates multiple adjusted 3D Gaussians that can generate a 2D image from any viewpoint for the scene in the original captured image. The multiple adjusted 3D Gaussians generated by the Gaussian generator 141 are notified to the image processing unit 142.

[0027] The image processing unit 142 acquires the plurality of adjusted 3D Gaussians notified by the Gaussian generation unit 141. When the image processing unit 142 receives an arbitrary viewpoint from the information processing device 120A and a request to generate a 2D image from an arbitrary viewpoint, the image processing unit 142 generates a 2D image from an arbitrary viewpoint for the scene in the original captured image by using the plurality of adjusted 3D Gaussians acquired. That is, in the example of FIG. 1B, the server device 140B also plays a role similar to that of the image processing device 130A in FIG. 1A. The image processing unit 142 transmits the generated arbitrary viewpoint 2D image to the information processing device 120A via the network 150.

[0028] (3) System Configuration 3 of Image Processing System Fig. 1C is a third diagram showing an example of the system configuration of an image processing system. As shown in Fig. 1C, an image processing system 100C includes an image capturing device 110 and an image processing device 130C.

[0029] The image capturing device 110 captures an image from each of a plurality of image capturing viewpoints. The plurality of image capturing images captured from the plurality of image capturing viewpoints are transmitted to the image processing device 130C. As an example, the image capturing device 110 may transmit the plurality of image capturing images captured from the plurality of image capturing viewpoints to the image processing device 130C.

[0030] A Gaussian generation program and an image processing program are installed in the image processing device 130C, and by executing these programs, the image processing device 130C functions as a Gaussian generation unit 141 and an image processing unit 142.

[0031] After receiving or upon receiving a plurality of captured images captured by the image capture device 110, the Gaussian generator 141 generates a plurality of 3D Gaussians based on the received captured images when a user inputs a request to generate adjusted 3D Gaussians. The Gaussian generator 141 generates a 2D image by projecting the generated 3D Gaussians onto a 2D image plane based on the capture viewpoint. The Gaussian generator 141 also adjusts each Gaussian parameter so that the generated 2D image approaches the original captured image. In this way, the Gaussian generator 141 generates a plurality of adjusted 3D Gaussians that can generate a 2D image from any viewpoint for the scene of the original captured image. The plurality of adjusted 3D Gaussians generated by the Gaussian generator 141 are notified to the image processor 142.

[0032] The image processing unit 142 acquires the plurality of adjusted 3D Gaussians notified by the Gaussian generation unit 141. The image processing unit 142 acquires an arbitrary viewpoint designated by the user, and generates a 2D image of the acquired arbitrary viewpoint for the scene of the original captured image by using the acquired plurality of adjusted 3D Gaussians. The image processing unit 142 displays the generated arbitrary viewpoint 2D image to the user.

[0033] <Hardware configuration of image processing device and server device> Next, a description will be given of the hardware configuration of the image processing device 130A or 130C and the server device 140A or 140B. Note that the image processing device 130A, the image processing device 130C, the server device 140A, and the server device 140C all have similar hardware configurations, so here, the hardware configuration of the image processing device 130A will be described.

[0034] Fig. 2 is a diagram showing an example of the hardware configuration of an image processing device. The image processing device 130A has, as its components, a processor 201, a main storage device 202 (memory), an auxiliary storage device 203 (memory), a network interface 204, and a device interface 205. The image processing device 130A may be realized as a computer in which these components are connected via a bus 206. Note that, in the example of Fig. 2, the image processing device 130A is shown as having one of each component, but the image processing device 130A may also have multiple of the same component.

[0035] Various computations of the image processing device 130A may be executed in parallel using one or more processors. Furthermore, various computations may be distributed to multiple computing cores within the processor 201 and executed in parallel. Furthermore, some or all of the processes, means, etc. disclosed herein may be executed by an external device 230 (at least one of a processor and a storage device) provided on a cloud that can communicate with the image processing device 130A via the network interface 204.

[0036] The processor 201 may be an electronic circuit (processing circuit, processing circuitry, CPU, GPU, FPGA, ASIC, etc.). The processor 201 may also be a semiconductor device including a dedicated processing circuit. The processor 201 is not limited to an electronic circuit using electronic logic elements, and may be realized by an optical circuit using optical logic elements. The processor 201 may also include an arithmetic function based on quantum computing.

[0037] The processor 201 performs various calculations based on various data and commands input from each device, etc., in the internal configuration of the image processing device 130A, and outputs the calculation results and control signals to each device, etc. The processor 201 controls each component included in the image processing device 130A by executing an OS (Operating System), applications, etc.

[0038] Furthermore, processor 201 may refer to one or more electronic circuits arranged on a single chip, or may refer to one or more electronic circuits arranged on two or more chips or devices. When multiple electronic circuits are used, the respective electronic circuits may communicate with each other via wires or wirelessly.

[0039] The main memory device 202 is a memory device that stores instructions executed by the processor 201 and various data, and the various data stored in the main memory device 202 is read by the processor 201. The auxiliary memory device 203 is a memory device other than the main memory device 202. Note that these memory devices refer to any electronic component that can store various data, and may be semiconductor memory. The semiconductor memory may be either volatile memory or non-volatile memory. The memory device for saving various data in the image processing device 130A may be realized by the main memory device 202 or the auxiliary memory device 203, or may be realized by an internal memory built into the processor 201.

[0040] Furthermore, multiple processors 201 may be connected (coupled) to one main storage device 202, or a single processor 201 may be connected. Alternatively, multiple main storage devices 202 may be connected (coupled) to one processor 201. When the image processing device 130A is configured with at least one main storage device 202 and multiple processors 201 connected (coupled) to this at least one main storage device 202, it may include a configuration in which at least one processor of the multiple processors 201 is connected (coupled) to at least one main storage device 202.

[0041] The network interface 204 is an interface for connecting to a communication network 220 wirelessly or via a wired connection.

[0042] The device interface 205 is an interface such as a USB that directly connects to an external device 240 .

[0043] The external device 240 may be, for example, an input device. In this embodiment, the input device is, for example, an electronic device such as a camera, a microphone, various sensors, a keyboard, a mouse, or a touch panel, and provides acquired information to the image processing device 130A.

[0044] Furthermore, the external device 240 may be, for example, an output device. In this embodiment, the output device may be, for example, a display device such as an LCD (Liquid Crystal Display), a CRT (Cathode Ray Tube), a PDP (Plasma Display Panel), or an organic EL (Electro Luminescence) panel, or may be a speaker that outputs sound or the like.

[0045] The external device 240 may also be a storage device (memory). For example, the external device 240 may be a network storage or the like, or may be a storage such as an HDD.

[0046] Furthermore, the external device 240 may be a device that has some of the functions of the components of the image processing device 130 A. In other words, the image processing device 130 A may transmit and receive processing results to and from the external device 240 .

[0047] <Image Processing Flow by Image Processing System> Next, the image processing flow by the image processing system will be described. Note that while the following description will mainly focus on image processing system 100A (FIG. 1A), the same applies to image processing system 100B (FIG. 1B) and image processing system 100C (FIG. 1C). FIG. 3 is an example of a flowchart showing the image processing flow. The Gaussian generator described here refers to Gaussian generator 141 in FIGS. 1A to 1C. Furthermore, the image processing unit described here refers to image processing units 131 and 142 in FIGS. 1A to 1C.

[0048] In step S301, the Gaussian generator acquires a plurality of captured images captured by the image capturing device 110 from a plurality of viewpoints.

[0049] In step S302, the Gaussian generator generates point cloud data based on the acquired plurality of captured images, and also generates a plurality of three-dimensional Gaussians based on the generated point cloud data.

[0050] In step S303, the Gaussian generator generates a two-dimensional image by projecting the generated three-dimensional Gaussians onto a two-dimensional image plane based on the shooting viewpoint, and adjusts the Gaussian parameters of the generated two-dimensional image so that the generated two-dimensional image approaches the original shot image.

[0051] In step S304, the Gaussian generator determines whether the adjustment of the Gaussian parameters is complete. If it is determined in step S304 that the adjustment of the Gaussian parameters is not complete (NO in step S304), the process returns to step S303. On the other hand, if it is determined in step S304 that the adjustment of the Gaussian parameters is complete (YES in step S304), the process proceeds to step S305.

[0052] In step S305, the Gaussian generator stores a plurality of adjusted 3D Gaussians.

[0053] In step S306, the Gaussian generator reads out the adjusted three-dimensional Gaussians and transmits or notifies them to the image processor.

[0054] In step S307, the image processing unit executes a linking process in which a plurality of chunks are set in the three-dimensional space and a plurality of adjusted three-dimensional Gaussians received from the Gaussian generator are linked to the set chunks. Details of the linking process performed by the image processing device 130A will be described later.

[0055] In step S308, the image processing unit acquires the arbitrary viewpoint input by the user.

[0056] In step S309, the image processing unit executes a sorting process to sort the chunks and the adjusted 3D Gaussians associated with each chunk according to the input arbitrary viewpoint. Details of the sorting process will be described later.

[0057] In step S310, the image processing unit executes an arbitrary viewpoint two-dimensional image generation process to generate a two-dimensional image according to the input arbitrary viewpoint based on the sorted chunks and the sorted adjusted three-dimensional Gaussians. The arbitrary viewpoint two-dimensional image generation process will be described in detail later.

[0058] In step S311, the image processing unit outputs and displays the generated arbitrary viewpoint two-dimensional image.

[0059] In step S312, the image processing unit determines whether to continue the arbitrary viewpoint 2D image generation process. If it is determined to continue in step S312 (YES in step S312), the process returns to step S308. On the other hand, if it is determined not to continue in step S312 (NO in step S312), the image processing ends.

[0060] <Functional Configuration of Gaussian Generator of Server Device> Next, the functional configuration of the Gaussian generator will be described. Fig. 4 is a diagram showing an example of the functional configuration of the Gaussian generator (here, an example of the functional configuration of the Gaussian generator of server device 140A is shown). As shown in Fig. 4, the Gaussian generator 141 of server device 140A includes a captured image acquisition unit 410, a point cloud data generation unit 420, a Gaussian adjustment unit 430, a drawing unit 440, and an adjusted 3D Gaussian transmission unit 450.

[0061] The captured image acquisition unit 410 acquires a plurality of captured images captured from a plurality of viewpoints by the image capturing device 110. The captured image acquisition unit 410 notifies the point cloud data generation unit 420 of the acquired plurality of captured images.

[0062] The point cloud data generator 420 generates point cloud data from multiple captured images using Structure from Motion (SfM). Each point in the point cloud data includes color and coordinate information. SfM is a technology that can generate point cloud data from multiple captured images captured from multiple viewpoints.

[0063] The point cloud data generation unit 420 estimates the shooting viewpoints of each of the multiple captured images. The point cloud data generation unit 420 notifies the Gaussian adjustment unit 430 of SfM data including the generated point cloud data and the multiple estimated shooting viewpoints. The point cloud data generation unit 420 stores the notified multiple captured images and the multiple estimated shooting viewpoints in the captured image storage unit 460.

[0064] The Gaussian adjustment unit 430 generates a plurality of three-dimensional Gaussians based on the SfM data and notifies the rendering unit 440. The Gaussian adjustment unit 430 adjusts the Gaussian parameters of each of the plurality of three-dimensional Gaussians notified to the rendering unit 440 based on the loss calculated by the rendering unit 440.

[0065] The Gaussian adjustment unit 430 repeatedly adjusts the Gaussian parameters and notifies the rendering unit 440 of the adjusted three-dimensional Gaussians until the loss calculated by the rendering unit 440 satisfies a predetermined condition. The Gaussian adjustment unit 430 stores, in the Gaussian storage unit 470, a plurality of three-dimensional Gaussians having Gaussian parameters when the loss calculated by the rendering unit 440 satisfies the predetermined condition, as a plurality of adjusted three-dimensional Gaussians.

[0066] When adjusting the multiple 3D Gaussians, the Gaussian adjustment unit 430 may also adjust the shooting viewpoints of the multiple captured images. The adjustment of the shooting viewpoint includes adjusting the position and orientation of the shooting viewpoint. When adjusting the multiple shooting viewpoints, the Gaussian adjustment unit 430 may update the multiple shooting viewpoints before adjustment by storing the adjusted multiple shooting viewpoints in the captured image storage unit 460 in association with the captured images. In the following description, it is assumed that the shooting viewpoint is also adjusted, but this is not limited to this.

[0067] The rendering unit 440 renders the multiple 3D Gaussians notified from the Gaussian adjustment unit 430 by projecting them onto a 2D image plane based on the shooting viewpoint, thereby generating a 2D image. The rendering unit 440 reads out the captured image associated with the shooting viewpoint from the captured image storage unit 460, and calculates the loss of the generated 2D image relative to the captured image. The rendering unit 440 notifies the Gaussian adjustment unit 430 of the calculated loss.

[0068] It should be noted that if the Gaussian adjustment section 430 has adjusted a plurality of imaging viewpoints, the rendering section 440 generates a two-dimensional image based on the plurality of imaging viewpoints being adjusted.

[0069] The adjusted three-dimensional Gaussian transmitter 450 reads out the adjusted three-dimensional Gaussians stored in the Gaussian storage unit 470 and transmits them to the image processor.

[0070] <Details of Processing by Each Unit of Gaussian Generator> Next, details of processing by the Gaussian adjuster 430 and the rendering unit 440 among the units of the Gaussian generator 141 will be described.

[0071] (1) Details of Processing by the Gaussian Adjustment Unit 430 First, a specific example of processing by the Gaussian adjustment unit 430 will be described. Fig. 5 is a diagram showing a specific example of processing by the Gaussian adjustment unit.

[0072] As shown in FIG. 5, the Gaussian adjustment unit 430 further includes an initial 3D Gaussian generator 510 and an optimizer 520 .

[0073] When the initial 3D Gaussian generator 510 receives SfM data from the point cloud data generator 420, it generates multiple initial 3D Gaussians based on the point cloud data included in the SfM data and notifies the optimizer 520 of these along with multiple shooting viewpoints. The parameters of the initial 3D Gaussians are initialized, for example, as follows: Position is initialized to the position of a point in the point cloud data. Rotation is initialized to zero rotation. Scale values ​​in the x-axis, y-axis, and z-axis directions are initialized to predetermined values. Alpha is initialized to a predetermined value. RGB values ​​are initialized to predetermined values. Coefficients of spherical harmonic functions are initialized to predetermined values.

[0074] The optimization unit 520 notifies the rendering unit 440 of the multiple initial 3D Gaussians notified by the initial 3D Gaussian generation unit 510, along with the multiple shooting viewpoints, and acquires losses from the rendering unit 440. The optimization unit 520 adjusts Gaussian parameters of the multiple initial 3D Gaussians and adjusts the multiple shooting viewpoints based on the acquired losses. The optimization unit 520 notifies the rendering unit 440 of the multiple adjusted 3D Gaussians and the multiple adjusted shooting viewpoints, and acquires losses from the rendering unit 440.

[0075] The optimization unit 520 further adjusts the Gaussian parameters of the plurality of 3D Gaussians after the previous adjustment and further adjusts the plurality of shooting viewpoints after the previous adjustment, based on the acquired losses. The optimization unit 520 notifies the rendering unit 440 of the plurality of 3D Gaussians after the second adjustment and the plurality of shooting viewpoints after the second adjustment, and acquires the losses from the rendering unit 440.

[0076] As shown in FIG. 5 , the optimization unit 520 includes an adaptive control unit 521. If the area of ​​the adjusted three-dimensional Gaussian is small, the adaptive control unit 521 performs a duplication process to enlarge the area of ​​the three-dimensional Gaussian. Furthermore, if the area of ​​the adjusted three-dimensional Gaussian is large, the adaptive control unit 521 performs a division process to reduce the area of ​​the three-dimensional Gaussian. Furthermore, the adaptive control unit 521 determines whether the value of "Alpha," among the Gaussian parameters of the adjusted three-dimensional Gaussian, is equal to or less than a predetermined threshold, and if it is determined that the value is equal to or less than the predetermined threshold, deletes the adjusted three-dimensional Gaussian.

[0077] Note that these processes by the adaptive control unit 521 may be executed, for example, every time a predetermined number of losses are acquired from the drawing unit 440 .

[0078] In this way, the optimization unit 520 notifies the rendering unit 440 of the multiple 3D Gaussians after the nth adjustment (n is an integer equal to or greater than 1) (or the multiple 3D Gaussians after the nth adjustment, which have been processed by the adaptive control unit 521). The optimization unit 520 also notifies the rendering unit 440 of the multiple shooting viewpoints after the nth adjustment.

[0079] The optimization unit 520 repeatedly performs these processes, and when the loss calculated by the rendering unit 440 satisfies a predetermined condition, the optimization unit 520 stores a plurality of adjusted three-dimensional Gaussians in the Gaussian storage unit 470. The optimization unit 520 also stores a plurality of adjusted shooting viewpoints in the captured image storage unit 460.

[0080] (2) Details of Processing by the Drawing Unit 440 Next, a specific example of processing by the drawing unit 440 will be described. Fig. 6 is a diagram showing a specific example of processing by the drawing unit.

[0081] As shown in FIG. 6, the rendering unit 440 further includes a shooting viewpoint two-dimensional image generating unit 601 and a loss calculating unit 602 .

[0082] The shooting viewpoint 2D image generation unit 601 generates multiple shooting viewpoint 2D images when notified by the Gaussian adjustment unit 430 of: multiple initial 3D Gaussians or multiple 3D Gaussians after the nth adjustment; and multiple shooting viewpoints before adjustment or multiple shooting viewpoints after the nth adjustment. In Figure 6, reference numeral 610 indicates how the shooting viewpoint 2D image generation unit 601 generates one shooting viewpoint 2D image based on the multiple 3D Gaussians after the nth adjustment and one shooting viewpoint after the nth adjustment. Specifically, reference numeral 610 indicates how the shooting viewpoint 2D image generation unit 601 generates a shooting viewpoint 2D image 612 by projecting the multiple 3D Gaussians after the nth adjustment onto a 2D image plane 611 based on one shooting viewpoint after the nth adjustment.

[0083] The loss calculation unit 602 reads out a plurality of captured images associated with a plurality of shooting viewpoints before adjustment from the captured image storage unit 460, calculates a loss each time each of the shooting viewpoint two-dimensional images is notified from the shooting viewpoint two-dimensional image generation unit 601, and notifies the Gaussian adjustment unit 430. The loss calculation unit 602 may, for example, calculate a difference between the captured image and the shooting viewpoint two-dimensional image for each shooting viewpoint, and notify the Gaussian adjustment unit 430 of the sum of the differences calculated for the plurality of shooting viewpoints as the loss. In this way, the Gaussian adjustment unit 430 adjusts the Gaussian parameters so that each of the shooting viewpoint two-dimensional images generated by the shooting viewpoint two-dimensional image generation unit 601 approaches each of the plurality of captured images read out from the captured image storage unit 460.

[0084] <Functional Configuration of Image Processing Unit> Next, the functional configuration of the image processing unit will be described. Fig. 7 is a diagram showing an example of the functional configuration of the image processing unit (here, an example of the functional configuration of the image processing unit of the image processing device 130A is shown). As shown in Fig. 7, the image processing unit 131 of the image processing device 130A has an adjusted 3D Gaussian receiving unit 710, an adjusted 3D Gaussian registering unit 720, a viewpoint accepting unit 730, a chunk sorting unit 740, a Gaussian sorting unit 750, a drawing unit 760, and a display unit 770.

[0085] Adjusted three-dimensional Gaussian receiver 710 receives a plurality of adjusted three-dimensional Gaussians from server device 140A via network 150. Adjusted three-dimensional Gaussian receiver 710 notifies adjusted three-dimensional Gaussian registration unit 720 of the received plurality of adjusted three-dimensional Gaussians.

[0086] The adjusted three-dimensional Gaussian registration unit 720 sets a plurality of chunks in three-dimensional space. A chunk refers to a classification destination set in three-dimensional space to classify a plurality of adjusted three-dimensional Gaussians arranged in three-dimensional space according to their positions in the three-dimensional space.

[0087] The adjusted 3D Gaussian registration unit 720 links multiple adjusted 3D Gaussians to one or more chunks based on the positions and shapes of the adjusted 3D Gaussians and the positions and shapes of the chunks. Here, "linking based on the positions and shapes of the 3D Gaussians and the positions and shapes of the chunks" includes, for example, linking 3D Gaussians based on the positional relationship between the 3D Gaussians and the representative points of the chunks, or linking 3D Gaussians that have portions that overlap (intersect) with the regions of the chunks. Note that if one adjusted 3D Gaussian spans multiple chunks, the single adjusted 3D Gaussian will be linked to multiple chunks.

[0088] The adjusted 3D Gaussian registration unit 720 notifies the chunk sort unit 740 of chunk data including the position and shape of the chunk in 3D space and the adjusted 3D Gaussian associated with the chunk.

[0089] The viewpoint receiving unit 730 receives an arbitrary viewpoint input by the user and notifies the chunk sorting unit 740 and Gaussian sorting unit 750 of the viewpoint.

[0090] The chunk sorting unit 740 sorts a plurality of chunks set in a three-dimensional space, for example, in order of distance from the arbitrary viewpoint, based on the arbitrary viewpoint notified by the viewpoint receiving unit 730. The chunk sorting unit 740 notifies the rendering unit 760 of the order of the sorted chunks.

[0091] The Gaussian sorting unit 750 sorts the multiple adjusted 3D Gaussians associated with each chunk, for example, in descending order of distance from the arbitrary viewpoint, based on the arbitrary viewpoint notified by the viewpoint receiving unit 730. The Gaussian sorting unit 750 notifies the rendering unit 760 of the order of the sorted adjusted 3D Gaussians.

[0092] The rendering unit 760 renders the plurality of adjusted 3D Gaussians by projecting them onto a 2D image plane based on an arbitrary viewpoint, thereby generating an arbitrary viewpoint 2D image. When rendering the plurality of adjusted 3D Gaussians onto the 2D image plane based on an arbitrary viewpoint, the rendering unit 760 renders them in accordance with: the order of chunks notified by the chunk sorting unit 740; and the order of the plurality of adjusted 3D Gaussians notified by the Gaussian sorting unit 750.

[0093] For example, the rendering unit 760 renders the adjusted 3D Gaussians by projecting them onto the 2D image plane in order, starting with the adjusted 3D Gaussians associated with the first chunk. If there are multiple adjusted 3D Gaussians associated with the first chunk, the rendering unit 760 renders the adjusted 3D Gaussians by projecting them onto the 2D image plane in the notified order.

[0094] Furthermore, suppose that the adjusted 3D Gaussians associated with the first chunk include an adjusted 3D Gaussian that spans multiple chunks. An adjusted 3D Gaussian that spans multiple chunks is referred to as a specific adjusted 3D Gaussian. Specifically, suppose that a portion of the specific adjusted 3D Gaussian is included within the first chunk, while the remaining portion is included within the second chunk. In this case, when rendering the multiple adjusted 3D Gaussians associated with the first chunk, the rendering unit 760 renders the portion of the specific adjusted 3D Gaussian that is included within the first chunk. Then, the rendering unit 760 renders the remaining portion of the specific adjusted 3D Gaussian when rendering the multiple adjusted 3D Gaussians included in the second chunk.

[0095] That is, the rendering unit 760 divides an adjusted three-dimensional Gaussian spanning multiple chunks into parts contained within each chunk and renders them at different times.

[0096] The display unit 770 controls a display device, which is an example of the external device 240 , to display the arbitrary viewpoint two-dimensional image generated by the drawing unit 760 .

[0097] In the above description, the adjusted 3D Gaussian registration unit 720 is configured to set chunks regardless of the arbitrary viewpoint accepted by the viewpoint accepting unit 730. However, the method of setting chunks is not limited to this. For example, the adjusted 3D Gaussian registration unit 720 may change the number, size, and shape of the chunks to be set depending on the arbitrary viewpoint accepted by the viewpoint accepting unit 730.

[0098] <Details of Processing by Each Unit of Image Processing Unit> Next, details of processing by the adjusted 3D Gaussian registration unit 720, chunk sort unit 740, Gaussian sort unit 750, and drawing unit 760 of the units of the image processing unit 131 will be described.

[0099] (1) Details of the linking process by the adjusted three-dimensional Gaussian registration unit First, details of the linking process (step S307 in FIG. 3) by the adjusted three-dimensional Gaussian registration unit 720 will be described. Here, an overview of the linking process will be first described. FIG. 8 is a diagram for explaining the overview of the linking process.

[0100] In Fig. 8 , reference numerals 810 and 820 denote examples of chunks set by the adjusted three-dimensional Gaussian registration unit 720. In the example of Fig. 8 , adjusted three-dimensional Gaussians denoted by reference numerals 811 and 812 are arranged as adjusted three-dimensional Gaussians close to the representative point (center position) of chunk 810. In the example of Fig. 8 , adjusted three-dimensional Gaussians denoted by reference numerals 821 and 822 are arranged as adjusted three-dimensional Gaussians close to the representative point (center position) of chunk 820.

[0101] Suppose that adjusted three-dimensional Gaussians 811 and 812 are linked to chunk 810, and adjusted three-dimensional Gaussians 821 and 822 are linked to chunk 820.

[0102] Here, chunk 810 is farther away from the arbitrary viewpoint than chunk 820. For this reason, adjusted three-dimensional Gaussians 811 and 812 linked to chunk 810 are rendered before adjusted three-dimensional Gaussians 821 and 822 linked to chunk 820. Furthermore, here, if adjusted three-dimensional Gaussian 811 is farther away from the arbitrary viewpoint than adjusted three-dimensional Gaussian 812 and adjusted three-dimensional Gaussian 822 is farther away from the arbitrary viewpoint than adjusted three-dimensional Gaussian 821, the adjusted three-dimensional Gaussians may be rendered in the order of symbols 811, 812, 822, and 821.

[0103] On the other hand, in the example of Figure 8, when viewed from an arbitrary viewpoint, a portion of the adjusted three-dimensional Gaussian 812 overlaps with a portion of the adjusted three-dimensional Gaussian 822, and the portion of the adjusted three-dimensional Gaussian 812 is located in front.

[0104] Therefore, when the adjusted three-dimensional Gaussians are drawn in the order of reference numerals 811, 812, 822, and 821, the adjusted three-dimensional Gaussian located in the foreground is drawn first, and the image quality of the arbitrary viewpoint two-dimensional image deteriorates.

[0105] In this way, when an adjusted 3D Gaussian is placed across multiple chunks, image quality deteriorates in boundary regions. In consideration of this, the adjusted 3D Gaussian registration unit 720 links the specific adjusted 3D Gaussian to each chunk. Then, when sorting and drawing the specific adjusted 3D Gaussians linked to each chunk, sorting is performed on a Gaussian-by-Gaussian basis, and the corresponding part inside the chunk is drawn.

[0106] In the example of FIG. 8 , reference numeral 812′ denotes a portion of the adjusted 3D Gaussian 812 spanning two chunks 810, 820, the portion corresponding to the interior of chunk 810, which is drawn when each adjusted 3D Gaussian associated with chunk 810 is drawn. Also, in the example of FIG. 8 , reference numeral 812″ denotes the remaining portion of the adjusted 3D Gaussian 812, the portion corresponding to the interior of chunk 820, which is drawn when each adjusted 3D Gaussian associated with chunk 820 is drawn.

[0107] The adjusted 3D Gaussian registration unit 720 performs the linking, and the Gaussian sort unit 750 and the rendering unit 760 perform the sorting and rendering, thereby making it possible to avoid a situation where an adjusted 3D Gaussian located in the foreground is rendered first, thereby improving the image quality of the generated arbitrary viewpoint 2D image.

[0108] Next, a description will be given of the flow of the linking process (step S307 in FIG. 3) performed by the adjusted three-dimensional Gaussian registration unit 720. Fig. 9 is an example of a flowchart showing the flow of the linking process.

[0109] In step S901, the adjusted three-dimensional Gaussian registration unit 720 sets a plurality of chunks that do not overlap each other in three-dimensional space.

[0110] In step S902, the adjusted three-dimensional Gaussian registration unit 720 associates the multiple adjusted three-dimensional Gaussians received from the server device 140A with one or multiple chunks.

[0111] In step S903, the adjusted three-dimensional Gaussian registration unit 720 determines whether there are any chunks linked by more than a predetermined number of adjusted three-dimensional Gaussians. If it is determined in step S903 that there are any chunks linked by more than a predetermined number of adjusted three-dimensional Gaussians (YES in step S903), the process returns to step S901, and a new chunk is set.

[0112] On the other hand, if it is determined in step S903 that there are no chunks linked by more than the predetermined number of adjusted three-dimensional Gaussians (NO in step S903), the linking process ends.

[0113] (2) Details of the sorting process by the chunk sort unit and Gaussian sort unit Next, the details of the sorting process (step S309 in FIG. 3) by the chunk sort unit 740 and Gaussian sort unit 750 will be described. Here, an overview of the sorting process will be first described. FIG. 10 is a diagram for explaining the overview of the sorting process.

[0114] In Figure 10, reference numeral 1010 denotes a view frustum that represents the range to be rendered as a two-dimensional image when viewed from an arbitrary viewpoint. The chunk sorting unit 740 identifies chunks that intersect with the view frustum indicated by reference numeral 1010 from among the chunks set by the adjusted 3D Gaussian registration unit 720. Note that subsequent processing is terminated for chunks not identified by the chunk sorting unit 740 (i.e., chunks that do not intersect with the view frustum indicated by reference numeral 1010). In this way, by performing view frustum culling (or simply referred to as culling), it is possible to speed up the generation of an arbitrary viewpoint two-dimensional image.

[0115] The chunk sort unit 740 calculates, for example, the distance of each identified chunk from an arbitrary viewpoint. The distance of each identified chunk from an arbitrary viewpoint refers to the distance between the representative point (center position) of each identified chunk and the arbitrary viewpoint.

[0116] The chunk sorting unit 740 sorts the identified chunks, for example, in ascending order of distance from the arbitrary viewpoint.

[0117] The Gaussian sorting unit 750 sorts the adjusted 3D Gaussians associated with each chunk sorted in ascending order of distance, for example, in descending order of distance from an arbitrary viewpoint. Note that if a specific adjusted 3D Gaussian is associated with a chunk, the chunk sorting unit 740 sorts the corresponding portion of the specific adjusted 3D Gaussian within the chunk, for example, in descending order of distance from an arbitrary viewpoint.

[0118] Next, a description will be given of the flow of the sorting process (step S309 in FIG. 3) performed by the chunk sort unit 740 and the Gaussian sort unit 750. Fig. 11 is an example of a flowchart showing the flow of the sorting process.

[0119] In step S1101, the chunk sort unit 740 identifies one or more chunks that intersect with a view frustum identified by an arbitrary viewpoint.

[0120] In step S1102, the chunk sorting unit 740 sorts the identified one or more chunks based on, for example, the distance from an arbitrary viewpoint.

[0121] In step S1103, the chunk sort unit 740 assigns "1" to counter i, which indicates the order of the sorted chunks.

[0122] In step S1104, the Gaussian sorting unit 750 sorts each adjusted 3D Gaussian associated with the i-th chunk, for example, based on the distance from an arbitrary viewpoint. Here, the Gaussian sorting unit 750 may sort each adjusted 3D Gaussian associated with the i-th chunk by Gaussian or by corresponding subunit within the chunk.

[0123] In step S1105, the chunk sorting unit 740 determines whether or not the Gaussian sorting unit 750 has completed sorting of each adjusted three-dimensional Gaussian for all chunks.

[0124] If it is determined in step S1105 that there are chunks for which sorting has not been completed (NO in step S1105), the process proceeds to step S1106.

[0125] In step S106, the chunk sort unit 740 increments the counter i, and then returns to step S1104.

[0126] On the other hand, if it is determined in step S1105 that sorting has been completed for all chunks (YES in step S1105), the sorting process ends.

[0127] (3) Details of Arbitrary Viewpoint 2D Image Generation Processing by the Rendering Unit Next, we will explain the details of the arbitrary viewpoint 2D image generation processing (step S310 in FIG. 3) by the rendering unit 760. Fig. 12 is an example of a flowchart showing the flow of the arbitrary viewpoint 2D image generation processing.

[0128] In step S1201, the drawing unit 760 assigns "1" to a counter i that indicates the order of the chunk.

[0129] In step S1202, the drawing unit 760 reads the i-th chunk.

[0130] In step S1203, the rendering unit 760 reads each adjusted three-dimensional Gaussian associated with the i-th chunk.

[0131] In step S1204, the drawing unit 760 assigns "1" to a counter j that indicates the order of the adjusted three-dimensional Gaussian within the i-th chunk.

[0132] In step S1205, the rendering unit 760 renders the jth adjusted 3D Gaussian or the corresponding portion inside the jth specific chunk of the adjusted 3D Gaussian according to an arbitrary viewpoint.

[0133] In step S1206, it is determined whether or not the rendering of all adjusted 3D Gaussians associated with the i-th chunk has been completed. In step S1206, if there are adjusted 3D Gaussians for which rendering has not been completed (NO in step S1206), the process proceeds to step S1207.

[0134] In step S1207, the drawing unit 760 increments the counter j, and the process returns to step S1205.

[0135] On the other hand, if all drawing is completed in step S1206 (YES in step S1206), the process proceeds to step S1208.

[0136] In step S1208, the drawing unit 760 determines whether all chunks have been read. If it is determined in step S1208 that there are chunks that have not been read (NO in step S1208), the process proceeds to step S1209.

[0137] In step S1209, the drawing unit 760 increments the counter i, and then the process returns to step S1202.

[0138] On the other hand, if it is determined in step S1208 that all chunks have been read (YES in step S1208), the arbitrary viewpoint two-dimensional image generation process ends.

[0139] <Summary> As is clear from the above description, the image processing device 130A according to the first embodiment: acquires a plurality of adjusted 3D Gaussians arranged in a three-dimensional space; links the acquired plurality of adjusted 3D Gaussians to a plurality of chunks set in the three-dimensional space; and links a specific adjusted 3D Gaussian arranged across a plurality of chunks (two or more chunks) to each chunk; sets an arbitrary viewpoint in the three-dimensional space; and draws, for each chunk, each adjusted 3D Gaussian linked to the chunk based on the arbitrary viewpoint. In this case, for an adjusted 3D Gaussian spanning two or more chunks, identifies a portion corresponding to the interior of the chunk, and draws the identified corresponding portion.

[0140] In this way, when drawing multiple adjusted three-dimensional Gaussians for each chunk, if they are placed across a first chunk and a second chunk, the adjusted three-dimensional Gaussians are drawn separately in the first chunk and the second chunk.

[0141] As a result, according to the first embodiment, when a plurality of adjusted 3D Gaussians are drawn for each chunk, the plurality of adjusted 3D Gaussians can be drawn in an appropriate order. As a result, according to the first embodiment, it is possible to suppress degradation in image quality when generating a 2D image from an arbitrary viewpoint from a plurality of adjusted 3D Gaussians.

[0142] That is, according to the first embodiment, it is possible to provide a technique that can be used when dividing an adjusted three-dimensional Gaussian into a plurality of chunks and generating a two-dimensional image from an arbitrary viewpoint.

[0143] [Second Embodiment] In the first embodiment, the adjusted 3D Gaussian registration unit 720 is configured to directly set multiple chunks in the 3D space. However, the method for setting multiple chunks is not limited to this. For example, the adjusted 3D Gaussian registration unit 720 may first divide the 3D space and assign multiple chunks to each divided space. Here, "dividing the 3D space" refers to classifying all coordinates in the 3D space based on some criteria. For example, when Voronoi tessellating the 3D space, this refers to classifying each coordinate in the 3D space into the closest kernel point to multiple kernel points set in the 3D space, and then determining a range consisting of the set of coordinates classified into the same kernel point. The second embodiment will be described below, focusing on the differences from the first embodiment.

[0144] <Flow of Image Processing by Image Processing System> First, the flow of image processing by the image processing system 100A having the image processing device 130A according to the second embodiment will be described. Fig. 13 is an example of another flowchart showing the flow of image processing. The difference from the flowchart shown in Fig. 3 is that the flowchart in Fig. 13 does not include step S307 and a new step S1301 has been added.

[0145] In step S1301, the image processing device 130A divides a three-dimensional space in which the adjusted three-dimensional Gaussians are to be placed to generate a plurality of divided spaces. The image processing device 130A also places the adjusted three-dimensional Gaussians in each divided space and sets chunks. As a result, the adjusted three-dimensional Gaussians belonging to each divided space or specific adjusted three-dimensional Gaussians are linked to the chunks set in each divided space.

[0146] <Details of Division and Linking Process by Adjusted 3D Gaussian Registration Unit> Next, details of the division and linking process (step S1301 in FIG. 13 ) by the adjusted 3D Gaussian registration unit 720 of the image processing unit 131 of the image processing device 130A will be described. Here, an overview of the division and linking process will first be described. FIGS. 14 and 15 are first and second diagrams for explaining the overview of the division and linking process.

[0147] 14 , reference numeral 1410 denotes a three-dimensional space in which the adjusted three-dimensional Gaussian is placed. In the second embodiment, the adjusted three-dimensional Gaussian registration unit 720 generates a plurality of divided spaces by: dividing the three-dimensional space indicated by reference numeral 1410 so that each divided space has a convex shape; dividing the divided spaces so that the divided spaces do not overlap each other; and dividing the three-dimensional space so that the boundary surface between two adjacent chunks forms part of the perpendicular bisector plane of the center points of the two chunks. Examples of the division method of the adjusted three-dimensional Gaussian registration unit 720 (a division method that satisfies the above three conditions) include Voronoi division and grid division.

[0148] In FIG. 14, reference numeral 1420 denotes an example of a divided space, and for convenience, a three-dimensional space is shown divided into a plurality of rectangular parallelepipeds.

[0149] 15 shows a state in which a plurality of adjusted three-dimensional Gaussians are arranged in each divided space. Reference numeral 1510 denotes an adjusted three-dimensional Gaussian arranged in the divided space indicated by reference numeral 1420.

[0150] As shown in FIG. 15, among the adjusted three-dimensional Gaussians arranged in each divided space, there is a “specific adjusted three-dimensional Gaussian” whose part belongs to one divided space and whose remaining part belongs to the other divided space.

[0151] Such a particular adjusted 3D Gaussian will be attached to both chunks.

[0152] Next, a detailed description will be given of the division and linking process (step S1301 in FIG. 13) performed by the adjusted three-dimensional Gaussian registration unit 720. Fig. 16 is a flowchart showing an example of the flow of the division and linking process.

[0153] In step S1601, the adjusted three-dimensional Gaussian registration unit 720 divides the three-dimensional space in which the adjusted three-dimensional Gaussian is placed, to generate a plurality of divided spaces.

[0154] In step S1602, adjusted 3D Gaussian registration unit 720 places each adjusted 3D Gaussian received from server device 140A in each divided space, thereby identifying an adjusted 3D Gaussian belonging to each divided space or a specific adjusted 3D Gaussian.

[0155] In step S1603, the adjusted three-dimensional Gaussian registration unit 720 determines whether there is a partition to which more than a predetermined number of adjusted three-dimensional Gaussians or a specific adjusted three-dimensional Gaussian belongs. If there is a partition to which more than a predetermined number of adjusted three-dimensional Gaussians or a specific adjusted three-dimensional Gaussian belongs (YES in step S1603), the process returns to step S1601. In this case, the adjusted three-dimensional Gaussian registration unit 720 generates a new partition.

[0156] On the other hand, in step S1603, if there is no divided space to which more than the predetermined number of adjusted three-dimensional Gaussians or a specific adjusted three-dimensional Gaussian belongs (NO in step S1603), the process proceeds to step S1604.

[0157] In step S1604, the adjusted three-dimensional Gaussian registration unit 720 sets chunks for each divided space, and associates the adjusted three-dimensional Gaussians belonging to each divided space or a specific adjusted three-dimensional Gaussian with each set chunk.

[0158] <Summary> As is clear from the above description, the image processing device 130A according to the second embodiment: acquires a plurality of adjusted 3D Gaussians to be arranged in a three-dimensional space; divides the three-dimensional space in which the acquired plurality of adjusted 3D Gaussians are to be arranged, to generate a plurality of divided spaces; arranges the acquired plurality of adjusted 3D Gaussians in each of the generated divided spaces, and identifies the adjusted 3D Gaussians or a specific adjusted 3D Gaussian belonging to each divided space; sets a chunk in each divided space, and associates the arranged adjusted 3D Gaussians or the specific adjusted 3D Gaussian with the plurality of chunks set in the three-dimensional space; sets an arbitrary viewpoint in the three-dimensional space; and draws, for each chunk, each adjusted 3D Gaussian associated with the chunk based on the arbitrary viewpoint. At this time, for a specific adjusted 3D Gaussian, identifies a portion corresponding to the interior of the chunk, and draws the identified corresponding portion.

[0159] As a result, according to the second embodiment, when a plurality of adjusted 3D Gaussians are drawn for each chunk, as in the first embodiment, the plurality of adjusted 3D Gaussians can be drawn in an appropriate order. As a result, according to the second embodiment, it is possible to suppress degradation in image quality when generating a 2D image from an arbitrary viewpoint from a plurality of adjusted 3D Gaussians.

[0160] That is, according to the second embodiment, it is possible to provide a technique that can be used when dividing an adjusted three-dimensional Gaussian into a plurality of chunks to generate a two-dimensional image from an arbitrary viewpoint.

[0161] In the second embodiment, the three-dimensional space is divided by a predetermined division method by the adjusted three-dimensional Gaussian registration unit 720. However, the presence or absence of division and the division method may be specified by the user.

[0162] For example, when an adjusted 3D Gaussian is acquired, the adjusted 3D Gaussian registration unit 720 may be configured to inquire of the user whether or not to divide the 3D space into chunks, and whether or not to associate an adjusted 3D Gaussian with each chunk.

[0163] The screen for querying the user may include options for selecting a division method, such as "no division," "grid division," "z division," "y division," and "x division." Furthermore, if, for example, the user selects "no division" on this screen, drawing is performed without linking the adjusted three-dimensional Gaussian to multiple chunks. Furthermore, if, for example, an option other than "no division" is selected on this screen, the three-dimensional space is divided based on the selected division method, and the adjusted three-dimensional Gaussian is linked to multiple chunks before drawing is performed. In other words, the adjusted three-dimensional Gaussian is drawn for each chunk.

[0164] [Fourth Embodiment] In each of the above embodiments, a plurality of chunks are sorted according to the distance from an arbitrary viewpoint to each of the plurality of chunks. However, the method of sorting the plurality of chunks is not limited to this. For example, a first index based on three-dimensional coordinates such as length or size may be calculated for each of the plurality of chunks using a predetermined algorithm, and then the chunks may be sorted according to the calculated first index.

[0165] It should be noted that the process of sorting according to the first index includes both a process of sorting directly using the calculated first index and a process of sorting indirectly using the calculated first index.

[0166] The process of directly sorting using the calculated first indicator includes, for example, in the case where length or size is calculated as the first indicator, a process of sorting by comparing the calculated length or size for each of multiple chunks.

[0167] Furthermore, the process of indirectly sorting using the calculated first index includes, for example, in the case where the length or size is calculated as the first index, a process of sorting using the length or size calculated for each of the multiple chunks by the BSP tree method, etc. Note that BSP is an abbreviation for Binary Space Partitioning.

[0168] In addition, in each of the above embodiments, the multiple adjusted 3D Gaussians associated with each of the multiple chunks are sorted for each chunk according to the distance from an arbitrary viewpoint. However, the sorting method for sorting the multiple adjusted 3D Gaussians associated with each of the multiple chunks is not limited to this.

[0169] For example, sorting may be performed according to a predetermined criterion based on the arbitrary viewpoint, rather than the distance from the arbitrary viewpoint itself. As an example, sorting may be performed according to information related to the distance from the arbitrary viewpoint. In other words, when sorting the multiple adjusted 3D Gaussians associated with the multiple chunks, sorting may be performed according to a second criterion including: the distance from the arbitrary viewpoint, or information related to the distance from the arbitrary viewpoint. Information related to the distance from the arbitrary viewpoint may include, for example, depth.

[0170] [Other Embodiments] In this specification (including the claims), when the expression "at least one of a, b, and c" or "at least one of a, b, or c" (including similar expressions) is used, it includes any of a, b, c, a-b, a-c, bc, or a-bc. It may also include multiple instances of any element, such as a-a, a-bb-b, a-a-bb-cc-c, etc. Furthermore, it also includes adding elements other than the enumerated elements (a, b, and c), such as having d, as in a-b-c-d.

[0171] Furthermore, in this specification (including claims), when expressions such as "using data as input / based on / according to / in response to" (including similar expressions) are used, unless otherwise specified, this includes cases where various data itself is used as input, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) is used as input. Furthermore, when it is stated that a result is obtained "based on / according to / in response to data," this includes cases where the result is obtained based solely on the data in question, as well as cases where the result is obtained in response to other data, factors, conditions, and / or states other than the data in question. Furthermore, when it is stated that "data is output," unless otherwise specified, this includes cases where various data itself is used as output, or where various data that has been processed in some way (e.g., noise-added, normalized, intermediate representation of various data, etc.) is output.

[0172] Furthermore, when the terms "connected" and "coupled" are used in this specification (including the claims), they are intended as open-ended terms that include any of direct connection / coupling, indirect connection / coupling, electrically connection / coupling, communicatively connection / coupling, functionally connection / coupling, and physically connection / coupling. These terms should be interpreted appropriately depending on the context in which they are used, but any connection / coupling form that is not intentionally or naturally excluded should be interpreted as being included in these terms without any restrictions.

[0173] Furthermore, in this specification (including the claims), when the expression "A configured to B" is used, it may include the physical structure of element A having a configuration capable of performing operation B, and the permanent or temporary setting / configuration of element A being configured / set to actually perform operation B. For example, if element A is a general-purpose processor, it is sufficient that the processor has a hardware configuration capable of performing operation B, and is configured to actually perform operation B by setting a permanent or temporary program (instruction). Furthermore, if element A is a dedicated processor or dedicated arithmetic circuit, it is sufficient that the circuit structure of the processor is implemented to actually perform operation B, regardless of whether control instructions and data are actually attached.

[0174] Furthermore, when words implying containing or possessing (e.g., "comprising / including" and "having") are used in this specification (including the claims), they are intended to be open-ended terms that include cases where something other than the object indicated by the object of the term is contained or possessed. When the object of such words implying containing or possessing does not specify a quantity or suggests a singular number (e.g., an expression using the article "a" or "an"), the expression should be construed as not being limited to a specific number.

[0175] Furthermore, although expressions such as "one or more" or "at least one" are used in some places in this specification (including the claims) and expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") are used in other places, the latter expressions are not intended to mean "one." In general, expressions that do not specify a quantity or suggest a singular number (expressions using the articles "a" or "an") should be interpreted as not necessarily being limited to a specific number.

[0176] Furthermore, if a particular advantage / result is described in this specification as being obtained with respect to a particular configuration of an embodiment, it should be understood that the same advantage / result can also be obtained with one or more other embodiments having the same configuration, unless otherwise stated. However, it should be understood that the presence or absence of the effect generally depends on various factors, conditions, and / or states, etc., and that the effect is not necessarily obtained with the configuration. The effect is merely obtained by the configuration described in the embodiment when various factors, conditions, and / or states, etc. are satisfied, and the effect does not necessarily occur in a claimed invention that defines the same or a similar configuration.

[0177] Furthermore, in this specification (including claims), when multiple pieces of hardware perform a predetermined process, the pieces of hardware may cooperate to perform the predetermined process, or some of the hardware may perform all of the predetermined process. Furthermore, some of the hardware may perform part of the predetermined process, and other hardware may perform the rest of the predetermined process. In this specification (including claims), when an expression such as "one or more pieces of hardware perform a first process, and the one or more pieces of hardware perform a second process" is used, the hardware performing the first process and the hardware performing the second process may be the same or different. In other words, it is sufficient that the hardware performing the first process and the hardware performing the second process are included in the one or more pieces of hardware. Note that hardware may include an electronic circuit, a device including an electronic circuit, etc.

[0178] Furthermore, in this specification (including the claims), when multiple storage devices (memories) store data, each of the multiple storage devices (memories) may store only a portion of the data, or may store the entire data.

[0179] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the individual embodiments described above. Various additions, modifications, substitutions, partial deletions, etc. are possible within the scope of the conceptual idea and spirit of the present invention derived from the content defined in the claims and their equivalents. For example, in all of the above-described embodiments, the numerical values ​​used in the explanations are shown as examples and are not limited to these. Furthermore, the order of each operation in the embodiments is shown as an example and is not limited to these.

[0180] The disclosed technology may take the form of the following supplementary notes: (Supplementary Note 1) An image processing device having at least one memory and at least one processor, wherein the at least one processor acquires a plurality of three-dimensional primitives arranged in a three-dimensional space, links the plurality of three-dimensional primitives to a plurality of chunks set in the three-dimensional space, sets a viewpoint in the three-dimensional space, and renders, for each chunk, each of the three-dimensional primitives linked to a chunk based on the viewpoint, wherein the linking of the three-dimensional primitives to the chunks includes linking a three-dimensional primitive spanning two or more chunks to each of the two or more chunks, and the rendering of the three-dimensional primitive for each chunk based on the viewpoint includes identifying, in each of the two or more chunks, a portion of the two or more chunks that corresponds to the interior of the chunk of the spanning three-dimensional primitive, and rendering the identified portion. (Supplementary Note 2) The image processing device according to Supplementary Note 1, wherein the at least one processor identifies a 3D primitive that spans two or more chunks based on the position and shape of the 3D primitive and the position and shape of a chunk, and links the identified 3D primitive with each of the two or more chunks. (Supplementary Note 3) The image processing device according to Supplementary Note 2, wherein each of the two or more chunks includes a different part of the identified 3D primitive. (Supplementary Note 4) The image processing device according to any of Supplements 1 to 3, wherein the at least one processor identifies the part of the spanning 3D primitive that corresponds to the inside of a chunk based on the position and shape of the spanning 3D primitive and the position and shape of the chunk. (Supplementary Note 5) The image processing device according to any one of Supplementary Notes 1 to 4, wherein the drawing of the 3D primitives for each chunk based on the viewpoint is performed by projecting each 3D primitive associated with a chunk in the order of the chunks according to the viewpoint onto a 2D image plane according to the viewpoint in the order according to the viewpoint.(Supplementary Note 6) The image processing device according to any one of Supplementary Notes 1 to 5, wherein the drawing of the 3D primitives for each chunk based on the viewpoint includes: sorting the multiple chunks in an order according to the viewpoint; and, for each chunk after sorting, sorting each of the 3D primitives associated with the chunk in an order according to the viewpoint and projecting them onto a 2D image plane according to the viewpoint. (Supplementary Note 7) The image processing device according to Supplementary Note 6, wherein the at least one processor sets the multiple chunks in the three-dimensional space so that each chunk is convex and does not overlap with another, and a boundary surface between two adjacent chunks is part of a perpendicular bisector of the center points of the two chunks. (Supplementary Note 8) The image processing device according to Supplementary Note 7, wherein the at least one processor sorts the multiple chunks based on a positional relationship between the viewpoint and each chunk. (Supplementary Note 9) The image processing device according to Supplementary Note 7, wherein the at least one processor sorts the plurality of chunks based on the position of the viewpoint and the position of a center point of each chunk. (Supplementary Note 10) The image processing device according to Supplementary Note 8, wherein chunks that intersect a view frustum defined by an arbitrary viewpoint are identified from the plurality of chunks, the identified chunks are sorted, and subsequent processing is terminated for chunks not identified. (Supplementary Note 11) The image processing device according to any of Supplements 1 to 9, wherein the three-dimensional primitive is a distribution that expresses a predetermined geometric shape. (Supplementary Note 12) The image processing device according to any of Supplements 1 to 9, wherein the three-dimensional primitive is a three-dimensional Gaussian or a two-dimensional Gaussian.(Supplementary Note 13) An image processing method including: at least one processor executing a process of obtaining a plurality of three-dimensional primitives arranged in a three-dimensional space; linking the plurality of three-dimensional primitives to a plurality of chunks set in the three-dimensional space; setting a viewpoint in the three-dimensional space; and drawing, for each chunk, each of the three-dimensional primitives linked to the chunk based on the viewpoint; wherein the linking of the three-dimensional primitives to the chunks includes linking a three-dimensional primitive spanning two or more chunks to each of the two or more chunks; and the drawing of the three-dimensional primitives for each chunk based on the viewpoint includes identifying, in each of the two or more chunks, a portion corresponding to the inside of the chunk of the spanning three-dimensional primitive and drawing the identified portion. (Supplementary Note 14) An image processing program that causes at least one processor to execute a process of obtaining a plurality of three-dimensional primitives arranged in a three-dimensional space, linking the plurality of three-dimensional primitives to a plurality of chunks set in the three-dimensional space, setting a viewpoint in the three-dimensional space, and drawing, for each chunk, each of the three-dimensional primitives linked to the chunk based on the viewpoint, wherein the linking between the three-dimensional primitives and the chunks includes linking a three-dimensional primitive that spans two or more chunks to each of the two or more chunks, and the drawing of the three-dimensional primitive for each chunk based on the viewpoint includes identifying, in each of the two or more chunks, a portion of the two or more chunks that corresponds to the inside of the chunk of the spanning three-dimensional primitive and drawing the identified portion.

[0181] This application claims priority based on Japanese Patent Application No. 2024-147912, filed on August 29, 2024, the entire contents of which are incorporated herein by reference.

Claims

1. An image processing device having at least one memory and at least one processor, wherein the at least one processor acquires a plurality of three-dimensional primitives arranged in a three-dimensional space, links the plurality of three-dimensional primitives to a plurality of chunks set in the three-dimensional space, sets a viewpoint in the three-dimensional space, and draws, for each chunk, each of the three-dimensional primitives linked to the chunk based on the viewpoint, wherein the linking of the three-dimensional primitives to the chunks includes linking a three-dimensional primitive that spans two or more chunks to each of the two or more chunks, and the drawing of the three-dimensional primitives for each chunk based on the viewpoint includes identifying, in each of the two or more chunks, a portion of the two or more chunks that corresponds to the interior of the chunk of the spanning three-dimensional primitive, and drawing the identified portion.

2. The image processing device according to claim 1, wherein the at least one processor identifies a three-dimensional primitive that spans two or more chunks based on the position and shape of the three-dimensional primitive and the position and shape of the chunk, and links the identified three-dimensional primitive with each of the two or more chunks.

3. The image processing device according to claim 2, wherein each of the two or more chunks includes a different portion of the identified three-dimensional primitive.

4. The image processing device according to claim 1, wherein the at least one processor identifies the portion of the spanning 3D primitive that corresponds to the inside of the chunk based on the position and shape of the spanning 3D primitive and the position and shape of the chunk.

5. The image processing device according to claim 1, wherein the drawing of the three-dimensional primitives for each chunk based on the viewpoint is performed by projecting each three-dimensional primitive associated with a chunk in the order of the chunks according to the viewpoint onto a two-dimensional image plane according to the viewpoint in the order according to the viewpoint.

6. The image processing device of claim 1, wherein the drawing of the three-dimensional primitives for each chunk based on the viewpoint includes: sorting the plurality of chunks in an order according to the viewpoint; and for each chunk after sorting, sorting each three-dimensional primitive associated with the chunk in an order according to the viewpoint and projecting it onto a two-dimensional image plane according to the viewpoint.

7. The image processing device according to claim 6, wherein the at least one processor sets the plurality of chunks in the three-dimensional space so that each chunk is convex, does not overlap with another chunk, and the boundary surface between two adjacent chunks is part of the perpendicular bisector of the center points of the two chunks.

8. The image processing device according to claim 7, wherein the at least one processor sorts the plurality of chunks based on a positional relationship between the viewpoint and each chunk.

9. The image processing device according to claim 7, wherein the at least one processor sorts the plurality of chunks based on the position of the viewpoint and the position of the center point of each chunk.

10. The image processing device according to claim 8, wherein chunks that intersect a view frustum defined by an arbitrary viewpoint are identified from the plurality of chunks, the identified chunks are sorted, and subsequent processing is terminated for chunks that are not identified.

11. The image processing device according to any one of claims 1 to 9, wherein the three-dimensional primitive is a distribution that represents a predetermined geometric shape.

12. The image processing device according to any one of claims 1 to 9, wherein the three-dimensional primitive is a three-dimensional Gaussian or a two-dimensional Gaussian.

13. An image processing method in which at least one processor executes a process of acquiring a plurality of three-dimensional primitives arranged in a three-dimensional space, linking the plurality of three-dimensional primitives to a plurality of chunks set in the three-dimensional space, setting a viewpoint in the three-dimensional space, and drawing, for each chunk, each of the three-dimensional primitives linked to the chunk based on the viewpoint; the linking of the three-dimensional primitives to the chunks includes linking a three-dimensional primitive that spans two or more chunks to each of the two or more chunks; and the drawing of the three-dimensional primitives for each chunk based on the viewpoint includes identifying, in each of the two or more chunks, a portion of the two or more chunks that corresponds to the interior of the chunk of the spanning three-dimensional primitive and drawing the identified portion.

14. An image processing program that causes at least one processor to execute a process of acquiring a plurality of three-dimensional primitives arranged in a three-dimensional space, linking the plurality of three-dimensional primitives to a plurality of chunks set in the three-dimensional space, setting a viewpoint in the three-dimensional space, and drawing, for each chunk, each of the three-dimensional primitives linked to the chunk based on the viewpoint, wherein the linking of the three-dimensional primitives to the chunks includes linking a three-dimensional primitive that spans two or more chunks to each of the two or more chunks, and the drawing of the three-dimensional primitives for each chunk based on the viewpoint includes identifying, in each of the two or more chunks, a portion of the three-dimensional primitive that corresponds to the interior of the chunk that the three-dimensional primitive spans, and drawing the identified portion.

Images