Image processing device, image processing method, and program

Abstract

Obtain a virtual viewpoint image that does not include the image of the floater. [Solution] The image processing apparatus 102 according to the present disclosure acquires a trained model relating to a three-dimensional field corresponding to the three-dimensional space, which includes at least information representing the degree of presence of an object in the three-dimensional space at each position in the three-dimensional space, and information relating to the number of the objects in the three-dimensional space. Based on the information representing the degree of presence and the information relating to the number of objects, a three-dimensional mask is generated that masks the positions in the three-dimensional field corresponding to positions in the three-dimensional space where the object does not exist. Here, the three-dimensional mask is generated such that the number of clusters remaining when clusters obtained based on the information representing the degree of presence are masked by the three-dimensional mask matches or approaches the number of objects.

Description

[Technical Field]

[0001] This disclosure relates to an image processing technique for generating virtual viewpoint images. [Background technology]

[0002] There is a technique for generating an image corresponding to the view from an arbitrary viewpoint (hereinafter referred to as a "virtual viewpoint") using multiple captured images obtained by capturing from multiple different viewpoints (hereinafter referred to as "multiview images"). Patent Document 1 discloses a technique called NeRF (Neural Radiance Fields) as a method for generating virtual viewpoint images. NeRF consists of a neural network that responds to density and color with an arbitrary position and direction, and volume rendering that calculates pixel values ​​by accumulating the colors obtained at multiple sampling points on a light ray corresponding to a pixel according to the density. Neural network training in NeRF is performed by using the pixel values ​​of each captured image constituting the multiview image as training data, and taking the squared error between the said pixel value and the pixel value calculated by volume rendering as the loss. [Prior art documents] [Patent Documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2023-066705 [Overview of the project] [Problems that the invention aims to solve]

[0004] However, in the technology disclosed in Patent Document 1 (hereinafter referred to as "prior art"), it is possible that an unnecessary density called a floater may be learned to exist in locations where no object actually exists. When volume rendering is performed using the results learned to assume the existence of such floaters, the virtual viewpoint image obtained has the problem that the image of the floater, which looks unnatural to a natural person, is depicted as floating object. [Means for solving the problem]

[0005] The image processing apparatus according to this disclosure includes: a model acquisition means for acquiring a trained model relating to a three-dimensional field corresponding to the three-dimensional space, which includes at least information representing the degree of existence of an object in the three-dimensional space at each position in the three-dimensional space; an object count acquisition means for acquiring information relating to the number of objects in the three-dimensional space; and a three-dimensional mask generation means for generating a three-dimensional mask that masks the positions in the three-dimensional field corresponding to positions in the three-dimensional space where the object does not exist, based on the information representing the degree of existence and the information relating to the number of objects. The three-dimensional mask generation means generates the three-dimensional mask such that the number of clusters remaining when clusters obtained based on the information representing the degree of existence are masked by the three-dimensional mask matches or approaches the number of objects. [Effects of the Invention]

[0006] According to the technology disclosed herein, a virtual viewpoint image can be obtained that does not include the image of the floater. [Brief explanation of the drawing]

[0007] [Figure 1] This figure shows an example of the configuration of the image processing system according to the first embodiment. [Figure 2] This is a block diagram showing an example of the hardware configuration of an image processing device and an information processing device according to the first embodiment. [Figure 3]This is a diagram for explaining the problems of conventional NeRF. [Figure 4] This is a diagram for explaining a method of suppressing the drawing of the image of the float by the three-dimensional mask described in the literature. [Figure 5] This is a diagram for explaining the basic concept regarding the generation of the three-dimensional mask according to the first embodiment. [Figure 6] This is a block diagram showing an example of the functional configuration of the image processing apparatus and the information processing apparatus according to the first embodiment. [Figure 7] This is a flowchart showing an example of the processing flow in the image processing apparatus according to the first embodiment. [Figure 8] This is a diagram for explaining an example of the generation process of the three-dimensional mask in the mask generation unit according to the first embodiment. [Figure 9] This is a flowchart showing an example of the processing flow in the information processing apparatus according to the first embodiment. [Figure 10] This is a diagram for explaining the problems of conventional NeRF. [Figure 11] This is a diagram for explaining an example of the estimation process of the three-dimensional shape of the object according to the second embodiment. [Figure 12] This is a block diagram showing an example of the functional configuration of the image processing apparatus according to the second embodiment. [Figure 13] Figure 13 is a flowchart showing an example of the processing flow in the image processing apparatus according to the second embodiment. [Figure 14] This is a diagram showing an example of the GUI displayed on the UI panel 103 according to the third embodiment. [Figure 15] This is a block diagram showing an example of the functional configuration of the image processing apparatus according to the third embodiment. [Figure 16] This is a flowchart showing an example of the processing flow in the image processing apparatus according to the third embodiment.

MODE FOR CARRYING OUT THE INVENTION

[0008] The embodiments of this disclosure will be described below with reference to the drawings. Note that the following embodiments are not intended to limit the solutions of this disclosure, and not all combinations of features described in these embodiments are necessarily essential to the solutions of this disclosure. The same components will be denoted by the same reference numerals. Furthermore, each step in the flowchart described later will be indicated by a reference numeral beginning with "S".

[0009] Hereinafter, a two-dimensional region in an image will be simply referred to as "region," and a three-dimensional region in the imaging space or virtual space will be referred to as "space." Furthermore, in the following embodiments, it is assumed that a trained model is generated by training a learning model that models the three-dimensional field (hereinafter referred to as the "three-dimensional field") within the imaging space to be trained, and the method for generating the trained model will be described accordingly. Furthermore, in the following embodiments, the learning model that models the three-dimensional field (hereinafter referred to as the "three-dimensional field model") is described as a radiance field generated by NeRF composed of a multilayer perceptron, but the three-dimensional field model is not limited to this.

[0010] The method of representing the three-dimensional field varies depending on the learning content. Specifically, for example, the three-dimensional field model may be constructed using InstantNGP, a high-speed method similar to NeRF. Furthermore, the three-dimensional field model is not limited to one constructed using a multilayer perceptron, but may also be constructed using Plenoxels or TensoRF (Tensorial Radiance Fields), etc., which explicitly represent the three-dimensional field. Additionally, the three-dimensional field model may be constructed using NeuS, etc., which improves the accuracy of shape estimation by representing the three-dimensional field using SDF (Signed Distance Field). Moreover, the three-dimensional field model may be constructed using various methods, such as 3D Gaussian Splatting, in which the three-dimensional field is represented by a set of points with a certain extent.

[0011] [First Embodiment] <Configuration of the image processing system> Figure 1 shows an example of the configuration of an image processing system according to the first embodiment. The image processing system includes multiple imaging devices 101, an image processing device 102, a user interface (hereinafter referred to as "UI") panel 103, a storage device 104, a display device 105, an information processing device 108, a display device 109, and an input device 110.

[0012] Multiple imaging devices 101 are composed of digital still cameras or digital video cameras, and each imaging device 101 is positioned at a different location from the others. Each imaging device 101 acquires data for multiple captured images (multi-view images) corresponding to each viewpoint by capturing objects 107 present in the imaging space 106 from different viewpoints in a synchronized manner according to predetermined imaging conditions. Note that synchronized imaging does not mean simultaneous imaging, but rather imaging after synchronization processing. In other words, synchronized imaging does not need to be performed at exactly the same time, but includes cases where it is performed at approximately the same time. The data of the captured images obtained by imaging by the imaging devices 101 (hereinafter referred to as "captured image data") may be still image data, moving image data, or both still image and moving image data. Hereafter, unless otherwise specified, the term "image" will be used to mean both "still image" and "moving image". The captured image data acquired by each imaging device 101 is transmitted to the image processing device 102.

[0013] The image processing device 102 acquires data from multiple captured images (multi-view images) transmitted from multiple imaging devices 101, and uses the acquired multi-view images to learn a three-dimensional field corresponding to the space containing the object 107 present in the imaging space 106. The information indicating the learned three-dimensional field obtained as a result of the learning by the image processing device 102 is output to the information processing device 108 via a network 111 such as the Internet. The information or signal indicating the learned three-dimensional field may also be output to a storage device 104 or a display device 105, etc. In addition, the image processing device 102 may generate a virtual viewpoint image based on the three-dimensional field in the process of learning or the learned three-dimensional field obtained as a result of learning. In this case, the data or signal of the virtual viewpoint image generated by the image processing device 102 is output to, for example, a storage device 104 or a display device 105, etc.

[0014] In this embodiment, as shown in Figure 1, it is assumed that each of the multiple imaging devices 101 is connected to the image processing device 102. However, the method of connecting the imaging devices 101 and the image processing device 102 is not limited to this. Specifically, for example, multiple imaging devices 101 may be cascaded by connecting adjacent imaging devices 101 to each other, and at least one of the multiple imaging devices 101 may be connected to the image processing device 102.

[0015] Furthermore, in this embodiment, as shown in Figure 1 as an example, multiple imaging devices 101 are described as being arranged at different positions from one another. However, the number and arrangement of imaging devices 101 are not limited to this. For example, if the position, shape, and color of the object 107 present in the imaging space 106, as well as the intensity or hue of the ambient light, do not change over time, at least one imaging device 101 whose position and orientation can be changed may be arranged. In this case, the imaging device 101 may be made to perform imaging at each of several different positions while changing its position and orientation, and the image processing device 102 may acquire the multiple image data obtained by such imaging as data for a multi-view image.

[0016] The UI panel 103 includes a display device such as a liquid crystal panel and displays a GUI (Graphical User Interface) on the display device to present information such as imaging conditions in the imaging device 101 and processing settings of the image processing device 102 to the user. The UI panel 103 may also include an input device such as a touch panel or buttons. In this case, the UI panel 103 receives instructions from the user regarding changes to the aforementioned imaging conditions or processing settings. In this case, information indicating the user instructions received by the UI panel 103 is transmitted to the image processing device 102. The input device may be provided separately from the UI panel 103, such as a mouse or keyboard.

[0017] The storage device 104 is composed of a hard disk drive or the like, and acquires data of the virtual viewpoint image output from the image processing device 102 and stores the acquired data. The storage device 104 also acquires information indicating a three-dimensional field output from the image processing device 102 and stores the acquired information. The display device 105 is composed of a liquid crystal display or the like, and acquires a signal of a display image including the virtual viewpoint image output from the image processing device 102 and displays the virtual viewpoint image corresponding to the signal. The display device 105 also acquires a signal of a display image including an image indicating a three-dimensional field output from the image processing device 102 and displays the image indicating a three-dimensional field corresponding to the signal.

[0018] The imaging space 106 is a three-dimensional space surrounded by multiple imaging devices 101 installed in a studio or the like. In Figure 1, the solid line shows the outline of the imaging space 106 on the floor. Below, as an example, we will describe an embodiment in which one or more objects are imaged from around them using eight imaging devices 101 installed in a studio. Furthermore, although the camera parameters of each imaging device 101 are described as being pre-stored in the memory device 204, they may also be estimated by the image processing device 102 using the captured image data. In this case, for example, the image processing device 102 estimates the camera parameters of each imaging device 101 using an algorithm called COLMAP, which is well known in the field of NeRF and other technologies, and estimates the shape of an object while estimating the imaging position based on the captured image.

[0019] Camera parameters include intrinsic parameters, extrinsic parameters, and distortion parameters. Here, intrinsic parameters represent the coordinates of the center of the image obtained by imaging by the imaging device and the focal length of the lens. External parameters represent the position and orientation of the imaging device, and distortion parameters represent the distortion of the lens. The camera parameters of multiple imaging devices 101, especially the intrinsic parameters and distortion parameters, may be common to each other. Distortion parameters and other parameters besides intrinsic and extrinsic parameters are data that may be included in the camera parameters as needed, but are not necessarily required to be included in the camera parameters.

[0020] The information processing device 108 generates a virtual viewpoint image based on a trained three-dimensional field model output from the image processing device 102. The data or signals of the virtual viewpoint image generated by the information processing device 108 are output to, for example, the display device 109. The display device 109 has the same configuration as the display device 105, so its description is omitted. The input device 110 consists of a mouse or keyboard, and receives input operations from the user of the information processing device 108 and transmits an input signal corresponding to the input operation to the information processing device 108.

[0021] <Hardware configuration of image processing and information processing equipment> Figure 2 is a block diagram showing an example of the hardware configuration of the image processing device 102 and the information processing device 108 according to the first embodiment. Specifically, Figure 2(a) shows an example of the hardware configuration of the image processing device 102. The image processing device 102 has a hardware configuration of a CPU 201, RAM 202, ROM 203, storage device 204, control interface (hereinafter referred to as "I / F") 205, input I / F 206, output I / F 207, and main bus 208.

[0022] The CPU 201 is a processor that comprehensively controls each part of the image processing device 102. The CPU 201 uses the RAM 202 as work memory and executes the OS (operating system) and various programs stored in the ROM 203 or storage device 204, etc. By executing various programs, the CPU 201 controls the entire image processing device 102 via the main bus 208. The processing of each step shown in the flowchart relating to the image processing device 102, which will be described later, is realized by the program code stored in the ROM 203 or storage device 204, etc., being loaded into the RAM 202 and executed by the CPU 201. The RAM 202 functions as the main memory and work area of ​​the CPU 201. The ROM 203 stores the programs executed by the CPU 201. The storage device 204 is composed of a hard disk drive, etc., and stores application programs executed by the CPU 201, and various data used in the processing of the CPU 201.

[0023] The control interface 205 is connected to each of the multiple imaging devices 101 and is a communication interface for controlling each imaging device 101, such as setting imaging conditions, starting imaging, and stopping imaging. The input interface 206 is a communication interface using a serial bus such as SDI (Serial Digital Interface) or HDMI (High-Definition Multimedia Interface) (registered trademark). Image data output from each imaging device 101 is acquired via the input interface 206. The output interface 207 is a communication interface using a serial bus such as USB (Universal Serial Bus) or IEEE1394. Data or signals such as virtual viewpoint images and three-dimensional field models are output to the storage device 104 or display device 105 via the output interface 207. The main bus 208 is a transmission path that connects the above-mentioned hardware configurations of the image processing device 102 so that they can communicate with each other.

[0024] Figure 2(b) shows an example of the hardware configuration of the information processing device 108. The information processing device 108 has a hardware configuration consisting of a CPU 251, RAM 252, ROM 253, storage device 254, input I / F 256, output I / F 257, and main bus 258. The CPU 251 is a processor that comprehensively controls each part of the information processing device 108. The CPU 251 uses RAM 252 as work memory and executes the OS (operating system) and various programs stored in ROM 253 or storage device 254, etc. The CPU 251 controls the entire information processing device 108 via the main bus 258 by executing various programs. The processing of each step shown in the flowchart relating to the information processing device 108, which will be described later, is realized by the program code stored in ROM 253 or storage device 254, etc. being loaded into RAM 252 and executed by the CPU 251. RAM 252 functions as the main memory and work area of ​​the CPU 251.

[0025] The ROM 253 stores a group of programs to be executed by the CPU 251. The storage device 254 is composed of a hard disk drive or the like, and stores application programs to be executed by the CPU 251, various data used for the processing of the CPU 251, and the like. The input I / F 256 is a communication interface via a serial bus such as USB. An input signal corresponding to a user input operation from the input device 110 is received via the input I / F 256. The output I / F 257 is a communication interface via a serial bus such as USB or IEEE 1394. A signal indicating the virtual viewpoint image is output to the display device 109 via the output I / F 257. The main bus 258 is a transmission path that communicably connects the above-described hardware configurations of the information processing device 108 to each other.

[0026] <Regarding the learning of NeRF> In the present embodiment, as an example, for the sake of explanation assuming that the three-dimensional field is represented by the radiance field by NeRF, first, an overview regarding the learning of NeRF will be described. NeRF is composed of a neural network that outputs a volume density σ and a color (r, g, b) for a five-dimensional input variable of three-dimensional coordinates (x, y, z) and a direction (θ, φ) indicating an arbitrary spatial position. Here, each element of the color (r, g, b) is a value corresponding to each color of red (R), green (G), and blue (B) in order. To obtain the pixel value (r, g, b), first, a plurality of (N (N is a positive integer of 2 or more)) sampling points P i (i is a positive integer not exceeding N) are prepared on the ray corresponding to the pixel. Subsequently, by inputting the position (x, y, z) of the sampling point P i and the direction (θ, φ) of the ray into the neural network, the volume density σ i corresponding to each sampling point P i and the color c i are output from the neural network. Further, by a rendering method called volume rendering that can represent a translucent object, a weighted sum c i of colors based on the volume density σ i is calculated, and thus the pixel value C v is determined.

[0027] In volume rendering, first, based on the volume density and the distance between sampling points, for each sampling point P i the cumulative transmittance T i at is obtained. The cumulative transmittance T i represents the ratio at which the color c i at the sampling point P i reaches the imaging position. Specifically, the cumulative transmittance T i is calculated, for example, by using Equation (1).

[0028]

Equation

[0029]

Equation

[0030]

number

[0031] <Problems with conventional NeRFs> Before describing the specific embodiments of this disclosure, we will explain the problems with conventional NeRFs. Figure 3 is a diagram illustrating the problems with conventional NeRFs. Figure 3(a) shows an object 107, in which a gray sphere 301 and a black sphere 302 are joined together, being imaged by multiple imaging devices 101 (101a and 101b). Figure 3(b) shows an example of a radiance field estimated as a result of NeRF learning using the image obtained from the imaging in Figure 3(a).

[0032] In a case like that shown in Figure 3(a), for space 311, which is captured as images of different colors by imaging device 101a and imaging device 101b, it can be seen from the difference in the colors of the images that no objects exist in that space. Therefore, the generation of floaters is suppressed in space 311. However, even if a floater 303 of the same color as the image is generated in space 312, which is captured as an image of the same color by imaging device 101a and imaging device 101b, there is no difference between the image obtained by rendering (virtual viewpoint image) and the captured image. In other words, even if a floater 303 exists in space 312, the value of the loss L becomes small. Therefore, when it is estimated that a floater 303 exists in space 312, the NeRF learning converges and the generation of the floater 303 cannot be suppressed. As a result, in the virtual viewpoint image corresponding to the view from the virtual viewpoint, an unnatural floating object image due to the floater 303 is drawn, and the quality of the virtual viewpoint image deteriorates.

[0033] <Regarding conventional methods for suppressing floaters> The following non-patent literature discloses a method for suppressing floaters. Specifically, in the method disclosed in the said non-patent literature (hereinafter referred to as the "literature method"), first, the radiance field is estimated using a conventional NeRF learning method. Next, based on the estimated radiance field, three-dimensional grid data (hereinafter referred to as the "three-dimensional mask") is created in which spaces with high density are represented by 1 and spaces with low density are represented by 0. Next, grids with high density in the three-dimensional mask, i.e., grids with a value of 1, are clustered. Next, clusters are selected from all clusters in descending order of cluster volume to represent objects until the proportion of the total volume of all clusters exceeds 85%. Next, grids belonging to clusters that were not selected are converted to 0 in the three-dimensional mask. When generating a virtual viewpoint image, spaces where the three-dimensional grid is 0 are excluded from sampling. According to such a literature method, even if a density corresponding to a floater exists in the radiance field, the image of the floater can be reduced in the virtual viewpoint image.

[0034] <Non-patent literature> Wirth Tristan, et al., “A Post Processing Technique to Automatically Remove Floater Artifacts in Neural Radiance Fields”, 2023, The Eurographics Association and John Wiley & Sons Ltd., [online], [Retrieved September 30, 2024], Internet<https: / / onlinelibrary.wiley.com / doi / 10.1111 / cgf.14977> Figure 4 illustrates a method for suppressing the rendering of floater images using a three-dimensional mask as described in the literature. Specifically, Figure 4(a) shows an example of volume rendering for a radiance field estimated using a conventional NeRF learning method. Figure 4(b) shows an example of volume rendering for a radiance field masked using a three-dimensional mask created using the literature method, where the radiance field estimated using a conventional NeRF learning method is masked.

[0035] As shown in Figure 4(a), the density and color at the sampling points are accumulated, so that the space in which the floater exists is also sampled, and the image of the floater is drawn in the virtual viewpoint image. On the other hand, in the example shown in Figure 4(b), a three-dimensional mask created using the literature method, in which high-density grids are represented in white and low-density grids in black, is used to mask small clusters corresponding to the floater in the estimated radiance field. In this case, points on the light rays are not sampled within grids where the value in the three-dimensional mask is 0. Therefore, in the example shown in Figure 4(b), the drawing of the floater image can be suppressed in the virtual viewpoint image obtained by volume rendering.

[0036] However, the aforementioned non-patent literature does not disclose the rationale behind the 85% threshold set as the criterion for selecting clusters to represent objects. For example, with an 85% threshold, if there are seven objects of the target in a scene, the smallest of these objects would be masked as a floater. In other words, while the aforementioned non-patent literature discloses a method for masking clusters that are presumed to be floaters, it does not disclose how to determine which clusters should be masked.

[0037] <Basic Concepts Related to the Technology in This Disclosure> The image processing device 102 obtains the number of target objects and generates a three-dimensional mask that masks clusters in the radiance field based on the obtained number of objects. Figure 5 is a diagram illustrating the basic concept of generating a three-dimensional mask according to the first embodiment. Figure 5(a) shows an example of a true three-dimensional shape corresponding to a target object. In Figure 5(a), the true three-dimensional shape corresponding to the target object is shown as a white region, and its number is 2. Figure 5(b) shows an example of a three-dimensional shape obtained by the viewing volume cross-section method using acquired multi-view images. In Figure 5(a), the three-dimensional shape obtained by the viewing volume cross-section method is shown as a white region. The image processing device 102 obtains the number of target objects by clustering the three-dimensional shape obtained by the viewing volume cross-section method. In the case of the three-dimensional shape shown as an example in Figure 5(b), the number of target objects is 2.

[0038] Figure 5(c) shows an example of a cluster obtained by clustering locations with high density in the radiance field estimated by the conventional NeRF learning method. In Figure 5(c), the cluster is shown as a white region, and its number is 5 due to the generation of floaters. Figure 5(d) shows an example of a three-dimensional mask generated by the image processing device 102. The image processing device 102 generates a three-dimensional mask in which, among the spaces in which the clusters shown in Figure 5(c) exist, spaces containing two clusters that correspond to the number of three-dimensional shapes obtained based on Figure 5(b) are set to 1, and the remaining spaces are set to 0.

[0039] <Functional Configuration of Image Processing Devices and Information Processing Devices> Figure 6 is a block diagram showing an example of the functional configuration of the image processing device 102 and the information processing device 108 according to the first embodiment. The image processing device 102 has an image acquisition unit 601, a model acquisition unit 602, an object count acquisition unit 603, a mask generation unit 604, and an output unit 605 as its functional configuration. The functional configuration of the information processing device 108 will be described later. Each part of the functional configuration of the image processing device 102 is realized by the CPU 201 executing a program stored in the ROM 203 or the like, using the RAM 202 as work memory. Note that not all of the processes shown below necessarily need to be realized by the execution of a program by the CPU 201, and the image processing device 102 may be configured so that some or all of the processes are executed by one or more processing circuits other than the CPU 201.

[0040] The image acquisition unit 601 acquires data of captured images (multi-view images) obtained by each of the multiple imaging devices 101, and the camera parameters of each imaging device 101 (hereinafter referred to as "imaging camera parameters"). The source of the captured image data is not limited to the imaging device 101; the image acquisition unit 601 may also acquire captured image data by reading it from a storage device 104 or the like. The captured image data and imaging camera parameters acquired by the image acquisition unit 601 are transmitted to the model acquisition unit 602 and the object count acquisition unit 603.

[0041] The model acquisition unit 602 performs training on the three-dimensional field model. Specifically, the model acquisition unit 602 performs training on the three-dimensional field model by using the captured image data transmitted from the image acquisition unit 601 as training data. After all training is completed, the model acquisition unit 602 transmits the trained three-dimensional field model data to the mask generation unit 604 and the output unit 605. The training method for the three-dimensional field model is omitted here, as the example of a radiance field was described above.

[0042] The object count acquisition unit 603 acquires information regarding the number of objects (hereinafter referred to as "object count") present in a predetermined learning space. Specifically, first, the object count acquisition unit 603 acquires data regarding the three-dimensional shape of the objects present in the learning space. For example, the object count acquisition unit 603 acquires data regarding the three-dimensional shape of an object by inferring its three-dimensional shape using the imaging camera parameters and imaging data transmitted from the image acquisition unit 601. Subsequently, the object count acquisition unit 603 acquires information regarding the object count by identifying the number of objects present in the learning space based on the acquired data regarding the three-dimensional shape of the object. Details of the three-dimensional shape inference process in the object count acquisition unit 603 will be described later. The object count acquired by the object count acquisition unit 603 is transmitted to the mask generation unit 604.

[0043] The mask generation unit 604 generates a three-dimensional mask to mask the floaters included in the trained three-dimensional field model. Specifically, the mask generation unit 604 generates a three-dimensional mask using information on the number of objects transmitted from the object count acquisition unit 603 and data of the trained three-dimensional field model transmitted from the model acquisition unit 602. Details of the three-dimensional mask generation process in the mask generation unit 604 will be described later. The data of the three-dimensional mask generated by the mask generation unit 604 is transmitted to the output unit 605.

[0044] The output unit 605 outputs the data of the trained three-dimensional field model transmitted from the model acquisition unit 602, and the data of the three-dimensional mask transmitted from the mask generation unit 604. Specifically, for example, the output unit 605 outputs the data of the trained three-dimensional field model and the three-dimensional mask to the information processing device 108. The destination of the output of the data of the trained three-dimensional field model and the three-dimensional mask is not limited to the information processing device 108; the output unit 605 may also output this data to the storage device 104 and store the data in the storage device 104.

[0045] Furthermore, the output of the output unit 605 is not limited to the data of the trained three-dimensional field model and three-dimensional mask. For example, in addition to outputting this data, the output unit 605 may output a signal of a display image including an image representing the trained three-dimensional field model to a display device such as the UI panel 103 or the display device 105, and have the display device display the image. Alternatively, in addition to the data of the trained three-dimensional field model and three-dimensional mask, the output unit 605 may output a virtual viewpoint image obtained by volume rendering using a three-dimensional field model that is in the process of being trained or has been trained. In this case, for example, the output unit 605 may output the data of the virtual viewpoint image to the storage device 104 and store the data in the storage device 104. Alternatively, for example, the output unit 605 may output the virtual viewpoint image to a display device such as the UI panel 103 or the display device 105, and have the display device display the virtual viewpoint image.

[0046] The information processing device 108 has a functional configuration that includes a model acquisition unit 606, a viewpoint acquisition unit 607, an image generation unit 608, and an output unit 609. Each of the functional components of the information processing device 108 is realized by the CPU 251 executing a program stored in the ROM 253 or the like, using the RAM 252 as work memory. However, not all of the processes shown below necessarily need to be realized by the execution of a program by the CPU 251, and the information processing device 108 may be configured so that some or all of the processes are executed by one or more processing circuits other than the CPU 251.

[0047] The model acquisition unit 606 acquires the trained three-dimensional field model and three-dimensional mask data output from the image processing device 102. The source of the trained three-dimensional field model and three-dimensional mask data is not limited to the image processing device 102. For example, the model acquisition unit 606 may acquire the trained three-dimensional field model and three-dimensional mask data by reading data stored in the storage device 104 or the like. The trained three-dimensional field model and three-dimensional mask data acquired by the model acquisition unit 606 is transmitted to the image generation unit 608.

[0048] The viewpoint acquisition unit 607 acquires information related to a virtual viewpoint (hereinafter referred to as "virtual viewpoint information"). The virtual viewpoint information includes, for example, information indicating the position of the virtual viewpoint and information indicating the direction of the line of sight at the virtual viewpoint. For example, the virtual viewpoint information is input based on user input operations to the information processing device 108 in the input device 110. The virtual viewpoint information acquired by the viewpoint acquisition unit 607 is transmitted to the image generation unit 608.

[0049] The image generation unit 608 generates a virtual viewpoint image corresponding to the view from the virtual viewpoint indicated by the virtual viewpoint information, based on the trained three-dimensional field model and three-dimensional mask data transmitted from the model acquisition unit 606, and the virtual viewpoint information transmitted from the viewpoint acquisition unit 607. Details of the virtual viewpoint image generation process in the image generation unit 608 will be described later. The virtual viewpoint image data generated by the image generation unit 608 is transmitted to the output unit 609.

[0050] The output unit 609 outputs the virtual viewpoint image transmitted from the image generation unit 608. Specifically, for example, the output unit 609 outputs the generated virtual viewpoint image to the display device 109, causing the display device 109 to display the virtual viewpoint image.

[0051] <Operation of the image processing device> The operation of the image processing device 102 will be explained with reference to Figure 7. Figure 7 is a flowchart showing an example of the processing flow in the image processing device 102 according to the first embodiment. Specifically, the flowchart shown in Figure 7(a) shows the overall processing flow in the image processing device 102. The flowcharts shown in Figures 7(b) and (c) will be described later.

[0052] First, in S701, the image acquisition unit 601 acquires the image data obtained by each imaging device 101, as well as the imaging camera parameters related to the imaging. Next, in S702, the model acquisition unit 602 acquires a trained three-dimensional field model by training the three-dimensional field model using the image data acquired in S701 as training data. Next, in S703, the object count acquisition unit 603 performs the process of acquiring information regarding the number of objects based on the imaging camera parameters and image data acquired in S701. Details of the object count acquisition process in the object count acquisition unit 603 will be described later.

[0053] Next, in S704, the mask generation unit 604 performs a three-dimensional mask generation process based on the information regarding the number of objects acquired in S703 and the data of the trained three-dimensional field model acquired in S702. Details of the three-dimensional mask generation process in the mask generation unit 604 will be described later. Next, in S705, the output unit 605 outputs the data of the trained three-dimensional field model acquired in S702 and the data of the three-dimensional mask generated in S704 to the information processing device 108. After S705, the image processing device 102 completes the processing of the flowchart shown in Figure 7(a). The image processing device 102 repeatedly executes the processing of the flowchart, for example, each time a new captured image is output from each imaging device 101. A new captured image is, for example, a new frame if the captured image is a moving image.

[0054] Figure 7(b) is a flowchart showing an example of the object count acquisition process by the object count acquisition unit 603, and is a flowchart showing an example of a detailed processing flow in S703. The processing in the flowchart shown in Figure 7(b) is executed after S702. After S702, first, in S706, the object count acquisition unit 603 generates a silhouette image showing the object region by extracting the region containing the image of an object (hereinafter referred to as the "object region") from each captured image acquired in S701.

[0055] Next, in S707, the object count acquisition unit 603 estimates the three-dimensional shape of the object using the visual volume cross-section method with the silhouette image. As an example, the following explanation assumes that the data of the three-dimensional shape (hereinafter referred to as "shape data") is represented by voxel data (hereinafter referred to as "voxel data"). Voxel data is a representation of three-dimensional space divided into small cubes called voxels, where each voxel is represented by a binary value such that, for example, 1 if it is inside an object and 0 if it is outside.

[0056] Next, in S708, the object count acquisition unit 603 performs clustering on the shape data, grouping adjacent voxels with a value of 1 into a single cluster. In this case, each cluster represents a three-dimensional shape corresponding to an object. Then, in S709, the object count acquisition unit 603 acquires information about the number of objects by counting the number of clusters. After S709, the object count acquisition unit 603 completes the process shown in the flowchart in Figure 7(b), i.e., the process in S703.

[0057] Figure 7(c) is a flowchart showing an example of the process flow for generating a three-dimensional mask by the mask generation unit 604, and is a flowchart showing an example of a detailed processing flow in S704. The processing shown in the flowchart in Figure 7(c) is executed after S703. After S703, first, in S710, the mask generation unit 604 generates shape data expressed in voxel format using information on the density in the radiance field shown by the trained three-dimensional field model acquired in S702. Specifically, for example, the mask generation unit 604 generates voxel data as shape data in which the density value corresponding to the center coordinate of the voxel is set to 1 if it is above a predetermined threshold, and to 0 if it is below the threshold. The mask generation unit 604 uses the generated shape data as the initial three-dimensional mask data.

[0058] Figure 8 is a diagram illustrating an example of the three-dimensional mask generation process in the mask generation unit 604 according to the first embodiment. Specifically, Figure 8(a) shows an example of information regarding density in the radiance field shown by the trained three-dimensional field model. In Figure 8(a), the density values ​​in the radiance field are represented in grayscale, with higher density locations being represented in a color closer to white and lower density locations in a color closer to black. Two floaters 801 are generated in the radiance field shown in Figure 8(a). Figure 8(b) shows an example of voxel data, for example, represented in binary, obtained based on the density values ​​in the radiance field of the trained three-dimensional field model shown in Figure 8(a).

[0059] Following S710, in S711, the mask generation unit 604 clusters the shape data, represented by the voxel data shown as an example in Figure 8(b), which was generated in S711, by performing the same processing as in S708. Hereafter, the mask generation unit 604 will be described as clustering voxels such that voxels with a value of 1 and a Euclidean distance of 1 or less between them belong to the same cluster.

[0060] Next, in S712, the mask generation unit 604 extracts clusters from the shape data after clustering in S711, in order of the largest cluster volume, corresponding to the number of objects obtained in S709. Next, in S713, the mask generation unit 604 generates a three-dimensional mask that masks the clusters other than those extracted in S712. Specifically, the mask generation unit 604 generates the three-dimensional mask data by changing the values ​​of all voxels not included in the clusters extracted in S712 to 0 in the initial three-dimensional mask data obtained in S710. Figure 8(c) shows an example of a three-dimensional mask based on Figure 8(b) generated by the mask generation unit 604. Figure 8(d) shows the density information in the radiance field shown in Figure 8(a) superimposed on the three-dimensional mask shown in Figure 8(c). As shown as an example in Figure 8(d), the region corresponding to the floater 801 that occurred in the radiance field shown in Figure 8(a) is masked by the three-dimensional mask shown in Figure 8(c).

[0061] Figure 9 is a flowchart showing an example of the processing flow in the information processing device 108 according to the first embodiment. First, in S901, the model acquisition unit 606 acquires the trained three-dimensional field model data and the three-dimensional mask data output from the image processing device 102. Next, in S902, the viewpoint acquisition unit 607 acquires information about a virtual viewpoint (virtual viewpoint information). Next, in S903, the image generation unit 608 generates rays corresponding to each pixel of the virtual viewpoint image corresponding to the field of view defined by the virtual viewpoint information acquired in S902. Next, in S904, the image generation unit 608 generates sampling points for volume rendering on the rays generated in S903. Next, in S905, the image generation unit 608 excludes sampling points from the sampling points generated in S904 that are included in the space where the value in the three-dimensional mask is 0.

[0062] Next, in S906, the image generation unit 608 obtains density and color values ​​from the data of the trained three-dimensional field model for the sampling points that were not excluded in S906 from the sampling points generated in S904. Next, in S907, the image generation unit 608 calculates the value of each pixel (pixel value) of the virtual viewpoint image by performing volume rendering using the density and color values ​​obtained in S906. The method for calculating these pixel values ​​has been described above and will not be explained here. Next, in S908, the output unit 609 outputs the virtual viewpoint image generated in S907. After S908, the information processing device 108 completes the processing of the flowchart shown in Figure 9. Note that if the trained three-dimensional field model and three-dimensional mask data obtained in S901, and the virtual viewpoint information obtained in S902, are time-series data, the information processing device 108 repeatedly executes the processes from S903 to S908. Through this repeated execution, a virtual viewpoint image corresponding to each frame of the video is generated in S907, and the generated virtual viewpoint image corresponding to each frame is output in S908.

[0063] As described above, the image processing system was configured to detect the space corresponding to the floaters that do not originally exist in the three-dimensional field as a result of the learning process, generate a three-dimensional mask, and then generate a virtual viewpoint image that does not include the image of the floaters by using the three-dimensional mask. With this configured image processing system, it is possible to suppress the degradation of image quality in the virtual viewpoint image.

[0064] In this embodiment, an example of how the image processing device 102 performs learning of a three-dimensional field model has been described, but the embodiment is not limited to this. For example, the image processing device 102 may acquire data of a three-dimensional field model obtained as a result of learning in an external device and generate a three-dimensional mask using the acquired learned three-dimensional field model data. Also, in this embodiment, an example of how the information processing device 108 generates a virtual viewpoint image using the learned three-dimensional field model data and the three-dimensional mask data output from the image processing device 102 has been described, but the embodiment is not limited to this. For example, the image processing device 102 may have additional functional configurations similar to those of the information processing device 108, and the image processing device 102 may generate the virtual viewpoint image.

[0065] [Second Embodiment] In the first embodiment, a method was described in which the mask generation unit 604 creates a three-dimensional mask based on learned information about the density in the radiance field, thereby suppressing the inclusion of the floater image in the virtual viewpoint image and suppressing a degradation in the quality of the virtual viewpoint image. However, in the configuration according to the first embodiment, a difference may occur between the virtual viewpoint image used for learning in the image processing device 102, i.e., for calculating the loss, and the virtual viewpoint image generated in the information processing device 108. In this case, the virtual viewpoint image generated by volume rendering using the imaging viewpoint after applying the three-dimensional mask as the virtual viewpoint may not be sufficiently optimal for the imaging image to be learned.

[0066] Figure 10 is a diagram illustrating the problems of conventional NeRF. Figure 10(a) is the same as Figure 3(a), showing how multiple imaging devices 101 (101a and 101b) are imaging an object 107 in which a gray sphere 301 and a black sphere 302 are joined together. Figure 10(b) shows an example of a radiance field estimated as a result of NeRF learning using the image obtained from imaging in Figure 10(a). Even if a hole 1013 penetrating the object 1007, as shown as an example in Figure 10(b), occurs in the radiance field obtained as a result of learning, the presence of the floater 303 may prevent loss to the learning image. In such a case, if the floater is masked by the method of the first embodiment, a discrepancy occurs between the virtual viewpoint image, which uses the position of imaging device 101a as a virtual viewpoint, and the image obtained by imaging by imaging device 101a.

[0067] Therefore, in this embodiment, we will describe an embodiment in which the image processing device 102 generates a three-dimensional mask and then retrains the previously trained three-dimensional field model using the three-dimensional mask. Specifically, we will describe an embodiment in which the image processing device 102 applies the generated three-dimensional mask to the previously trained three-dimensional field model and retrains the previously trained three-dimensional field model based on the loss to the volume rendering result using the applied three-dimensional field model. By retraining the previously trained three-dimensional field model in this way, it is possible to suppress the occurrence of holes 1013 in the trained radiance field that are not present in the original object 107, as shown as an example in Figure 10(b). As a result, the reproducibility of the virtual viewpoint image can be improved.

[0068] Furthermore, in the first embodiment, a method was described in which the object count acquisition unit 603 identifies the number of objects based on the viewing volume cross-section method. However, while the viewing volume cross-section method is less likely to generate floaters, it has the characteristic that estimation errors may occur due to occlusion and other factors.

[0069] Figure 11 is a diagram illustrating an example of the three-dimensional shape estimation process of an object according to the second embodiment. Specifically, Figure 11(a) is a diagram showing an example of the three-dimensional shape of an object, and represents the true three-dimensional shape of the object. In Figure 11(a), there are three-dimensional shapes corresponding to four objects. That is, the true number of objects is 4. Figure 11(b) shows an example of the estimation result of the three-dimensional shape of an object using the viewing volume cross-section method. When estimating the three-dimensional shape of an object like Figure 11(a) using the viewing volume cross-section method, there are cases where multiple objects are estimated as a single object, as in Figure 11(b). In Figure 11(b), there are three-dimensional shapes corresponding to two objects. In the case of an estimation result like Figure 11(b), the number of objects identified based on the estimation result is 2. Therefore, if the density values ​​in the learned radiance field are masked using the three-dimensional mask generated by the method described in the first embodiment, the two smaller objects will be masked as floaters.

[0070] Therefore, in this embodiment, the image processing device 102 will be described in which it calculates the product of voxel data acquired by the viewing volume cross-section method and a three-dimensional mask obtained based on a trained three-dimensional field model before clustering processing. By calculating this product, it is possible to estimate the three-dimensional shape while reflecting the advantages of the viewing volume cross-section method, such as the suppression of floaters, and the advantages of NeRF, such as the accuracy of shape estimation.

[0071] Figure 11(c) shows an example of the three-dimensional shape of an object estimated based on the radiance field obtained as a result of training a three-dimensional field model. In Figure 11(c), there are four three-dimensional shapes corresponding to the object, as well as four floaters. The image processing device 102 calculates the product of the estimation result of the three-dimensional shape of the object by the viewing volume crossing method and the estimation result of the three-dimensional shape of the object based on the trained three-dimensional field model. Figure 11(d) shows an example of the product of the estimation result of the three-dimensional shape of the object by the viewing volume crossing method shown in Figure 11(b) and the estimation result of the three-dimensional shape of the object based on the radiance field shown in Figure 11(c). In Figure 11(d), there are four three-dimensional shapes corresponding to the object, as well as three remaining floaters. Therefore, the image processing device 102 performs a shrinkage process on the three-dimensional shape shown in Figure 11(b).

[0072] Figure 11(e) shows an example of the three-dimensional shape after the shrinkage process. The image processing device 102 identifies the number of objects based on the three-dimensional shape after the shrinkage process and obtains information about the number of objects. By performing such a shrinkage process, small floaters generated in the trained radiance field are removed, and the accuracy of obtaining the number of objects can be improved. Using the number of objects obtained in this way, the image processing device 102 extracts three-dimensional shapes for that number of objects from the estimation results of the three-dimensional shape of the objects based on the trained three-dimensional field model. Figure 11(f) shows the result of extracting four three-dimensional shapes, which are the number of objects identified based on the three-dimensional shape shown in Figure 11(e), from the three-dimensional shapes shown in Figure 11(c), in order from the largest volume. Using this method, a three-dimensional shape close to the true three-dimensional shape of the object shown in Figure 11(a), such as the example shown in Figure 11(f), can be obtained.

[0073] <Image Processing Device Functional Configuration> Figure 12 is a block diagram showing an example of the functional configuration of the image processing apparatus 102 according to the second embodiment (hereinafter simply referred to as "image processing apparatus 102"). The image processing apparatus 102 has an image acquisition unit 601, a model acquisition unit 1202, an object count acquisition unit 1203, a mask generation unit 604, and an output unit 1205 as its functional configuration. Compared to the image processing apparatus 102 according to the first embodiment, the image processing apparatus 102 has been changed from the model acquisition unit 602, the object count acquisition unit 603, and the output unit 605 to the model acquisition unit 1202, the object count acquisition unit 1203, and the output unit 1205. Each part of the functional configuration of the image processing apparatus 102 is realized by the CPU 201 executing a program stored in ROM 203 or the like, using RAM 202 as work memory.

[0074] The model acquisition unit 1202 performs training on the three-dimensional field model. Specifically, the model acquisition unit 602 performs training on the three-dimensional field model by using the captured image data transmitted from the image acquisition unit 601 as training data. This training process is the same as the training process of the model acquisition unit 602 in the first embodiment, so a detailed explanation is omitted. The trained three-dimensional field model data obtained as a result of training in the model acquisition unit 1202 is transmitted to the object count acquisition unit 1203 and the mask generation unit 604.

[0075] The object count acquisition unit 1203 acquires information regarding the number of objects. Specifically, first, the object count acquisition unit 1203 uses the captured image data and imaging camera parameters transmitted from the image acquisition unit 601 to infer the three-dimensional shape of the objects present in the learning space. This inference process is the same as the inference process for the three-dimensional shape of objects in the object count acquisition unit 603 according to the first embodiment, so a detailed explanation is omitted. Next, the object count acquisition unit 1203 uses the data of the three-dimensional shape of the objects obtained by estimation and the trained three-dimensional field model received from the model acquisition unit 1202 to determine the number of objects using the method described above, and acquires information regarding the number of objects. The information regarding the number of objects acquired by the object count acquisition unit 1203 is transmitted to the mask generation unit 604.

[0076] The mask generation unit 604 generates a three-dimensional mask using the data of the trained three-dimensional field model received from the model acquisition unit 1202 and the information regarding the number of objects received from the object count acquisition unit 1203. This generation process is the same as the three-dimensional mask generation process in the mask generation unit 604 according to the first embodiment, so a detailed explanation is omitted. The data of the three-dimensional mask generated by the mask generation unit 604 is transmitted to the model acquisition unit 1202 and the output unit 1205. In addition to the training process described above, the model acquisition unit 1202 retrains the trained three-dimensional field model using the three-dimensional mask received from the mask generation unit 604. The data of the retrained three-dimensional field model from the model acquisition unit 1202 is transmitted to the output unit 1205. The output unit 1205 outputs the data of the retrained three-dimensional field model received from the model acquisition unit 1202 and the data of the three-dimensional mask received from the mask generation unit 604 to the information processing device 108.

[0077] <Operation of the image processing device> The operation of the image processing device 102 will be explained with reference to Figure 13. Figure 13 is a flowchart showing an example of the processing flow in the image processing device 102 according to the second embodiment. Specifically, the flowchart shown in Figure 13(a) shows the overall processing flow in the image processing device 102. The flowchart shown in Figure 13(b) will be described later. First, the image processing device 102 executes the processes of S701 and S702. After S702, in S1303, the object count acquisition unit 1203 acquires information on the number of objects based on the imaging camera parameters and imaging image data acquired in S701 and the trained three-dimensional field model obtained as a result of learning in S702. Details of this acquisition process in the object count acquisition unit 1203 will be described later. Next, the image processing device 102 executes the process of S704.

[0078] After S704, in S1305, the model acquisition unit 1202 retrains the previously trained three-dimensional field model. The difference between the training process in S702 and the retraining process in S1305 is whether or not a three-dimensional mask is applied to the virtual viewpoint image generated during training by volume rendering. In the retraining process in S1305, by applying the three-dimensional mask to the virtual viewpoint image, the loss between the captured image and the pixel values ​​with the floater masked is calculated. Next, in S1306, the output unit 1205 outputs the data of the retrained three-dimensional field model obtained as a result of the retraining process in S1305, and the three-dimensional mask generated in S704. After S1306, the image processing device 102 completes the processing shown in the flowchart in Figure 13.

[0079] Figure 13(b) is a flowchart showing an example of the object count acquisition process by the object count acquisition unit 1203, and is a flowchart showing an example of a detailed processing flow in S1303. The processing in the flowchart shown in Figure 13(b) is executed after S702. After S702, the object count acquisition unit 1203 first executes the processes in S706 and S707. After S707, in S1308, the object count acquisition unit 1203 acquires a three-dimensional mask based on a trained three-dimensional field model. Specifically, the object count acquisition unit 1203 generates shape data using the trained three-dimensional field model data acquired in S702 by executing the same process as in S710, and acquires the generated shape data as three-dimensional mask data.

[0080] Next, in S1309, the object count acquisition unit 1203 calculates the product of the shape data estimated in S707 and the three-dimensional mask data acquired in S1308, thereby masking the shape data using the three-dimensional mask. This is achieved by setting the values ​​of the voxels in the shape data that correspond to voxels with a value of 0 in the three-dimensional mask to 0.

[0081] Next, in S1310, the object count acquisition unit 1203 performs a shrinkage process on the shape data after the masking process in S1309. For example, in the shrinkage process, for a voxel with a value of 1, the object count acquisition unit 1203 refers to the values ​​of surrounding voxels, such as those with a Euclidean distance of 5 or less between them, and changes the value of the voxel to be processed to 0 if there is a voxel with a value of 0. Next, in S1311, the object count acquisition unit 1203 performs clustering on the shape data after the shrinkage process in S1310. After S1311, the object count acquisition unit 1203 performs the process in S709. After S709, the object count acquisition unit 1203 completes the process shown in the flowchart in Figure 13(b), i.e., the process in S1303.

[0082] Note that the information processing device 108 according to the second embodiment is the same as the information processing device 108 according to the first embodiment, so its description will be omitted. With the image processing system configured as described above, it is possible to suppress the deterioration of image quality in virtual viewpoint images.

[0083] [Third Embodiment] In the first embodiment, a method for automatically identifying the number of objects was described, based on shape data obtained by a method for estimating the three-dimensional shape of objects that is less prone to generating floaters compared to a method for estimating the three-dimensional shape of objects based on the learning results of a three-dimensional field model. However, the number of objects that can be identified based on the estimated three-dimensional shape of objects is not always accurate, and there may be discrepancies with the number of objects that a user perceives as appropriate when observing a virtual viewpoint image. Therefore, in this embodiment, a method is described in which the user specifies the number of objects that should be included as images in the virtual viewpoint image via a user interface such as a UI panel 103.

[0084] Figure 14 shows examples of GUIs 1410, 1420, and 1430 displayed on the UI panel 103 according to Embodiment 3. Specifically, GUI 1410 shown in Figure 14(a) includes the rendering result of a virtual viewpoint image when the number of objects specified in the UI panel 103 is too large. The virtual viewpoint image included in GUI 1410 shown in Figure 14(a) includes images of a large number of floaters. GUI 1420 shown in Figure 14(b) includes the rendering result of a virtual viewpoint image when the number of objects specified in the UI panel 103 is too small. In the virtual viewpoint image included in GUI 1420 shown in Figure 14(b), some of the images of multiple captured objects (natural people in Figure 14) are excluded from the rendering target, and it cannot be said to be a virtual viewpoint image with high reproducibility.

[0085] The GUI 1430 shown in Figure 14(c) includes the rendering result of a virtual viewpoint image when an appropriate number of objects is specified in the UI panel 103. The virtual viewpoint image included in the GUI 1430 shown in Figure 14(c) includes images of all captured objects, and the images of floaters have been appropriately removed. The number of captured objects is a value that is relatively easy for the user to set. Therefore, providing the GUI described in this embodiment is particularly effective when removing floaters from a three-dimensional field model trained by an external device as a retouching process before image editing.

[0086] Figure 15 is a block diagram showing an example of the functional configuration of the image processing device 102 according to the third embodiment (hereinafter simply referred to as "image processing device 102"). The image processing device 102 has a model acquisition unit 1502, an object count acquisition unit 1503, a mask generation unit 604, a viewpoint acquisition unit 1506, an image generation unit 1507, and an output unit 605 as its functional configuration. Compared to the image processing device 102 according to the first embodiment, the image acquisition unit 601 is removed, and the viewpoint acquisition unit 1506 and the image generation unit 1507 are added to the image processing device 102. Also, compared to the image processing device 102 according to the first embodiment, the model acquisition unit 602 and the object count acquisition unit 603 are changed to the model acquisition unit 1502 and the object count acquisition unit 1503. Each part of the functional configuration of the image processing device 102 is realized by the CPU 201 executing a program stored in the ROM 203, etc., using the RAM 202 as work memory.

[0087] The model acquisition unit 1502 acquires data from an external device, such as by reading data of a trained three-dimensional field model from the storage device 104. The object count acquisition unit 1503 acquires information about the number of objects specified by the user via a user interface such as the UI panel 103. The mask generation unit 604 generates a three-dimensional mask based on the trained three-dimensional field model data acquired by the model acquisition unit 1502 and the information about the number of objects acquired by the object count acquisition unit 1503. The viewpoint acquisition unit 1506 acquires information about a virtual viewpoint (virtual viewpoint information) specified by the user.

[0088] The image generation unit 1507 generates a virtual viewpoint image based on the trained three-dimensional field model data acquired by the model acquisition unit 1502, the three-dimensional mask data generated by the mask generation unit 604, and the virtual viewpoint information acquired by the viewpoint acquisition unit 1506. The virtual viewpoint image generated by the image generation unit 1507 is displayed in the UI panel 103, etc., as part of the GUI shown as an example in Figure 14. The user can specify an appropriate number of objects by viewing the virtual viewpoint image displayed as part of the GUI.

[0089] <Operation of the image processing device> The operation of the image processing device 102 will be explained with reference to Figure 16. Figure 16 is a flowchart showing an example of the processing flow in the image processing device 102 according to the third embodiment. First, in S1601, the model acquisition unit 1502 acquires data of a trained three-dimensional field model. Next, in S1602, the object number acquisition unit 1503 acquires information by reading information regarding the initial value of the number of objects stored in the ROM 203 or the like. Next, in S704, the mask generation unit 604 performs a three-dimensional mask generation process based on the information regarding the initial value of the number of objects acquired in S1602 and the trained three-dimensional field model data acquired in S1601. The process in S704 is the same as the process in S704 shown in Figure 7(a), so the explanation is omitted. Note that the processes in S1602 and S704 following S1602 are not mandatory and may be omitted.

[0090] Subsequently, in S1610, the object count acquisition unit 1503 determines whether the number of objects has been specified by the user via the user interface and whether new information regarding the number of objects has been acquired. If it is determined in S1610 that new information regarding the number of objects has been acquired, in S704, the mask generation unit 604 performs a three-dimensional mask generation process based on the newly acquired information regarding the number of objects and the trained three-dimensional field model data acquired in S1601. The process in S704 is the same as the process in S704 shown in Figure 7(a), so its explanation is omitted. Subsequently, the image processing device 102 performs the process in S1621. If it is determined in S1610 that new information regarding the number of objects has not been acquired, in S1620, the viewpoint acquisition unit 1506 determines whether the virtual viewpoint has been specified by the user and whether new information regarding the virtual viewpoint (virtual viewpoint information) has been acquired.

[0091] If it is determined in S1620 that new virtual viewpoint information has been acquired, in S1621 the image generation unit 1507 generates a virtual viewpoint image corresponding to the virtual viewpoint information. Specifically, the image generation unit 1507 generates a virtual viewpoint image corresponding to the view from the virtual viewpoint indicated by the acquired virtual viewpoint information, based on the trained three-dimensional field model data acquired in S1601 and the latest three-dimensional mask data generated in S704. The processing in S1621 is the same as the processing in S903 to S907 in the information processing device 108, so the explanation is omitted. After S1621, in S1622 the image processing device 102 outputs the virtual viewpoint image generated in S1621. Specifically, for example, the image processing device 102 outputs the virtual viewpoint image to the GUI, as shown in Figure 14 as an example. By viewing the GUI including the virtual viewpoint image, the user can visually confirm whether the currently generated three-dimensional mask is appropriate. After S1622, the image processing device 102 returns to the process of S1610.

[0092] If it is determined in S1620 that no new virtual viewpoint information has been acquired, in S1630 the output unit 605 receives an instruction from the user to output the data of the three-dimensional field model and the three-dimensional mask data that have been learned, and determines whether or not it has acquired information regarding the said output instruction. If it is determined in S1630 that it has acquired information regarding the output instruction, in S705 the output unit 605 outputs the data of the three-dimensional field model and the three-dimensional mask data to the information processing device 108. After S705, the image processing device 102 returns to the process of S1610.

[0093] If it is determined in S1630 that information regarding the output instruction has not been obtained, in S1640 the image processing device 102 determines whether the user has given an instruction to terminate processing and whether information regarding the termination instruction has been obtained. If it is determined in S1640 that information regarding the termination instruction has not been obtained, the image processing device 102 returns to the process in S1610 and repeatedly executes the processes from S1610 onward until it is determined in S1640 that information regarding the termination instruction has been obtained. If it is determined in S1640 that information regarding the termination instruction has been obtained, the image processing device 102 terminates the process shown in the flowchart in Figure 16.

[0094] Note that the information processing device 108 according to the third embodiment is the same as the information processing device 108 according to the first embodiment, so its description will be omitted. With the image processing system configured as described above, it is possible to suppress the deterioration of image quality in the virtual viewpoint image.

[0095] [Other embodiments] The technology of this disclosure can also be implemented by supplying a program that implements one or more of the functions of the embodiments described above to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. Furthermore, the technology of this disclosure can also be implemented by a processing circuit such as an ASIC that implements one or more functions.

[0096] Furthermore, within the scope of this disclosure, the technologies described herein allow for free combination of each embodiment, modification of any component of each embodiment, or omission of any component in each embodiment.

[0097] [Technical Features of This Disclosure] This disclosure includes the following configurations, methods, and programs.

[0098] <Configuration 1> A model acquisition means for acquiring a trained model relating to a three-dimensional field corresponding to the three-dimensional space, which includes at least information representing the degree of existence of an object in the three-dimensional space at each position in the three-dimensional space. An object count acquisition means for acquiring information regarding the number of objects present in the three-dimensional space, A three-dimensional mask generation means generates a three-dimensional mask that masks the positions in the three-dimensional field corresponding to the positions in the three-dimensional space where the object does not exist, based on the information representing the degree of existence and the information regarding the number of objects. It has, The three-dimensional mask generation means generates the three-dimensional mask such that the number of clusters remaining when the clusters obtained based on the information representing the degree of existence are masked by the three-dimensional mask matches or approaches the number of objects. An image processing device characterized by the following.

[0099] <Configuration 2> The three-dimensional mask generation means generates the three-dimensional mask such that, when the clusters obtained based on the information representing the degree of existence are masked by the three-dimensional mask, the clusters with larger volumes are preferentially retained. An image processing apparatus according to configuration 1, characterized by the above.

[0100] <Structure 3> The aforementioned trained model is obtained as a result of training a model relating to the three-dimensional field using the data of the captured images obtained by imaging the three-dimensional space and the camera parameters in the imaging. The model acquisition means acquires the trained model from an external device. An image processing apparatus according to configuration 1 or 2, characterized by the above.

[0101] <Structure 4> The trained model is obtained as a result of training the model for the three-dimensional field based on the difference between the image obtained by volume rendering the model for the three-dimensional field using the camera parameters and the captured image corresponding to the camera parameters. An image processing apparatus according to configuration 3, characterized by the above.

[0102] <Composition 5> Image acquisition means for acquiring data of an image obtained by imaging the three-dimensional space, and camera parameters corresponding to the imaging. It further possesses, The model acquisition means acquires the trained model by training the model relating to the three-dimensional field using the captured image and the camera parameters. An image processing apparatus according to any one of configurations 1 to 4 characterized by the above.

[0103] <Composition 6> The model acquisition means performs training on the model relating to the three-dimensional field based on the difference between an image obtained by volume rendering the model relating to the three-dimensional field using the camera parameters and the captured image corresponding to the camera parameters. The image processing apparatus described in configuration 5, characterized by the above.

[0104] <Composition 7> Image acquisition means for acquiring data of an image obtained by imaging the three-dimensional space, and camera parameters corresponding to the imaging. It further possesses, The model acquisition means updates the trained model by retraining the trained model using the captured image, the camera parameters, and the three-dimensional mask. An image processing apparatus according to any one of configurations 1 to 6 characterized by the above.

[0105] <Structure 8> The model acquisition means retrains the trained model based on the difference between an image obtained by volume rendering the three-dimensional field corresponding to the trained model, masked by the three-dimensional mask, using the camera parameters, and the captured image corresponding to the camera parameters. The image processing apparatus according to configuration 7, characterized by the above.

[0106] <Composition 9> The object count acquisition means acquires information regarding the number of objects specified by the user via the user interface. An image processing apparatus according to any one of configurations 1 to 8 characterized by the above.

[0107] <Composition 10> The object count acquisition means acquires information regarding the number of objects by acquiring the three-dimensional shape of the objects and identifying the number of clusters obtained by clustering the three-dimensional shapes. An image processing apparatus according to any one of configurations 1 to 9 characterized by the above.

[0108] <Composition 11> The three-dimensional shape is obtained by estimation using the data of the captured image obtained by imaging the three-dimensional space and the camera parameters in the imaging. An image processing apparatus according to configuration 10, characterized by the above.

[0109] <Composition 12> The three-dimensional shape is obtained by estimation using the viewing volume cross-section method with the data of the captured image and the camera parameters. An image processing apparatus according to configuration 11, characterized by the above.

[0110] <Composition 13> Image acquisition means for acquiring the data of the captured image and the camera parameters, It further possesses, The object count acquisition means acquires the three-dimensional shape by estimating the shape of the object using the data of the captured image and the camera parameters. An image processing apparatus according to configuration 11 or 12, characterized by the above.

[0111] <Composition 14> The object count acquisition means clusters the three-dimensional shape after the three-dimensional shape has been contracted. An image processing apparatus according to any one of configurations 10 to 13 characterized by the above.

[0112] <Composition 15> The object count acquisition means generates a provisional three-dimensional mask based on the cluster obtained based on the information representing the degree of existence, and shrinks the three-dimensional shape after masking the three-dimensional shape using the generated provisional three-dimensional mask. An image processing apparatus according to configuration 14, characterized by the above.

[0113] <Method> A model acquisition step involves acquiring a trained model relating to a three-dimensional field corresponding to the three-dimensional space, which includes at least information representing the degree of existence of an object in the three-dimensional space at each position in the three-dimensional space. An object count acquisition step for acquiring information regarding the number of objects present in the three-dimensional space, A three-dimensional mask generation step generates a three-dimensional mask that masks the positions in the three-dimensional field corresponding to the positions in the three-dimensional space where the object does not exist, based on the information representing the degree of existence and the information regarding the number of objects. Includes, The three-dimensional mask generation step involves generating the three-dimensional mask such that the number of clusters remaining when the clusters obtained based on the information representing the degree of existence are masked by the three-dimensional mask matches or approaches the number of objects. An image processing method characterized by the following.

[0114] <Program> A program for causing a computer to function as an image processing device described in any one of configurations 1 to 15. [Explanation of Symbols]

[0115] 102 Image Processing Device 602 Model Acquisition Section 603 Object Count Acquisition Section 604 Mask Generation Unit

Claims

1. Model acquisition means for acquiring a trained model relating to a three-dimensional field corresponding to the three-dimensional space, which includes at least information representing the degree of existence of an object existing in the three-dimensional space at each position in the three-dimensional space, An object count acquisition means for acquiring information regarding the number of objects present in the three-dimensional space, A three-dimensional mask generation means generates a three-dimensional mask that masks the positions in the three-dimensional field corresponding to the positions in the three-dimensional space where the object does not exist, based on the information representing the degree of existence and the information regarding the number of objects. It has, The three-dimensional mask generation means generates the three-dimensional mask such that the number of clusters remaining when the clusters obtained based on the information representing the degree of existence are masked by the three-dimensional mask matches or approaches the number of objects. An image processing device characterized by the following.

2. The three-dimensional mask generation means generates the three-dimensional mask such that, when the clusters obtained based on the information representing the degree of existence are masked by the three-dimensional mask, clusters with larger volumes are preferentially retained. The image processing apparatus according to claim 1, characterized in that

3. The aforementioned trained model is obtained as a result of training a model relating to the three-dimensional field using the data of the captured images obtained by imaging the three-dimensional space and the camera parameters in the imaging. The model acquisition means acquires the trained model from an external device. The image processing apparatus according to claim 1, characterized in that

4. The trained model is obtained as a result of training the model for the three-dimensional field based on the difference between the image obtained by volume rendering the model for the three-dimensional field using the camera parameters and the captured image corresponding to the camera parameters. The image processing apparatus according to claim 3, characterized by the following:

5. Image acquisition means for acquiring data of an image obtained by imaging the three-dimensional space, and camera parameters corresponding to the imaging. It further possesses, The model acquisition means acquires the trained model by training a model relating to the three-dimensional field using the captured image and the camera parameters. The image processing apparatus according to claim 1, characterized in that

6. The model acquisition means performs training on the model relating to the three-dimensional field based on the difference between an image obtained by volume rendering the model relating to the three-dimensional field using the camera parameters and the captured image corresponding to the camera parameters. The image processing apparatus according to claim 5, characterized by the following:

7. Image acquisition means for acquiring data of an image obtained by imaging the three-dimensional space, and camera parameters corresponding to the imaging. It further possesses, The model acquisition means updates the trained model by retraining the trained model using the captured image, the camera parameters, and the three-dimensional mask. The image processing apparatus according to claim 1, characterized in that

8. The model acquisition means retrains the trained model based on the difference between an image obtained by volume rendering the three-dimensional field corresponding to the trained model, masked by the three-dimensional mask, using the camera parameters, and the captured image corresponding to the camera parameters. The image processing apparatus according to claim 7, characterized by the following:

9. The object count acquisition means acquires information regarding the number of objects specified by the user via the user interface. The image processing apparatus according to claim 1, characterized in that

10. The object count acquisition means acquires information regarding the number of objects by acquiring the three-dimensional shape of the objects and identifying the number of clusters obtained by clustering the three-dimensional shapes. The image processing apparatus according to claim 1, characterized in that

11. The three-dimensional shape is obtained by estimation using the data of the captured image obtained by imaging the three-dimensional space and the camera parameters in the imaging. The image processing apparatus according to claim 10, characterized by the above.

12. The three-dimensional shape is obtained by estimation using the viewing volume cross-section method with the data of the captured image and the camera parameters. The image processing apparatus according to claim 11, characterized in that

13. Image acquisition means for acquiring the data of the captured image and the camera parameters, It further possesses, The object count acquisition means acquires the three-dimensional shape by estimating the shape of the object using the data of the captured image and the camera parameters. The image processing apparatus according to claim 11, characterized in that

14. The object count acquisition means clusters the three-dimensional shape after the three-dimensional shape has been contracted. The image processing apparatus according to claim 10, characterized by the above.

15. The object count acquisition means generates a provisional three-dimensional mask based on the cluster obtained based on the information representing the degree of existence, and shrinks the three-dimensional shape after masking the three-dimensional shape using the generated provisional three-dimensional mask. The image processing apparatus according to claim 14, characterized by the above.

16. A model acquisition step involves acquiring a trained model relating to a three-dimensional field corresponding to the three-dimensional space, which includes at least information representing the degree of existence of an object in the three-dimensional space at each position in the three-dimensional space. An object count acquisition step for acquiring information regarding the number of objects present in the three-dimensional space, A three-dimensional mask generation step generates a three-dimensional mask that masks the positions in the three-dimensional field corresponding to the positions in the three-dimensional space where the object does not exist, based on the information representing the degree of existence and the information regarding the number of objects. Includes, The three-dimensional mask generation step involves generating the three-dimensional mask such that the number of clusters remaining when the clusters obtained based on the information representing the degree of existence are masked by the three-dimensional mask matches or approaches the number of objects. An image processing method characterized by the following.

17. A program for causing a computer to function as an image processing device according to any one of claims 1 to 15.

Images