Image processing device, image processing method, and program

Abstract

This refers to a decrease in the accuracy of learning the three-dimensional shape of an object. [Solution] The image processing device includes an acquisition means for acquiring multiple captured images obtained by capturing an object present in the learning space from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints; a setting means for setting a color for adjusting the color obtained as a result of inputting viewpoint information of one of the multiple viewpoints into spatial information relating to the learning space, such that the color is different from the color of the object; and a learning means for learning the spatial information based on the multiple captured images, camera parameters corresponding to each of the multiple viewpoints, and the set color information for adjustment.

Description

[Technical Field]

[0001] This disclosure relates to a technique for estimating information about three-dimensional space. [Background technology]

[0002] There is a technique that generates an image corresponding to how a target object would appear from a virtual viewpoint, based on multiple images obtained by capturing from multiple viewpoints. The image corresponding to how the object would appear from a virtual viewpoint is called a virtual viewpoint image. One method for generating a virtual viewpoint image is to estimate the radiance field of the space containing the target object and use the estimated radiance field to generate a virtual viewpoint image corresponding to how the object would appear when viewed from that virtual viewpoint.

[0003] Non-patent document 1 describes a method for estimating radiance fields using NeRF (Neural Radiance Fields), which is composed of a deep learning network called MLP (Multi-layer Perceptron). NeRF is a neural network that, when given the position coordinates of a point in three-dimensional space and the direction in which that point is observed, outputs the volume density (opacity) and color (RGB) for that point. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] NeRF:Representing Scene As Neural Radiance Fields For View Synthesis [Overview of the project] [Problems that the invention aims to solve]

[0005] Non-patent document 1 describes generating an output image using the same viewpoint as the viewpoint used when the captured image was taken, and training the MLP to minimize the difference (loss) between the output image and the captured image used as the training image.

[0006] Regarding the color values ​​of the output image, one possible method is to input viewpoint information into a model representing a three-dimensional space constructed using MLP (Multi-Level Programming), perform volume rendering using the output results, and then blend the resulting colors with a pre-set color. This pre-set color is called the training background color.

[0007] However, when training with a background color set during training, output images that should have a large loss between the model and the captured image may be generated as output images with a small loss between the model and the captured image. For example, suppose the training results in a model with missing 3D shapes where objects are not present in the expected locations. Even in this case, if viewpoint information is input to the model and the output result is used for volume rendering, and the training background color is applied to the resulting image, an output image may be obtained in which the training background color is drawn in the missing regions of the object. Since this output image has a small loss, the training results may converge to a model where object information is absent in the locations where it should be. [Means for solving the problem]

[0008] The image processing apparatus of the present disclosure is characterized by comprising: acquisition means for acquiring a plurality of captured images obtained by capturing an object present in a learning space from a plurality of viewpoints, and camera parameters corresponding to each of the plurality of viewpoints; setting means for setting a color for adjusting the color obtained as a result of inputting viewpoint information of one of the plurality of viewpoints into spatial information relating to the learning space, such that the color is different from the color of the object; and learning means for learning the spatial information based on the plurality of captured images, camera parameters corresponding to each of the plurality of viewpoints, and the set color information for adjustment. [Effects of the Invention]

[0009] According to this disclosure, it is possible to suppress the decrease in the accuracy of learning the three-dimensional shape of an object. [Brief explanation of the drawing]

[0010] [Figure 1] A diagram showing the configuration of an image processing system. [Figure 2] A diagram showing the hardware configuration of an image processing device. [Figure 3] A block diagram showing an example of the functional configuration of an image processing device. [Figure 4] A flowchart illustrating the processing flow in an image processing device. [Figure 5] A diagram illustrating how pixel color is determined. [Figure 6] A diagram illustrating a comparative example of captured images, output images, and training results. [Figure 7] A flowchart to explain the process of setting the background color during learning. [Figure 8] A diagram illustrating the captured image, output image, and training results. [Figure 9] A flowchart illustrating the processing flow in an image processing device. [Figure 10] A diagram illustrating the captured image, background image, and foreground image. [Figure 11]A flowchart to explain the process of setting the background color during learning. [Figure 12] A diagram to explain the output image. [Figure 13] A flowchart to explain the process of setting the background color during learning. [Figure 14] A figure showing an example of a histogram. [Figure 15] A flowchart to explain the process of setting the background color during learning. [Modes for carrying out the invention]

[0011] Embodiments of the technology of this disclosure will be described below with reference to the drawings. Note that the following embodiments are not limiting to the technology of this disclosure, and not all combinations of features described in the following embodiments are necessarily essential to the solutions of the technology of this disclosure. Identical components will be denoted by the same reference numerals.

[0012] <Embodiment 1> [System Configuration] Figure 1 shows an example of the configuration of an image processing system that generates a virtual viewpoint image by capturing an object with multiple imaging devices. The image processing system of this embodiment has multiple imaging devices 101a to h, an image processing device 110, a UI panel 130, a storage device 140, and a display device 150.

[0013] The multiple imaging devices 101a to h are a group of cameras and other devices that image objects 106 in the imaging space 105 according to set imaging conditions. The position and orientation of the imaging devices 101a to h are assumed to be fixed. The multiple imaging devices 101a to h are connected to the image processing device 110 via LAN. Based on control signals from the image processing device 110, the multiple imaging devices 101a to h start imaging, stop imaging, change imaging settings such as shutter speed and aperture, and transfer the data of the image obtained from imaging. The number of multiple imaging devices 101a to h is not limited to the number shown in the figure. One of the multiple imaging devices 101a to h may be referred to as imaging device 101.

[0014] The image processing device 110 is an information processing device that estimates the radiance field of the learning space through learning. The image processing device 110 is a device that learns the parameters of the MLP using images from multiple viewpoints obtained by synchronous imaging by multiple imaging devices 101a to h in order to estimate the radiance field of the learning space. Furthermore, the image processing device 110 uses the MLP learning results to generate a virtual viewpoint image, which is an image of an object viewed from a viewpoint different from the viewpoint from the actual imaging device (called a virtual viewpoint).

[0015] The UI panel 130 is a user interface panel such as a touch panel and is an input device that accepts user operations. The UI panel 130 accepts user operations and, based on the content of the accepted operations, inputs the settings for imaging conditions, learning space, and processing conditions to the image processing device 110. The UI panel 130 also accepts input of viewpoint information for virtual viewpoints when generating virtual viewpoint images. It is not necessarily required for the user to operate the UI panel 130 in order to input to the image processing device 110. For example, the image processing device 110 may be connected to other input devices such as a mouse or keyboard, in which case the user may input to the image processing device 110 via these other input devices.

[0016] The memory device 140 stores images from multiple viewpoints obtained by synchronous imaging by multiple imaging devices 101a to h, as well as information about a model for estimating the radiance field, which is the learning result of the MLP.

[0017] The display device 150 displays a virtual viewpoint image generated by the image processing device 110 based on a virtual viewpoint camera path configured with a set virtual viewpoint. The image processing system includes various other components besides the devices described above, but their explanation is omitted here.

[0018] [Hardware configuration] Figure 2 shows an example of the hardware configuration of the image processing device 110. The image processing device 110 has a CPU 201, main memory 202, storage unit 203, input unit 204, display unit 205, and external I / F unit 206, and each unit is connected via a bus 207.

[0019] The CPU 201 is an arithmetic processing unit that comprehensively controls the image processing device 110 and performs various processing by executing various programs stored in the memory unit 203, etc. The image processing device 110 may have one or more dedicated hardware components different from the CPU 201, and the dedicated hardware components may be configured to perform at least a portion of the processing performed by the CPU 201. Examples of dedicated hardware components include GPUs (Graphics Processing Units), ASICs (Application-Specific Integrated Circuits), FPGAs (Field-Programmable Gate Arrays), and DSPs (Digital Signal Processors).

[0020] Main memory 202 is, for example, RAM, which temporarily stores data and parameters used in various processes, information for setting up the learning space, and also provides a workspace for the CPU 201.

[0021] The memory unit 203 is a large-capacity storage device that stores various data necessary for various programs and GUI (Graphical User Interface) display, and is a non-volatile memory such as ROM, hard disk, or silicon disk. The memory unit 203 stores camera parameters corresponding to each image captured by the multiple imaging devices 101a to h.

[0022] Camera parameters are defined as parameters including external parameters, internal parameters, and distortion parameters that indicate the position and orientation of the imaging device 101. Camera parameters can be obtained by camera calibration. One method of camera calibration involves attaching a piece of paper printed with a specific pattern to a flat board, detecting feature points from the image obtained by the imaging device 101 capturing an image of the board, and obtaining camera parameters by analyzing the coordinates of the feature points.

[0023] The input unit 204 consists of devices such as a keyboard, mouse, electronic pen, and touch panel, and accepts user input. The display unit 205 consists of an LCD panel and performs GUI display of learning results. The external I / F unit 206 is connected to each imaging device 101 via LAN 208 and transmits and receives captured image data and control signal data. The bus 207 connects the above-mentioned units and performs data transfer.

[0024] The CPU 201 operates as a display control unit that controls the screens displayed on the display unit 205, UI panel 130, and display device 150, and as an input control unit that controls the input unit 204. In this embodiment, the display unit 205 and the input unit 204 have been described as being included in the image processing device 110, but at least one of the display unit 205 and the input unit 204 may exist as a separate device outside the image processing device 110.

[0025] [Functional Configuration] Figure 3 is a block diagram showing the functional configuration of the image processing apparatus 110 in this embodiment. The image processing apparatus 110 in this embodiment includes a learning data acquisition unit 301, an object color acquisition unit 302, a learning background color setting unit 303, and a learning unit 304.

[0026] The learning data acquisition unit 301 acquires multiple captured images obtained by multiple imaging devices 101a to h, and the camera parameters of each of the multiple imaging devices 101a to h stored in the storage unit 203. The learning data acquisition unit 301 acquires camera parameters obtained by a camera calibration performed in advance. Alternatively, camera calibration may be performed using the multiple captured images acquired by the learning data acquisition unit 301, and camera parameters may be obtained as a result.

[0027] The object color acquisition unit 302 acquires the color value of each pixel and its pixel value in the multiple captured images acquired by the learning data acquisition unit 301.

[0028] The learning background color setting unit 303 sets a learning background color for each pixel in the multiple captured images acquired by the object color acquisition unit 302. The learning background color is a color used to adjust the color obtained by volume rendering using the result of inputting viewpoint information into spatial information related to the learning space. In this embodiment, the learning background color is set to be a different color from the color indicated by the color value acquired by the object color acquisition unit 302. Details will be described later.

[0029] The learning unit 304 learns the radiance field of the learning space based on camera parameters acquired by the learning data acquisition unit 301, multiple captured images obtained by multiple imaging devices 101, and the learning background color set by the learning background color setting unit 303. Based on the learning results, the learning unit 304 outputs a trained model that outputs a virtual viewpoint image corresponding to viewpoint information when viewpoint information is input. Details will be described later.

[0030] In this embodiment, the entire imaging space is treated as the learning space, and the entire scene is described as an object. Alternatively, only a portion of the imaging space may be treated as the learning space. Alternatively, the imaging space may be divided and multiple learning spaces may be set up. When a portion of the imaging space is treated as the learning space, or when the imaging space is divided and multiple learning spaces are set up, the learning data acquisition unit 301 acquires information for setting up the learning space (hereinafter referred to as "learning space information"). The learning space information is, for example, text data containing position coordinates in three-dimensional space, or data input by user operation at the input unit 204.

[0031] Each functional unit within the image processing device 110 in Figure 3 is realized by the CPU 201 executing a predetermined program, but is not limited to this. Other hardware, such as a GPU (Graphics Processing Unit) or FPGA (Field Programmable Gate Array), may also be used to speed up calculations. Each functional unit may be realized through the cooperation of software and hardware such as dedicated ICs, or some or all of the functions may be realized by hardware alone.

[0032] [flowchart] Figure 4 is a flowchart illustrating the processing flow performed by the image processing device 110. The series of processes shown in the flowchart of Figure 4 are performed by the CPU 201 of the image processing device 110 loading the program code stored in the memory unit 203 into the main memory 202 and executing it. Some or all of the functions of the steps in Figure 4 may also be implemented by hardware such as an ASIC or electronic circuit. The symbol "S" in the description of each process means that it is a step in that flowchart, and the same applies to subsequent flowcharts.

[0033] In S401, the CPU 201, as part of the processing of the learning data acquisition unit 301, acquires multiple captured images obtained by the multiple imaging devices 101 synchronously, as well as the camera parameters of each of the multiple imaging devices 101. When the CPU 201 sends an imaging command to the multiple imaging devices 101 via the LAN 208, the multiple imaging devices 101 perform imaging at the same time. The multiple imaging devices 101 transmit the resulting captured images to the image processing device 110. In the image processing device 110, the received captured images are stored in the main memory 202 via the LAN 208, the external I / F unit 206, and the bus 207. Camera parameters are acquired from those previously stored in the storage unit 203, etc.

[0034] In S402, the CPU 201, as part of the processing of the object color acquisition unit 302, acquires the color values ​​contained in each pixel value of the multiple captured images acquired in S401. In this embodiment, the entire scene captured in the image is considered an object, and the color values ​​of all pixels in the multiple captured images are acquired.

[0035] In S403, the CPU 201 sets a training background color for each pixel acquired in S402 as part of the processing of the training background color setting unit 303. In this embodiment, a training background color corresponding to each pixel of the output image is set. That is, the training background color is set on a pixel-by-pixel basis. The training background color is set to a color such that, for example, the color difference from the color in the training image is large in a predetermined color space. Details of S403 will be described later.

[0036] In S404, the CPU201 estimates the radiance field as spatial information of the learning space as part of the processing of the learning unit 304.

[0037] If the learning space is a part of the imaging space, CPU201 acquires information about the learning space. Based on the information about the learning space, CPU201 sets the estimated range of the radiance field.

[0038] The CPU 201 estimates the radiance field as spatial information of the learning space based on the camera parameters acquired in S401, the captured images obtained by the imaging devices 101a to h, and the learning background color set in S403. Estimating the radiance field of the learning space means estimating the color and volume density of objects in the learning space. For example, the CPU 201 uses the function F shown in the following equation. Θ Using this method, we estimate the color and volume density of a three-dimensional space modeled after the learning space.

[0039] F Θ :(x,y,z,θ,φ)→(R,G,B,σ) (x,y,z) are position coordinates in three-dimensional space, (θ,φ) are directions in three-dimensional space, and (R,G,B) represent color values ​​indicating the object's color determined by its position and direction. Furthermore, σ (volumetric scene density) represents the volumetric density of the object, determined by its position. F Θ This is a function that outputs values ​​representing color and volume density for values ​​representing position and direction in three-dimensional space. Function F Θ The set of color information and volume density information in the learning space represented by is the radiance field. In this embodiment, the function F Θ This will be explained as an implementation using an MLP (Multi-layer perceptron), and the estimation of the radiance field by learning the MLP parameters through deep learning. In this case, the radiance field is represented as the MLP parameters, that is, as the weight parameters related to the nodes that constitute the MLP. Thus, estimating the radiance field means using multiple captured images to learn the MLP parameters, and estimating the corresponding color and volume density when an arbitrary position (x, y, z) and direction (θ, φ) are specified in the three-dimensional space representing the learning space.

[0040] The CPU 201 learns the parameters of the MLP by repeatedly executing the following series of processes, and obtains a model that, when inputting the position coordinates of a point in three-dimensional space and the direction of observing that point, outputs the volume density and color for that point. As a result, when inputting viewpoint information, a model is obtained that outputs a virtual viewpoint image corresponding to the viewpoint information by weighting and accumulating the colors output along the rays determined based on the position of each pixel and the viewpoint information with the volume density.

[0041] In the present embodiment, the captured image acquired in S401 is used as a learning image (also referred to as a teacher image or a correct image) during learning. Also, the viewpoint information of the imaging device 101 that captured the learning image is input to the MLP, and an image obtained using the output result thereof is used as an output image. Then, by learning the parameters of the MLP so that an output image with less difference (loss) between the learning image and the output image is output, a learned model that outputs a virtual viewpoint image when inputting viewpoint information is generated. In the present embodiment, the output image is an image drawn with the color obtained by alpha-blending the color obtained by volume rendering by inputting the viewpoint information to the MLP with the background color at the time of learning set in S403. Hereinafter, a specific method of learning by the learning unit 304 will be described.

[0042] First, a ray r in the direction of each pixel (u k , v k ) of an image representing what can be seen from the position and line-of-sight direction indicated by the viewpoint information of the viewpoint k of the imaging device that captured the imaging image serving as the learning image is emitted.

[0043] Next, a plurality of points are sampled on the ray r, and the color c i and the volume density σ i at each sampled point i are output from the MLP. The color C A (r) for each ray r corresponding to each pixel is calculated by volume rendering that weights and accumulates the colors c i of the sampled points output along the ray r with the volume density σ i . The color C obtained by volume rendering for the ray rA (r) is calculated using the following formulas (1), (2), and (3).

[0044]

number

[0045] C A (r) is a pixel (u k ,v k Color c on the light ray r passing through ) i and volume density σ i These are RGB values ​​that represent the color estimated by volume rendering using the following method: i is the index of the sampling point on the ray r. N is the number of sampling points. c i This is the RGB value that represents the color at sampling point i. σ i This is the volume density at sampling point i. δ j This is the distance from the j-th sampling point to the (j+1)th sampling point. α i This is the opacity along the sampling point i to i+1. T i This is the cumulative transmittance up to sampling point i.

[0046] Cumulative transmittance T i This value, calculated as the accumulated opacity, adjusts how much of the object's color is reflected in the color value. By alpha blending the background color used during training when calculating the color value, it is a value used to reproduce transparent and semi-transparent objects.

[0047] Next, we determine the color values ​​of the output image to be compared with the training image from viewpoint k. Specifically, we determine the color C obtained by volume rendering. A (r) The learning background color C set in S403 bgThe color C(r) of the output image is calculated by alpha blending using equation 4. In equation 4, α(r) is the α of the sampling point on the ray r, as shown in equation (5). i This is the cumulative opacity.

[0048] CCC A (r) + (1-α(r))C bg (4)

[0049]

number

[0050] In this embodiment, the background color C during learning is used. bg This is the pixels (u) of the output image and training image corresponding to viewpoint k. k ,v k This is set for each pixel (u) of the image that represents the view from viewpoint k. k ,v k ) Color value C(u k ,v k If we represent it as ), the background color during learning is C bg (u k ,v k ) is expressed as follows. Therefore, in this embodiment, equation (4) can be expressed by the following equation (4)'.

[0051] C(u k ,v k )=C A (r) + (1-α(r))C bg (u k ,v k ) (4)´

[0052] Figure 5 shows the pixels (u) of the output image based on the color obtained by volume rendering and the background color during training. k ,v k This is a diagram to explain how the color values ​​of ) are determined.

[0053] The output image 501 in Figure 5(a) is at the position P of the viewpoint k of the imaging device. CAMkThis is an image of the three-dimensional space 503, which was estimated from the learning space. Object 500 is an object in the three-dimensional space 503 estimated as a result of the learning process. u,v This is the pixel (u) of output image 501. k ,v k This shows the light ray passing through ). Sampling point 504 is the light ray r u,v The sampling point i is shown above.

[0054] Figure 5(b) is a view of Figure 5(a) along the y-axis. In Figure 5(b), the light ray r passing through the pixel (u0,v0) u0,v0 and the light ray r passing through the pixel (u2,v2) u2,v2 This indicates a ray of light intersecting with the opaque object 500. In the estimated three-dimensional space 503, the opacity α is "1" when an opaque object of color exists on the ray r. Therefore, the training background color C in equation (4) bg The weight related to this becomes "0". Therefore, the color C(u0,v0) of the pixel (u0,v0) and the color C(u2,v2) of the pixel (u2,v2) that represent the object in the three-dimensional space 503 in the output image are determined by the following equation, which is the learning background color C bg It is calculated in a way that does not affect it.

[0055] C(u0,v0)=C A (u0,v0) C(u2,v2)=C A (u2,v2)

[0056] On the other hand, in Figure 5(b), the light ray r passing through the pixel (u1,v1) u1,v1 It does not intersect with object 500 in the estimated three-dimensional space 503. Therefore, the color C obtained by volume rendering at pixel (u1,v1) is not obtained. AThe value of is "0" because the volume density is "0", and therefore, according to equation (1), it represents transparency. Also, if there are no objects on the light ray r within the estimated three-dimensional space 503, the opacity α on the light ray r is "0". Therefore, the color C(u1,v1) of the pixel (u1,v1) at a position where there are no objects in the three-dimensional space 503 in the output image is calculated as follows.

[0057] C(u1,v1)=C bg (u1,v1)

[0058] Thus, the light r u1,v1 The color values ​​of the pixels in the output image when the object does not intersect are the background color C used during training. bg This will be used.

[0059] The signal consisting of the RGB values ​​of all pixels in the output image generated in this way is used as the prediction signal, and the signal consisting of the RGB values ​​of all pixels in the captured image (training image) taken at the position and orientation indicated by the input viewpoint information to obtain the output image is used as the training signal. The CPU 201 updates the parameters of the MLP, which are the network weights, to minimize the difference between the color values ​​of the output image and the color values ​​of the ground truth image, for example, by using backpropagation, where the squared Euclidean distance between the prediction signal and the training signal is the loss function. For example, training is terminated when the parameters have been updated a set number of times or when a set training time has elapsed. This completes the training process.

[0060] In S405, CPU201 outputs a model containing the MLP weight parameters obtained through training in S404 as a trained model.

[0061] In the inference phase, when arbitrary viewpoint information is input to the trained model output in S405, the pixel color of the virtual viewpoint image, which is the output image, is determined based on equation (4). In the inference phase, C in equation (4) bg For example, one color (such as white) is used.

[0062] [Assuming the background color during training is the same as the object's color] As mentioned above, color values ​​obtained by volume rendering can sometimes represent transparency. Therefore, in this embodiment, a training background color is set in order to accurately estimate the radiance field. However, if the training background color is not set appropriately, training may fail.

[0063] Figure 6 illustrates a comparative example of a method for setting the background color during training. Figure 6 describes a model that can be generated when the background color during training is uniformly set for all pixels to a color close to the color of an object contained in the captured image, which is the training image. Specifically, it describes what kind of model can be obtained as a result of training the MLP parameters when there is a white object in the imaging space and white or a color close to white is set as the background color during training in S403.

[0064] In Figure 6(a), (a1) shows the actual imaging space containing the white object 611. Therefore, as shown in Figure 6(a), the color of the object region 613 included in the captured image, which is the training image 612, is also white.

[0065] Figure 6(b) (b1) shows a model in which, as a result of training the MLP, the three-dimensional shape 621 of the white object is obtained without loss in the three-dimensional space representing the training space. The output image 622 shown in Figure 6(b) is obtained by inputting the viewpoint information at the time of acquisition of the training image 612 into the model shown in Figure 6(a) (b1), and determining the color according to equation (4). The object region 623 included in this output image 622 is the actual white object itself. Therefore, the loss between the output image 622 and the training image 612 is calculated to be small.

[0066] Figure 6(c) (c1) shows a model in which, as a result of training the MLP, the three-dimensional shape 631 of the object is obtained in three-dimensional space, including a hole that was not present in the actual object. The output image 632 shown in Figure 6(c) is obtained by inputting the viewpoint information at the time of capturing the training image 612 into the model shown in Figure 6(c) (c1), and determining the color according to equation (4). In the model in Figure 6(c) (c1), the region of the output image 632 corresponding to the region where the three-dimensional shape 631 of the object is obtained is the object itself. Also, in the region 633 of the output image 632 corresponding to the region where a hole is formed on the object in the model in Figure 6(c) (c1), the white background color used during training is drawn.

[0067] Using Figure 5(c), we will explain why the perforated region 633 in the output image 632 is drawn with the training background color. If the pixel position of the perforated region in the output image 632 is (u2,v2), then the light ray r passing through (u2,v2) u2,v2 The pixel does not intersect with the three-dimensional shape 631 of the object on the model. Therefore, as mentioned above, the color C(u2,v2) of the pixel (u2,v2) that does not intersect with the object is, from Equation 4, the background color C during training. bg It is drawn in white.

[0068] Models like (c1) in Figure 6(c) are obtained as a result of failed MLP parameter training. Therefore, by making the loss between the output image 632 produced by a model like (c1) in Figure 6(c) and the training image 612 large, the output of a model like (c1) in Figure 6(c) as a training result can be suppressed. However, if a color close to the object's color is set as the background color during training, the loss between the output image and the training image may be calculated as small even if there is missing information in the three-dimensional shape of the object in the model. As a result, the MLP parameters may converge to either the model (b1) in Figure 6(b) or the model (c1) in Figure 6(c) as the MLP training result. If the parameters converge to the model shown in (c1), a model with missing information in the three-dimensional shape will be output as the training result. Therefore, it is necessary to set the background color during training appropriately.

[0069] [Regarding background color setting during training] Figure 7 is a flowchart illustrating the details of the background color setting process during learning in S403.

[0070] In S701, CPU201 obtains the maximum and minimum values ​​that can be represented by the color depth of the acquired image's color values. For example, if the color depth of the acquired image is 8 bits, the maximum value of the pixel value for each channel is 2 8 -1 = 255, the minimum value is 2 0 -1 = 0 is obtained.

[0071] In S702, CPU201 calculates the average of the maximum and minimum values ​​obtained in S701, and determines the threshold value from this average. For example, if the color depth of the image is 8 bits, the maximum pixel value for each channel is "255" and the minimum value is "0". Therefore, the average value is (255 + 0) / 2 = 127.5, and CPU201 determines the threshold value to be "127.5".

[0072] S703 to S709 is a loop process (loop 1). In S703, CPU201 selects pixels to be processed from all pixels of the multiple captured images acquired in S402. Then CPU201 executes the processes in S704 to S708 for the pixels to be processed. In S709, if CPU201 determines that there are pixels that have not been selected for processing (S709 is NO), it returns to S703, selects new pixels to be processed from the unselected pixels, and repeats the processes in S704 to S708.

[0073] S704-S708 is a loop process (loop 2). In S704, CPU201 selects the channels to be processed from among the channels that make up the color value. Then CPU201 executes the processes in S705-S707 for the channels to be processed in loop 2 for the pixels to be processed in loop 1. In S708, if CPU201 determines that there are channels that have not been selected as the target of processing (S708 is NO), it returns to S704, selects a new channel to be processed from among the unselected channels, and repeats the processes in S705-S707. In S708, if CPU201 determines that all channels have been processed (S708 is YES), the learning background color corresponding to the pixels to be processed has been set, and the process proceeds to S709.

[0074] In S705, CPU201 determines whether the signal value of the channel to be processed in the pixel to be processed is greater than the threshold determined in S702. If CPU201 determines that the signal value is greater than the threshold (S705 is YES), the process proceeds to S707. In S707, CPU201 sets the signal value of the channel to be processed in the learning background color corresponding to the pixel to be processed to the minimum value that can be represented by the color depth. If the color depth of the image is 8 bits, it is set to "0".

[0075] On the other hand, if the CPU determines that the signal value is below the threshold (S705 is NO), it proceeds to S706. In S706, CPU201 sets the signal value of the channel to be processed in the learning background color corresponding to the pixel to be processed to the maximum value that can be represented by the color depth. If the color depth of the image is 8 bits, it is set to "255".

[0076] For example, if the color value is an RGB value, the signal values ​​of the channels representing the color value are expressed as (signal value of the R channel, signal value of the G channel, signal value of the B channel). Then, the pixels to be processed are the pixels (u) of the captured image captured from viewpoint k. k ,v k ) is selected, and pixels (u k ,v k Let's assume the color value of the background color is (70, 220, 60). In this case, the signal value of the G channel, "220," is greater than the threshold of "127.5," so in S707, the signal value of the G channel of the background color during training is set to "0." Also, since the signal values ​​of the R channel and B channel are below the threshold, the process proceeds to S706, where the signal values ​​of the R channel and B channel of the background color during training are set to "225."

[0077] As a result, pixels (u) of the captured image taken from viewpoint k k ,v k ) corresponds to learning background color C bg (u k ,v k ) is set to (255, 0, 255). In the learning process of S404, the pixels (u) of the output image corresponding to viewpoint k are set. k ,v k ) Color C(u k ,v k ) is obtained by equation (4)', and as a result, the output image is obtained. Therefore, the background color C during training is bg (u k ,v k It can also be said that this is set for each pixel of the output image.

[0078] In S709, if CPU201 determines that it has set a training background color for all pixels (S709 is YES), it terminates the training background color setting process.

[0079] Thus, in this embodiment, a training background color is set for each pixel of the training image (output image) such that the difference in color value between the training image and the captured image is large for each pixel. As a result, a color with a larger color difference than the color of object 106 is set as the training background color.

[0080] Figure 8 illustrates the results of training an MLP using the method of this embodiment, where black, a color with a large color difference from the color of the training image, is set as the training background color. In this embodiment, by setting the training background color to a color different from the color of the object in the captured image, it is possible to suppress the occurrence of loss in the object region. In Figure 8, the person is the object.

[0081] (a1) in Figure 8(a) is the same figure as (a1) in Figure 6(a), and shows the imaging space in which the white object 811 exists.

[0082] In the explanation of Figure 8, as shown in Figure 8(a), the background region 814, which is the area in the captured image where object 811 is not present, is filled with the same color as the training background color to create the training image 812. The color of the object region 813 included in the training image 812 is white, the same as the color of the object in the captured image.

[0083] (b1) in Figure 8(b) shows a model in which, as a result of training the MLP, the three-dimensional shape 821 of the white object is obtained without loss in the three-dimensional space representing the training space. The output image 822 shown in Figure 8(b) is obtained by inputting the viewpoint information at the time of capturing the training image 812 into the model shown in (b1), and determining the color according to equation (4). As shown in this output image 822, the object region 823 is drawn in the same way as the actual object.

[0084] Figure 8(c) (c1), similar to Figure 6(c) (c1), shows a model in which the three-dimensional shape 831 of the object is obtained in three-dimensional space as a result of training the MLP, and includes holes that were not present in the actual object.

[0085] The output image 832 in Figure 8(c) is obtained by inputting the viewpoint information at the time of capturing the training image 812 into the model shown in (c1) in Figure 8(c), and determining the color according to equation (4). In the model (c1), the region 833 in the output image 832 corresponding to the region where a hole is formed on the object will be drawn with black, which is the background color during training.

[0086] In this way, by setting the background color during training to a color with a large color difference from the color of object 811, the region 833 in the output image 832 corresponding to the missing region of the three-dimensional shape 831 of the object on the model is drawn in black. Therefore, the loss between the output image 832 and the training image 812 shown in Figure 8(c) is greater than the loss between the output image 822 and the training image 812 shown in Figure 8(b). For this reason, in this embodiment, it is possible to suppress the convergence of the MLP parameters to a model in which missing parts of the object, such as shown in (c1), occur during training.

[0087] Note that in Figure 8, the captured image was explained as including a background region for illustrative purposes. However, in this embodiment, the captured image is assumed to contain an object throughout the entire image, so the captured image does not have a background region. When the captured image does not have a background region, as in this embodiment, it is not necessary to fill the background region of the captured image with the training background color to use it as a training image. In other words, in this embodiment, the captured image may be used as a training image.

[0088] As explained above, in this embodiment, a training background color is set for each pixel such that the difference in the color values ​​of objects in the training image is large. By setting the training background color in this way, it is possible to suppress the output of models in which objects are missing or absent. As a result, it is possible to generate a virtual viewpoint image in which no objects are missing or absent.

[0089] Furthermore, the complementary color to the object's color in the training image may be used as the background color for each pixel during training. One method for calculating the complementary color is to calculate the sum of the maximum and minimum signal values ​​for each RGB channel and subtract the signal value of each channel from the sum. For example, if the signal values ​​for each RGB channel representing the object's color are (70, 220, 60), the sum of the maximum and minimum values ​​is 60 + 220 = 280, so the complementary color is (210, 60, 220).

[0090] Alternatively, the complementary or opposite color on the color wheel to the object's color in the training image may be used as the training background color for each pixel. When complementary or opposite colors are used, black is set as the training background color for the achromatic color white, and black is set as the training background color for white.

[0091] Alternatively, the color value of each pixel in the captured image may be converted to various color spaces other than the RGB color system, such as the CIE color system, XYZ color system, or L*a*b color system, and the color with the largest color difference from the object's color in the converted color space may be set as the background color for learning each pixel.

[0092] Furthermore, any method that allows you to set a color with a large color difference relative to the object's color, other than the methods described above, may be used.

[0093] In this embodiment, the learning of the radiance field was described as the spatial information, but the method for setting the background color during learning described in this embodiment is also applicable to other methods of rendering virtual viewpoint images by volume rendering. For example, instead of using MLP, Plenoxels or TensoRF, which employ explicit field representation of color and volume density, may be used. Alternatively, NeuS, which improves the accuracy of shape estimation by representing the field with SDF, may be used, or 3D Gaussian Splatting, which represents the field as a set of points with extent, may be used. Furthermore, the spatial information to be learned may hold information on color and opacity for a specified position and direction in the form of a multilayer neural network, a three-dimensional grid, a three-dimensional grid with an octree structure, a three-dimensional point cloud, or a set of three-dimensional Gaussian functions.

[0094] Furthermore, a different background color may be set for the background area outside the object area, compared to the background color used during training for the object area. A specific example of this process will be explained in the embodiment described later.

[0095] <Embodiment 2> Embodiment 1 described a method in which the entire scene is treated as an object, a training background color is set for all pixels, and training is performed using the set training background color. This embodiment describes a method in which different training background colors are set for each region. As an example, it is conceivable to set different training background colors for the region of a person and the other regions. This embodiment will be described mainly in terms of the differences from Embodiment 1. Unless otherwise specified, the configuration and processing are the same as in Embodiment 1.

[0096] Figure 9 is a flowchart illustrating the processing flow of this embodiment performed by the image processing device 110. The series of processes shown in the flowchart of Figure 9 are performed by the CPU 201 of the image processing device 110 loading the program code stored in the storage unit 203 into the main memory 202 and executing it.

[0097] S901 is a similar process to S401, in which the CPU201 acquires multiple captured images obtained by the synchronized imaging of multiple imaging devices 101, as well as the camera parameters of each of the multiple imaging devices 101a to h. Furthermore, in S901, the CPU201 acquires the captured images (called background images) taken by each of the multiple imaging devices 101a to h, assuming that there are no dynamic objects 106 in the imaging space 105.

[0098] Figure 10 is a diagram illustrating the images used in this embodiment. Figure 10(a) is an image captured by the imaging device 101, which includes a person, a dynamic object, and a tree, a stationary object. Figure 10(b) is an image captured by the imaging device 101, which captured the image in Figure 10(a), when there are no dynamic objects in the imaging space, and this image is acquired as the background image of the imaging device 101. As shown in Figure 10(b), the difference between the background image and the captured image is the presence or absence of a person, a dynamic object.

[0099] In S902, CPU201 compares the captured image acquired by S901 with the background image and extracts the region of dynamic objects in the captured image (called the foreground region). One method for extracting the foreground region from the captured image is background subtraction. Then, CPU201 generates a foreground image using the position of the foreground region.

[0100] Figure 10(c) shows an example of a foreground image. The foreground image is an image in which the α channel value is added to the color value of each pixel in the captured image. The CPU 201 sets the α channel value to "1" for pixels included in the foreground region and sets the α channel value to "0" for pixels included in regions other than the foreground region. As shown in Figure 10(c), in the foreground image, the color value of pixels in regions other than the foreground region, i.e., pixels with an α channel value of "0", may be uniformly set to one color (for example, white). The CPU 201 generates this foreground image for each captured image acquired by S901.

[0101] S903 acquires all pixel values ​​in the foreground image obtained by S902.

[0102] S904 performs the background color setting process during learning in this embodiment.

[0103] Figure 11 is a flowchart illustrating the details of the background color setting process during learning in this embodiment.

[0104] Sections S1101 to S1102 are the same as S701 to S702, so their explanation will be omitted.

[0105] S1103 to S1111 is a loop process. In S1103, CPU201 selects the pixels to be processed from all the pixels of the multiple foreground images acquired in S903. Then CPU201 executes the processes in S1104 to S1110 on the pixels to be processed.

[0106] In S1111, if CPU201 determines that there are pixels in the foreground image that have not been selected for processing (S1111 is NO), it returns to S1103. Then, it selects new pixels to be processed from among the unselected pixels in the foreground image and repeats the processing in S1104 to S1110. In S1111, if CPU201 determines that all pixels have been selected for processing (S1111 is YES), it terminates the background color setting process during learning and proceeds to S905.

[0107] In S1104, CPU201 determines whether the pixel to be processed is a pixel included in the foreground region, which is the area of ​​a dynamic object. CPU201 determines that if the value of the alpha channel of the pixel value of the pixel to be processed is "1", it is a pixel in the foreground region, and if the value of the alpha channel is "0", it is a pixel in an area other than the foreground region.

[0108] If CPU201 determines that the pixel to be processed is outside the foreground region (S1104 is NO), it proceeds to S1105. In S1105, CPU201 sets a predetermined color as the learning background color corresponding to the pixel to be processed. For example, white is set as the learning background color for pixels outside the foreground region.

[0109] If CPU201 determines that the pixel to be processed is a pixel in the foreground area (S1104 is YES), it executes S1106 to S1110.

[0110] S1106 to S1110 are the same processes as S704 to S708 in Figure 7 described in Embodiment 1, and a color is set as the learning background color corresponding to the pixels to be processed such that the color difference between it and the color value of the pixels in the training image is large. That is, in this embodiment, only the pixels included in the foreground region are set as the learning background color such that the color difference between it and the training image is large.

[0111] Returning to Figure 9, let's continue the explanation of the flowchart. In S905, the CPU201 estimates the radiance field as spatial information of the learning space. S906 is the same as S405 in Embodiment 1, so its explanation is omitted.

[0112] Figure 12 illustrates the MLP training results when the background color during training is set to a color with a large color difference from the object's color (black) using the method of this embodiment.

[0113] In this embodiment, the image obtained by filling the pixels in the captured image 1230 that correspond to pixels in the foreground image where the α channel value is "0" with white, which is the color set in S1105, is used as the training image 1220. Alternatively, as shown in Figure 10(c), the training image 1220 may be a foreground image in which the color values ​​of all areas other than the foreground region are uniformly set to white.

[0114] Image 1201 is an image obtained by volume rendering using equation (3) by inputting the viewpoint information when the training image 1220 was captured into a model in which the three-dimensional shape of the white object is obtained without any loss, as shown in (b1) in Figure 8(b).

[0115] Image 1203 shows each pixel (u k ,v k The color value of the corresponding pixel is the training background color C bg (u k ,v k This is an image obtained by converting to ). In this embodiment, only the foreground region, which is the area of ​​the person, is set as the background color during training to a color with a large color difference from the object. White is set as the background color during training for areas other than the foreground region. Therefore, image 1203 is an image in which the color of the area of ​​the person, which is the foreground region, is a black-like color, and the color of the other areas is white.

[0116] By alpha blending the volume-rendered image 1201 and the training background color image 1203 using equation (4), an output image 1211 is obtained that is compared with the training image 1220. Since the accuracy of the model shown in (b1) in Figure 8(b) is appropriate, the loss between this output image 1211 and the training image 1220 should be small. As shown in Figure 12, the loss between the output image 1211 and the training image 1220 is calculated to be small.

[0117] On the other hand, Image 1202 is an image obtained by inputting the viewpoint information from when the training image 1220 was captured into a model in which there is a defect in the three-dimensional shape of a white object like (c1) in Figure 8(c), and then performing volume rendering using equation (3).

[0118] By alpha blending image 1202 and image 1203, which shows the background color during training, using equation (4), an output image 1212 is obtained that is compared with the training image 1220. The model shown in (c1) in Figure 8(c) is a model with insufficient accuracy. Therefore, the loss between this output image 1212 and the training image 1220 should be calculated to be large, but as shown in Figure 12, the loss between output image 1212 and training image 1220 is calculated to be small.

[0119] As described above, in this embodiment, a color value that has a large color difference from the object's color is set as the background color during training for a specific region of the training image (output image). Therefore, according to this embodiment, even when it is only possible to train a specific region, it is possible to prevent the loss or disappearance of objects in that region.

[0120] In this embodiment, it has been described that an image and a background image are acquired in S901, and a foreground image is generated in S902 using the background subtraction method. Alternatively, an image in which the foreground and background regions are pre-separated may be acquired in S901. If an image in which the foreground and background regions are pre-separated is acquired, S902 may be skipped after S901 and the process may proceed to S903. Furthermore, if an image acquired in S901 is obtained by imaging in an imaging space with a specific background color, such as a green screen, the foreground region may be extracted in S902 by extracting a region of a color different from the background color.

[0121] <Embodiment 3> In Embodiment 1, a training background color was set for each pixel. In this embodiment, a method of training by setting a single training background color for all viewpoints is described. In this embodiment, as in Embodiment 1, the entire scene is described as an object. This embodiment will be described mainly in terms of the differences from Embodiment 1. Unless otherwise specified, the configuration and processing are the same as in Embodiment 1.

[0122] Figure 13 is a flowchart illustrating the details of the learning-time background color setting process performed in S403 of Figure 4 in this embodiment. The series of processes shown in the flowchart of Figure 13 are performed by the CPU 201 of the image processing device 110 loading the program code stored in the storage unit 203 into the main memory 202 and executing it.

[0123] In S1301, CPU201 generates a histogram of the color values ​​of all pixels acquired in S402 (Figure 4) for each RGB channel.

[0124] In S1302, the CPU 201 selects multiple high-frequency signal values ​​from each channel histogram acquired in S1301, in order from the most frequent signal value. In this embodiment, as an example, we will explain the process by selecting only two high-frequency signal values.

[0125] Figure 14 shows an example of a histogram for one of the three RGB channels. In Figure 14, point 1401 represents the most frequent signal value, and point 1402 represents the next most frequent signal value. Therefore, in S1302, the signal value "20" indicated by point 1401 and the signal value "140" indicated by point 1402 are acquired.

[0126] In S1303, the CPU201 selects low-frequency signal values ​​whose occurrence frequency is less than or equal to a predetermined ratio to the number of pixels of the most frequent signal value. In this embodiment, 5% of the number of pixels of the most frequent pixel value is used as the threshold, and multiple low-frequency signal values ​​are calculated for which the number of pixels is less than or equal to the threshold.

[0127] For example, from the histogram shown in Figure 14, the signal value "120" indicated by point 1403 and the signal value "230" indicated by point 1404 are selected as low-frequency signal values.

[0128] In S1304, the CPU 201 determines, for each channel, a low-frequency signal value that is sufficiently distant from each of the high-frequency signal values acquired in S1302, and determines a color value obtained by combining the determined low-frequency signal values as the background color during learning.

[0129] Specifically, the CPU 201 calculates the minimum value D of the distances D between each high-frequency signal value and the low-frequency signal value min using the following formula (6), and determines the low-frequency signal value when the value of the minimum value D min obtained from formula (6) is maximized for each channel. In formula (6), C high_i represents each high-frequency signal value, and C min represents the low-frequency signal value.

[0130] D min = min|C high_i - C min | (6)

[0131] For example, among the high-frequency signal values “140” and “20” acquired in S1302, let C high_1 be “20” and C high_2 be “140”.

[0132] When the low-frequency signal value C min is “120” (point 1403 in FIG. 14), the distance D Cmin=120 is D Cmin=120 = |C high_1 - 120| = |20 - 120| = 100 D Cmin=120 = |C high_2 - 120| = |140 - 120| = 20 Therefore, “20” is obtained as the minimum value D min when the low-frequency signal value C min(Cmin=120) is “120”.

[0133] On the other hand, when the low-frequency signal value C min is “230” (1404 in FIG. 14), the distance D Cmin=230 is D Cmin=230 = |C high_1 - 230| = |20 - 230| = 210 D Cmin=230 =|C high_2 -230|=|140-230|=90 Therefore, the low-frequency signal value C min The minimum value D when is "230" min(Cmin=230) The result is "90".

[0134] D min(Cmin=120) =20 <D min(Cmin=230) Since =90, the minimum value D min The value that is large is the minimum value D min(Cmin=230) It is the one. Minimum value D min(Cmin=230) This is the low-frequency signal value C min Since this is the value when the signal is "230", the signal value at point 1404 in Figure 14, "230", is determined to be a low-frequency signal value that is sufficiently far from each high-frequency signal value.

[0135] In this way, low-frequency signal values ​​that are sufficiently far from high-frequency signal values ​​are determined from each channel, and the color value composed of these determined signal values ​​becomes the color value representing the learning background color in this embodiment. In this embodiment, a uniform learning background color is set for all viewpoints and all pixels. Therefore, C in equation (4) bg The pixel (u) is as shown in the following equation. k ,v k It becomes a constant that does not depend on ).

[0136] C(u k ,v k )=C A (r) + (1-α(r))C bg

[0137] As described above, in this embodiment, the background color during training is set using the histogram information of the color values ​​of all captured images, and training is performed using the set background color during training. Therefore, according to this embodiment, it is possible to suppress the generation of a model in which the three-dimensional shape of an object with defects or partial loss is estimated. Furthermore, in this embodiment, instead of setting a background color during training for each pixel, one common background color is set for all pixels. Therefore, according to this embodiment, it is possible to suppress the generation of a model in which the three-dimensional shape of an object with defects or partial loss is estimated while suppressing the amount of memory used during training.

[0138] In this embodiment, it has been explained that in S1302, multiple high-frequency signal values ​​are selected from the histogram of each channel. However, in S1302, only the most frequent signal value may be selected as the high-frequency signal value. If only the most frequent pixel value is to be obtained as the high-frequency signal value, S1301 may be skipped if the most frequent pixel value can be obtained without using a histogram.

[0139] Furthermore, although this embodiment describes selecting multiple low-frequency signal values ​​for each channel whose occurrence frequency is below a threshold in S1303, S1303 may also select only the signal value with the lowest occurrence frequency in the histogram.

[0140] Furthermore, in this embodiment as well, as described in Embodiment 2, a foreground image may be generated and separated into a foreground region such as the human region and a background region other than the foreground region. In this case, an image in which the background region of the captured image is filled with the background color used for training may be used as the training image.

[0141] Furthermore, in this embodiment, the background color during training was set using the histogram of the captured images, but the average pixel value of all captured images may be calculated and the background color during training may be set based on the average pixel value. Figure 15 is a flowchart illustrating the details of the background color setting process performed in S403 when the background color during training is set based on the average pixel value.

[0142] In S1501, CPU201 obtains the average pixel value of all captured images.

[0143] S1502 is the same process as S701, and CPU201 obtains the maximum and minimum values ​​that can be represented by the color depth of the captured image.

[0144] S1503 is a similar process to S702, where CPU201 calculates the average of the maximum and minimum values ​​obtained in S1502 and determines that average as the threshold. For example, if the maximum value is "255" and the minimum value is "0", the average value is calculated to be "127.5".

[0145] In S1504, CPU201 compares the signal value of each channel in the average pixel value with the threshold and determines whether the signal value is greater than the threshold. If the signal value of a channel is greater than the threshold, the process proceeds to S1505. If the signal value of each channel is less than or equal to the threshold, the process proceeds to S1506.

[0146] In S1505, CPU201 sets the signal values ​​of channels in the background color channel during training that have a signal value greater than the threshold at the average pixel value to the minimum value that can be represented by color depth.

[0147] In S1506, CPU201 sets the signal values ​​of channels in the background color channel during training that have a signal value at the average pixel value below a threshold to the maximum value that can be represented by color depth.

[0148] The color value, composed of the signal values ​​set in S1505 and S1506, is set as the common learning background color for all viewpoints and all pixels.

[0149] Furthermore, in this embodiment, a single learning background color common to all viewpoints of the multiple imaging devices 101a to h has been described. Alternatively, a single learning background color common to each of the multiple imaging devices 101a to h may be set. That is, a single learning background color may be set for each captured image.

[0150] For example, when setting a single learning background color for each captured image, in S1301, a histogram is generated for each of the multiple captured images obtained by each of the multiple imaging devices 101a to h. Then, in S1302 to S1303, the high-frequency signal value and low-frequency signal value are determined for each histogram corresponding to each captured image, and in S1304, the learning background color is set for each captured image. When setting a single learning background color for each viewpoint or for each captured image, the average pixel value or the mode pixel value may also be used.

[0151] <Other Embodiments> This disclosure can also be implemented by supplying a program that implements one or more of the functions of the above-described embodiments to a system or device via a network or storage medium, and by having one or more processors in the computer of that system or device read and execute the program. It can also be implemented by a circuit (e.g., an ASIC) that implements one or more functions.

[0152] The above-described embodiments include the following configurations.

[0153] (Composition 1) An acquisition means for acquiring multiple captured images obtained by capturing an object present in the learning space from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints, A setting means for adjusting the color obtained as a result of inputting viewpoint information of one of the multiple viewpoints into the spatial information relating to the learning space, to set a color that is different from the color of the object, A learning means for learning spatial information based on the plurality of captured images, camera parameters corresponding to each of the plurality of viewpoints, and the set color information for adjustment, An image processing apparatus characterized by having

[0154] (Configuration 2) The learning means is The color obtained by inputting the viewpoint information into the spatial information is The spatial information is trained so that the difference between the output image, which is drawn using the color obtained by alpha blending the aforementioned adjustment color, and the training image, which is obtained from the target image that corresponds to the output image among the multiple captured images, becomes smaller. The image processing apparatus according to configuration 1, characterized in that...

[0155] (Composition 3) The setting means sets a color that has a larger color difference than the color of the object in a predetermined color space as the color for adjustment. An image processing apparatus according to configuration 1 or 2, characterized by the above.

[0156] (Composition 4) The setting means sets the opposite color on the color wheel of the object's color as the color for adjustment. An image processing apparatus according to configuration 1 or 2, characterized by the above.

[0157] (Composition 5) The setting means sets the color for adjustment corresponding to each of the multiple viewpoints. An image processing apparatus according to any one of configurations 1 to 4, characterized in that

[0158] (Composition 6) The setting means sets the color for adjustment for each pixel of the output image. The image processing apparatus according to configuration 2, characterized in that...

[0159] (Composition 7) The setting means determines the color to be adjusted corresponding to the pixel to be processed in the output image from the color value of the corresponding pixel in the target captured image. The image processing apparatus according to configuration 6, characterized by the features described therein.

[0160] (Composition 8) The setting means sets a color for adjustment corresponding to the pixel to be processed, which has a larger color difference in a predetermined color space than the color indicated by the color value of the corresponding pixel in the target image. The image processing apparatus according to configuration 7, characterized by the features described above.

[0161] (Composition 9) The setting means is, If the pixel in the target image corresponding to the pixel to be processed in the output image is a pixel in the object's region, then a color different from the object's color is set as the color for adjustment corresponding to the pixel to be processed. The image processing apparatus according to configuration 6, characterized by the features described therein.

[0162] (Composition 10) The setting means is, If the pixel in the target image corresponding to the pixel to be processed in the output image is a pixel in a region other than the object region, a predetermined color is set as the color for adjustment corresponding to the pixel to be processed. The image processing apparatus according to configuration 9, characterized by the features described therein.

[0163] (Composition 11) A second acquisition means for acquiring a background image obtained by capturing the learning space without the object from the viewpoint at which the aforementioned target image was captured, The system further includes an extraction means for comparing the target image with the background image to extract the region of the object in the target image. The image processing apparatus according to configuration 9 or 10, characterized by the above.

[0164] (Composition 12) The setting means is, Set one color as the color for adjusting the output image corresponding to each of the multiple viewpoints. The image processing apparatus according to configuration 2, characterized in that...

[0165] (Composition 13) The setting means sets a color whose frequency of appearance in the plurality of captured images is less than or equal to a predetermined value as the color for adjustment. The image processing apparatus according to configuration 12, characterized in that...

[0166] (Composition 14) The setting means is For each channel constituting the color value, a signal value whose frequency of appearance in the plurality of captured images is less than or equal to a predetermined value is determined, and the color represented by the color value composed of the combination of the determined signal values ​​is set as the color for adjustment. The image processing apparatus according to configuration 13, characterized by the above.

[0167] (Composition 15) The aforementioned spatial information is, The information regarding color and opacity for a specified position and direction is stored in one of the following forms: a multilayer neural network, a three-dimensional grid, a three-dimensional grid with an octvine structure, a three-dimensional point cloud, or a set of three-dimensional Gaussian functions. An image processing apparatus according to any one of configurations 1 to 14, characterized by the features described herein.

[0168] (Composition 16) The color obtained by inputting viewpoint information into the aforementioned spatial information is, The color is obtained by volume rendering using the color and opacity information obtained from the spatial information, by inputting the viewpoint information into the spatial information. The image processing apparatus according to configuration 15, characterized by the features described herein.

[0169] (Composition 17) An acquisition step to obtain multiple captured images obtained by capturing an object present in the learning space from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints, A setting step of setting a color for adjusting the color obtained as a result of inputting viewpoint information of one of the multiple viewpoints into the spatial information relating to the learning space to be a different color from the color of the object, A learning step in which spatial information is learned based on the plurality of captured images, camera parameters corresponding to each of the plurality of viewpoints, and the set color information for adjustment, An image processing method characterized by having the following features.

[0170] (Composition 18) A program for causing a computer to execute each of the means of the image processing apparatus described in any one of configurations 1 to 16. [Explanation of Symbols]

[0171] 301 Training Data Acquisition Unit 303 Background color setting section during learning 304 Learning Department

Claims

1. An acquisition means for acquiring multiple captured images obtained by capturing an object present in the learning space from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints, A setting means for adjusting the color obtained as a result of inputting viewpoint information of one of the multiple viewpoints into the spatial information relating to the learning space, to set a color that is different from the color of the object, A learning means for learning spatial information based on the plurality of captured images, camera parameters corresponding to each of the plurality of viewpoints, and the set color information for adjustment, An image processing apparatus characterized by having

2. The learning means is The spatial information is trained so that the difference between the output image, which is drawn using a color obtained by alpha blending the color obtained by inputting the viewpoint information into the spatial information with the color used for adjustment, and the training image obtained from the target image, which is one of the multiple captured images that corresponds to the output image, becomes small. The image processing apparatus according to feature 1.

3. The setting means sets a color that has a larger color difference than the color of the object in a predetermined color space as the color for adjustment. The image processing apparatus according to feature 1.

4. The setting means sets the opposite color on the color wheel of the object's color as the color for adjustment. The image processing apparatus according to feature 1.

5. The setting means sets the color for adjustment corresponding to each of the multiple viewpoints. The image processing apparatus according to feature 1.

6. The setting means sets the color for adjustment for each pixel of the output image. The image processing apparatus according to claim 2.

7. The setting means determines the color to be adjusted corresponding to the pixel to be processed in the output image from the color value of the corresponding pixel in the target captured image. The image processing apparatus according to claim 6.

8. The setting means sets a color for adjustment corresponding to the pixel to be processed, which has a larger color difference in a predetermined color space than the color indicated by the color value of the corresponding pixel in the target image. The image processing apparatus according to feature 7.

9. The setting means is, If the pixel in the target image corresponding to the pixel to be processed in the output image is a pixel in the object's region, then a color different from the object's color is set as the color for adjustment corresponding to the pixel to be processed. The image processing apparatus according to claim 6.

10. The setting means is, If the pixel in the target image corresponding to the pixel to be processed in the output image is a pixel in a region other than the object region, a predetermined color is set as the color for adjustment corresponding to the pixel to be processed. The image processing apparatus according to feature 9.

11. A second acquisition means for acquiring a background image obtained by capturing the learning space without the object from the viewpoint at which the aforementioned target image was captured, The system further includes an extraction means for comparing the target image with the background image to extract the region of the object in the target image. The image processing apparatus according to feature 9.

12. The setting means is, Set one color as the color for adjusting the output image corresponding to each of the multiple viewpoints. The image processing apparatus according to claim 2.

13. The setting means sets a color whose frequency of appearance in the plurality of captured images is less than or equal to a predetermined value as the color for adjustment. The image processing apparatus according to feature 12.

14. The setting means is For each channel constituting the color value, a signal value whose frequency of appearance in the plurality of captured images is less than or equal to a predetermined value is determined, and the color represented by the color value composed of the combination of the determined signal values ​​is set as the color for adjustment. The image processing apparatus according to feature 13.

15. The aforementioned spatial information is, The information regarding color and opacity for a specified position and direction is stored in one of the following forms: a multilayer neural network, a three-dimensional grid, a three-dimensional grid with an octvine structure, a three-dimensional point cloud, or a set of three-dimensional Gaussian functions. The image processing apparatus according to feature 1.

16. The color obtained by inputting viewpoint information into the aforementioned spatial information is, The color is obtained by volume rendering using the color and opacity information obtained from the spatial information, by inputting the viewpoint information into the spatial information. The image processing apparatus according to feature 15.

17. An acquisition step to obtain multiple captured images obtained by capturing an object present in the learning space from multiple viewpoints, and camera parameters corresponding to each of the multiple viewpoints, A setting step of setting a color for adjusting the color obtained as a result of inputting viewpoint information of one of the multiple viewpoints into the spatial information relating to the learning space to be a different color from the color of the object, A learning step in which spatial information is learned based on the plurality of captured images, camera parameters corresponding to each of the plurality of viewpoints, and the set color information for adjustment, An image processing method characterized by having the following features.

18. A program for causing a computer to execute each means of the image processing apparatus described in any one of claims 1 to 16.

Images