Image processing device, image processing method and program

Abstract

To provide an image processing device for shortening a time required for learning NeRF.SOLUTION: An image processing device 102 acquires virtual viewpoint information including at least any of imaging parameters of each of a plurality of imaging devices disposed at positions different from each other, data of captured images obtained by imaging with each of the plurality of imaging devices, information representing a position of a virtual viewpoint and information representing a direction of a line of sight from the virtual viewpoint; determines a learning condition of a learning model for estimating a radiance field corresponding to an object present in an imaging area of the plurality of imaging devices, based on the virtual viewpoint information; and trains the learning model based on the learning condition, the imaging parameters and the captured image data.SELECTED DRAWING: Figure 4

Description

[Technical field]

[0001] The present disclosure relates to a technique for generating a virtual viewpoint image of a target area imaged by an imaging device. [Background technology]

[0002] There is a technique for estimating the radiance fields of an object (hereinafter referred to as a "scene") present in an imaging target area using multiple captured image data (hereinafter referred to as "multi-viewpoint images") obtained by imaging from different viewpoints. There is also a technique for generating an image (hereinafter referred to as a "virtual viewpoint image") corresponding to the appearance of a scene from an arbitrary virtual viewpoint (hereinafter referred to as a "virtual viewpoint") using the estimated radiance field. Non-Patent Document 1 discloses a technique for estimating a radiance field and generating a virtual viewpoint image using NeRF (Neural Radiance Fields) configured by a deep learning network. By inputting information indicating an arbitrary position and viewpoint direction in a three-dimensional space to a trained NeRF obtained as a result of training NeRF using multi-viewpoint image data, the color and volume density corresponding to the scene are estimated. Here, the volume density is an index indicating the opacity of a color.

[0003] When a virtual viewpoint image is generated using the trained NeRF described above, the following process is executed in the trained NeRF. First, based on the input virtual viewpoint position and line of sight direction, an image corresponding to the size of the virtual viewpoint image to be generated is projected from the virtual viewpoint position into a three-dimensional space, and a light ray is emitted from the virtual viewpoint position toward the direction of each pixel in the image. Next, a plurality of points in the three-dimensional space existing on the light ray corresponding to each pixel are sampled, and the color and volume density corresponding to each sampled point are calculated. Next, the calculated color and volume density corresponding to each point are integrated for each light ray from the virtual viewpoint to determine each pixel value in the virtual viewpoint image, and the virtual viewpoint image is generated.

[0004] Moreover, when learning NeRF, a series of processes as described below are repeatedly executed. First, information indicating the position and optical axis direction (hereinafter referred to as "attitude") of the imaging device is input to NeRF. Based on the input information, NeRF executes a process similar to the above-mentioned virtual viewpoint image generation process to generate an image corresponding to the captured image obtained by imaging with the imaging device. Next, using the captured image data as training data, the weight parameters of the deep learning network constituting NeRF are updated so that the difference between each pixel value of the image generated by NeRF and each pixel value of the captured image becomes smaller. [Prior art documents] [Non-patent literature]

[0005] [Non-Patent Document 1] Ben Mildenhall, 5 others, “NeRF: Representing Scenes as Neural Radiance Fields for View Synthesis”, [online], August 3, 2020, arXiv, [searched on August 9, 2020], Internet<https: / / arxiv.org / pdf / 2003.08934.pdf> Summary of the Invention [Problem to be solved by the invention]

[0006] In order to estimate the radiance field with high accuracy using NeRF, it is necessary to repeatedly train NeRF as described above using a large amount of multi-viewpoint image data, which poses the problem that training NeRF takes a long time. [Means for solving the problem]

[0007] The image processing device according to the present disclosure includes an imaging parameter acquisition means for acquiring imaging parameters of each of a plurality of imaging devices arranged at different positions from each other, an image acquisition means for acquiring data of an image obtained by imaging by each of the plurality of imaging devices, a viewpoint acquisition means for acquiring virtual viewpoint information including at least one of information indicating the position of a virtual viewpoint and information indicating the direction of a line of sight from the virtual viewpoint, a determination means for determining learning conditions of a learning model that estimates a radiance field corresponding to an object present in the imaging area of ​​the plurality of imaging devices based on the virtual viewpoint information, and a learning means for learning the learning model based on the learning conditions, the imaging parameters, and the data of the image. Effect of the Invention

[0008] According to the present disclosure, the time required for NeRF training can be reduced. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of a configuration of an image processing system according to a first embodiment. [Diagram 2] 1 is a block diagram showing an example of a hardware configuration of an image processing device according to a first embodiment. [Diagram 3] FIG. 2 is a diagram for explaining captured image data used for learning of NeRF according to the first embodiment. [Figure 4] 1 is a block diagram showing an example of a functional configuration of an image processing device according to a first embodiment. [Diagram 5] 4 is a flowchart showing an example of a series of processing flows of the image processing device according to the first embodiment. [Figure 6] 5 is a flowchart showing an example of the flow of a learning process in a learning unit according to the first embodiment. [Figure 7] 10 is a flowchart showing an example of the flow of a learning process in a learning unit according to the second embodiment. [Figure 8] FIG. 11 is a block diagram showing an example of a functional configuration of an image processing device according to a third embodiment. [Figure 9] 11 is a flowchart showing an example of a series of processing flows of an image processing device according to a third embodiment. [Figure 10] FIG. 11 is a diagram showing an example of the arrangement of an imaging device according to a third embodiment. [Figure 11] FIG. 11 is a diagram showing an example of a captured image according to the third embodiment. [Figure 12] 13 is a flowchart showing an example of a process flow in an importance determining unit according to the third embodiment. [Figure 13] 11 is a flowchart showing an example of a process flow in a condition determining section according to the third embodiment. [Figure 14] 13 is a flowchart showing an example of a process flow in an importance determining section according to the fifth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the present disclosure, and not all of the combinations of features described in the embodiments are necessarily essential to the solution of the present disclosure. Note that the same components will be described with the same reference numerals.

[0011] [Embodiment 1] In this embodiment, a method for reducing the number of captured image data used for learning NeRF configured by a deep learning network based on information indicating the position of a virtual viewpoint and the virtual line of sight direction at the virtual viewpoint (hereinafter simply referred to as "line of sight direction" will be described. Hereinafter, the virtual viewpoint will be referred to as a virtual camera, and the line of sight direction of the virtual viewpoint will be referred to as the attitude of the virtual camera. Moreover, information indicating the position and line of sight direction of the virtual viewpoint, i.e., the position and attitude of the virtual camera, will be referred to as virtual camera information.

[0012] FIG. 1 is a diagram showing an example of the configuration of an image processing system according to the first embodiment. The image processing system includes a plurality of imaging devices 101, an image processing device 102, a UI panel 103, a storage device 104, and a display device 105. Each imaging device 101 is composed of a digital still camera or a digital video camera, and captures an object 106 present in an imaging target area in synchronization with each other according to set imaging conditions. The imaging devices 101 are arranged to surround the periphery of the imaging target area. The image processing device 102 performs NeRF learning using data of a plurality of captured images (multiple viewpoint images) obtained by synchronous imaging by each imaging device 101. In addition, the image processing device 102 estimates a radiance field of the object 106 corresponding to a given virtual viewpoint using the learned NeRF, and generates a virtual viewpoint image corresponding to the appearance from the virtual viewpoint. Hereinafter, a captured image obtained by imaging by the imaging device 101 is referred to as a "captured image of the imaging device 101".

[0013] The UI panel 103 accepts input operations from the user for setting the imaging conditions for each imaging device 101 and the processing conditions for each imaging device 101 and the image processing device 102. The UI panel 103 may also accept input operations for setting the position and line of sight of the virtual viewpoint, i.e., the position and attitude of the virtual camera, when generating a virtual viewpoint image based on the radiance field estimated by NeRF. The above-mentioned input operations do not necessarily have to be performed via the UI panel, and may be performed via another operation input device connected to the image processing device 102, such as a mouse or a keyboard. The storage device 104 stores the multi-view image data acquired by the image processing device 102, and information on the radiance field of the object 106 estimated by the image processing device 102, etc. The display device 105 displays the virtual viewpoint image generated by the image processing device 102. The configuration of the image processing system is not limited to the above, and various components may exist other than the above configuration. However, since configurations other than the above configuration are not the main focus of this disclosure, their description will be omitted.

[0014] 2 is a block diagram showing an example of a hardware configuration of the image processing device 102 according to the first embodiment. The image processing device 102 includes a CPU 201, a main memory 202, a storage unit 203, an input unit 204, a display unit 205, and an external I / F 206, which are communicably connected to each other via a bus 207. The CPU 201 is an arithmetic processing device that comprehensively controls the image processing device 102, and executes various programs stored in the storage unit 203 and the like to perform various processes. The main memory 202 temporarily stores various data, parameters, and the like used when the CPU 201 executes various processes. The main memory 202 also provides a working area for the CPU 201.

[0015] The storage unit 203 is a large-capacity storage device that stores various programs and various data required for displaying a GUI (Graphical User Interface) on the display device 105, and is configured with a non-volatile memory such as a hard disk or a silicon disk. The input unit 204 is an operation input device such as a keyboard, a mouse, an electronic pen, or a touch panel, and receives input operations from a user. The display unit 205 is configured with a liquid crystal panel or the like, and displays the GUI output by the image processing device 102. The CPU 201 also operates as a display control unit that controls the display unit 205 and an input control unit that controls the input unit 204. In the first embodiment, the display unit 205 and the input unit 204 are described as existing inside the image processing device 102, but at least one of the display unit 205 and the input unit 204 may be present as a separate device outside the image processing device 102. The external I / F unit 206 is connected to each imaging device 101 via a LAN 208, and transmits and receives captured image data and control signals.

[0016] Each imaging device 101 is connected to the image processing device 102 via a LAN 208. Each imaging device 101 receives a control signal output from the image processing device 102, and based on the received control signal, starts and stops imaging, changes settings of imaging conditions such as shutter speed and aperture value, and transmits captured image data obtained by imaging.

[0017] Fig. 3 is a diagram for explaining captured image data used for training of the NeRF according to the embodiment 1. Fig. 3(a) is a diagram for explaining an example of captured image data used for training of the conventional NeRF. In training of the conventional NeRF, data of captured images (multiple viewpoint images) of all the imaging devices 101 that can capture the object 106 is used for training of the NeRF, regardless of the position and orientation of the virtual camera 301.

[0018] FIG. 3(b) is a diagram for explaining an example of captured image data used in the learning of the NeRF according to this embodiment. In the learning of the conventional NeRF, as described above, multi-view image data obtained by imaging by all the imaging devices 101, regardless of the position and orientation of the virtual camera 301, is used in the learning of the NeRF. In contrast, in the learning of the NeRF according to this embodiment, as shown in FIG. 3(b), only captured image data of one or more imaging devices 101 close to the position or orientation of the virtual camera 301 is used in the learning of the NeRF. That is, the captured image data of the imaging devices 101 other than the imaging device 101 close to the position or orientation of the virtual camera 301 is not used in the learning of the NeRF, and the number of captured image data used in the learning is reduced. By reducing the captured image data used in the learning, the time required for the learning of the NeRF can be shortened.

[0019] Fig. 3(c) is a diagram showing a conventional general example of reducing the number of captured image data used for training NeRF. As shown as an example in Fig. 3(c), the number of captured image data used for training NeRF can be reduced by uniformly reducing the number of imaging devices 101 surrounding the object 106. However, such a reduction method reduces the accuracy of the radiance field estimated by the trained NeRF. As a result, such a reduction method reduces the quality of the generated virtual viewpoint image.

[0020] On the other hand, the image processing device 102 according to the present embodiment performs NeRF training by intensively using captured image data of the imaging device 101 whose position or orientation is close to that of the virtual camera 301, using a reduction method as shown in Fig. 3(b). The trained NeRF obtained as a result of such training can suppress deterioration in the quality of the radiance field estimated and the virtual viewpoint image generated at least in the position and orientation of the virtual camera that is the same or approximately the same as that of the virtual camera 301. That is, by performing NeRF training by intensively using captured image data of the imaging device 101 whose position or orientation is close to that of the virtual camera 301, it is possible to maintain high accuracy of the estimation of the radiance field while shortening the time required for training the NeRF.

[0021] FIG. 4 is a block diagram showing an example of a functional configuration of the image processing device 102 according to the first embodiment. The image processing device 102 includes a virtual camera information acquisition unit 400, an imaging parameter acquisition unit 401, an image acquisition unit 402, a specification unit 403, a learning unit 404, a generation unit 405, and an output unit 406. The virtual camera information acquisition unit 400 acquires virtual camera information. The virtual camera information acquisition unit 400 acquires virtual camera information set by a user via, for example, a mouse or a keyboard, which is the input unit 204 of the image processing device 102, or a UI panel 103. The virtual camera information acquired by the virtual camera information acquisition unit 400 is transmitted to the specification unit 403 and the generation unit 405. The image acquisition unit 402 acquires captured image data of each imaging device 101, that is, multi-view image data. Specifically, for example, the image acquisition unit 402 acquires multi-view image data obtained by synchronous imaging of all the imaging devices 101. The multi-view image data acquired by the image acquisition unit 402 is transmitted to the learning unit 404. NeRF is trained using a part of the multiple captured image data constituting the multi-viewpoint image data as training image data.

[0022] The imaging parameter acquisition unit 401 acquires imaging parameters including external parameters, internal parameters, and distortion parameters of each imaging device 101. The imaging parameters are generated based on the results of camera calibration performed in advance by an external device different from the image processing device 102, for example, and the imaging parameter acquisition unit 401 acquires the imaging parameters generated by the external device. The method of acquiring the imaging parameters is not limited to the above-mentioned method, and for example, the imaging parameter acquisition unit 401 may generate and acquire the imaging parameters by performing camera calibration using the multi-view image data acquired by the image acquisition unit 402. The imaging parameters acquired by the imaging parameter acquisition unit 401 are transmitted to the identification unit 403 and the learning unit 404.

[0023] The identification unit 403 receives the virtual camera information transmitted from the virtual camera information acquisition unit 400 and the imaging parameters of each imaging device transmitted from the imaging parameter acquisition unit 401. Based on the virtual camera information and the imaging parameters of each imaging device, the identification unit 403 identifies the imaging device 101 that outputs captured image data to be used as learning image data for NeRF learning from among all the imaging devices 101. That is, the identification unit 403 identifies the imaging device 101, thereby determining captured image data to be used as learning image data for NeRF learning from among the multi-view image data acquired by the image acquisition unit 402. Details of the method of identifying the imaging device 101 will be described later. Information indicating the imaging device 101 identified by the identification unit 403 is transmitted to the learning unit 404.

[0024] The learning unit 404 performs learning of the NeRF based on the imaging parameters and captured image data of the imaging device 101 identified by the identification unit 403. Specifically, first, the learning unit 404 inputs the imaging parameters of any one of the one or more imaging devices 101 identified by the identification unit 403 to the NeRF. More specifically, the learning unit 404 inputs at least information indicating the position and orientation of the imaging device 101, which is an external parameter among the imaging parameters, to the NeRF. In the NeRF, the radiance field of the object 106 is estimated based on the input information indicating the position and orientation of the imaging device 101. Furthermore, in the NeRF, an image is generated by a process similar to the generation of a virtual viewpoint image using the estimation result of the radiance field. The learning unit 404 acquires data of the image generated by the NeRF. Next, the learning unit 404 updates the weight parameters of the NeRF so that the difference between the image generated by the NeRF and the captured image becomes smaller, using the captured image data of the imaging device 101 as the correct learning image data (teacher image data). The training unit 404 trains the NeRF by repeatedly executing the above-mentioned series of processes. The trained NeRF obtained as a result of training by the training unit 404 is transmitted to the generation unit 405.

[0025] The generation unit 405 receives the trained NeRF transmitted from the learning unit 404 and the virtual camera information transmitted from the virtual camera information acquisition unit 400, and generates a virtual viewpoint image based on the received trained NeRF and virtual camera information. Specifically, the generation unit 405 inputs virtual camera information, i.e., information indicating the position and orientation of the virtual camera, to the trained NeRF. In the trained NeRF, the radiance field of the object 106 is estimated based on the input information indicating the position and orientation of the virtual camera. Furthermore, in the trained NeRF, a virtual viewpoint image is generated using the estimation result of the radiance field. The generation unit 405 acquires data of the virtual viewpoint image generated by the trained NeRF. The data of the virtual viewpoint image generated by the generation unit 405 is transmitted to the output unit 406. The output unit 406 receives the data of the virtual viewpoint image and outputs it. Specifically, for example, the output unit 406 generates a GUI including a preview image of the virtual viewpoint image, and outputs the generated GUI to the display unit 205 for display.

[0026] The operation of the image processing device 102 will be described with reference to Figs. 5 and 6. Fig. 5 is a flowchart showing an example of a series of processing flows of the image processing device 102 according to the first embodiment. In the following description, "S" at the beginning of a reference symbol indicates a step (process). The processing of each step shown in the flowchart in Fig. 5 is realized by the CPU 201 reading a predetermined program from the storage unit 203, expanding it in the main memory 202, and executing it.

[0027] First, in S501, the imaging parameter acquisition unit 401 acquires imaging parameters of each imaging device 101. Next, in S502, the image acquisition unit 402 acquires multi-view image data acquired by imaging by each imaging device 101. Specifically, for example, the image acquisition unit 402 transmits a signal to each imaging device 101 via the LAN 208 to instruct imaging. Upon receiving the signal, each imaging device 101 performs imaging in synchronization with each other, and transmits the captured image data acquired by imaging to the image processing device 102 via the LAN 208. The image acquisition unit 402 receives the captured image data transmitted from each imaging device 101 via the LAN 208, the external I / F unit 206, and the bus 207. Each captured image data received by the image acquisition unit 402, i.e., the multi-view image data, is stored in the main memory 202.

[0028] Next, in S503, the virtual camera information acquisition unit 400 acquires virtual camera information. Next, in S504, the identification unit 403 identifies, from among all the imaging devices 101, an imaging device 101 that outputs captured image data used as learning image data for NeRF learning, based on the virtual camera information acquired in S503. Specifically, for example, the identification unit 403 identifies the imaging device 101 that outputs captured image data used as learning image data for NeRF learning, using information indicating the attitude of the virtual camera, i.e., the direction of the line of sight at the virtual viewpoint.

[0029] As described above, in the conventional NeRF learning method, all captured image data constituting the multi-viewpoint image data are used as learning image data to learn the NeRF. By such learning, the conventional trained NeRF can estimate the color and volume density corresponding to a virtual viewpoint specified at an arbitrary position in a three-dimensional space and an arbitrary line of sight direction specified at the virtual viewpoint. On the other hand, in the NeRF learning method according to the present embodiment, only captured image data of the imaging device 101 with a posture close to that of the virtual camera is used as learning image data to learn the NeRF. According to the trained NeRF obtained as a result of such learning, it is possible to estimate the color and volume density when the object 106 is viewed from the posture of the virtual camera, that is, from a direction close to the line of sight direction at the virtual viewpoint. At this time, since the captured image data obtained by imaging with the imaging device 101 with a posture close to that of the virtual camera is used as the learning image data, the generated virtual viewpoint image is an image with very little degradation in quality. On the other hand, when the posture of the virtual camera is changed significantly, the accuracy of the generated virtual viewpoint image is significantly reduced.

[0030] For example, the specifying unit 403 first determines a threshold value θ of an angle between a direction vector indicating the attitude of the virtual camera and a direction vector indicating the attitude of each image capture device 101. th Next, the identification unit 403 identifies, from among all the imaging devices 101, imaging devices 101 whose directional vectors satisfy, for example, the following formula (1), as imaging devices 101 that output captured image data to be used as training image data for training of NeRF. |arccos((v m ·v r ) / (|v m ||v r |))|<θ th ··· Equation (1)

[0031] where v m represents the direction vector indicating the orientation of the virtual camera, and v r represents a direction vector indicating the attitude of the image capture device 101, and v m ·v r vm and v r It is assumed that arccos(x) takes a value of -180 to 180 [deg]. The identification unit 403 identifies each of one or more image capture devices 101 that satisfies formula (1) as an image capture device 101 that outputs captured image data to be used as training image data for NeRF training.

[0032] In this embodiment, a single predetermined threshold value θ th However, the threshold value used by the identification unit 403 is not limited to one, and the identification unit 403 may identify the imaging device 101 using a plurality of threshold values. Furthermore, the identification unit 403 may identify the imaging device 101 based on the position of the virtual camera and the position of the imaging device 101, instead of the orientation of the virtual camera and the orientation of the imaging device 101. For example, in this case, the identification unit 403 first identifies the imaging device 101 based on a threshold value d th Next, the identification unit 403 identifies an imaging device 101 whose distance between the position of the virtual camera and the position of the imaging device 101 satisfies, for example, the following formula (2), as the imaging device 101 that outputs captured image data to be used as training image data for training of NeRF. ||p m -p r ||2 <d th ··· Equation (2)

[0033] Here, p m is a position vector that represents the position of the virtual camera in 3D space, and p r is a position vector representing a position in three-dimensional space corresponding to the position of the image capture device 101, and ||x||2 represents the Euclidean norm of vector x. In addition, for example, the identification unit 403 may use formula (1) and formula (2) to identify an image capture device 101 that satisfies both formula (1) and formula (2) as an image capture device 101 that outputs captured image data to be used as training image data for training of NeRF.

[0034] After S504, in S505, the learning unit 404 uses only the captured image data of the imaging device 101 specified in S504 as the learning image data among the multi-view image data acquired in S502, and performs learning of the NeRF. Details of the processing of the learning unit 404 in S505 will be described later with reference to FIG. 6. Next, in S506, the generation unit 405 generates a virtual viewpoint image based on the trained NeRF obtained by the result of the learning in S505 and the virtual camera information acquired in S503. Specifically, the generation unit 405 inputs the virtual camera information to the trained NeRF and acquires data of the virtual viewpoint image generated and output by the trained NeRF. A volume rendering method described later can be used as a method of generating the virtual viewpoint image in the trained NeRF. After S506, in S507, the output unit 406 outputs the virtual viewpoint image generated in S506. After S507, the image processing device 102 ends the processing of the flowchart shown in FIG. 5.

[0035] FIG. 6 is a flowchart showing an example of the flow of the learning process in the learning unit 404 according to the first embodiment, and is a flowchart showing an example of a detailed flow of the process of S505 shown in FIG. 5. The process of this flowchart is executed after the process of S504 shown in FIG. 5. First, in S601, the learning unit 404 acquires the number of epochs of learning when performing NeRF learning. The number of epochs is a numerical value indicating the number of times of learning in deep learning. In this embodiment, the number of epochs is set to the number of times that the learning image data is n L It is assumed that this is an index indicating how many times learning using each piece of captured image data is performed for each piece of captured image data when there is data for each piece of captured image data.

[0036] Next, in S602, the learning unit 404 selects any one of the one or more image capturing devices 101 identified in S504, and inputs the imaging parameters of the selected image capturing device 101 to NeRF. The imaging parameters input to NeRF include at least information indicating the position and orientation of the image capturing device 101.

[0037] The NeRF disclosed in Non-Patent Document 1 is configured with a deep learning network of multilayer perceptrons (hereinafter referred to as "MLP"). The NeRF according to this embodiment may be configured with a deep learning network of MLP, or may be configured with a deep learning network other than MLP. When the NeRF according to this embodiment is configured with a deep learning network of MLP, in the learning phase of the NeRF, the following process is executed in the NeRF, simply as in the method disclosed in Non-Patent Document 1.

[0038] First, information indicating the position and orientation of the image capture device 101 is input to the NeRF by the learning unit 404. At this time, it is preferable that the learning unit 404 performs a conversion process to multi-dimensionalize the input information so that the change in the high-frequency component of the radiance field can be captured, and the information after the conversion process is input to the NeRF. Next, the NeRF projects an image of a size to be generated from a position in the three-dimensional space corresponding to the input position of the image capture device 101 into the three-dimensional space, and emits a virtual light ray from the position toward each pixel in the projected image. Next, the NeRF samples a plurality of points existing on the emitted light ray in the three-dimensional space, and calculates the color and volume density of each sampled point. Next, the NeRF generates an image by integrating the calculated colors and volume densities of the plurality of points from the position in the three-dimensional space corresponding to the position of the image capture device 101 for each emitted light ray. Such an image generation method is generally called a volume rendering method. Finally, the NeRF outputs data of the generated image.

[0039] After S602, in S603, the learning unit 404 updates the weight parameters of NeRF by using the captured image data of the imaging device 101 corresponding to the imaging parameters input to NeRF in S602 as correct learning image data (teacher image data). Specifically, the learning unit 404 updates the weight parameters of NeRF so that the difference between the image data generated and output by NeRF and the learning image data becomes small. Next, in S604, the learning unit 404 determines whether the imaging parameters of all the imaging devices 101 identified in S504 have been input in S602. If it is determined in S604 that the imaging parameters of at least some of the imaging devices 101 have not been input, the learning unit 404 returns to the process of S602. At this time, in S602, the learning unit 404 inputs the imaging parameters of the imaging devices 101 whose imaging parameters have not yet been input, among all the imaging devices 101 identified in S504, to NeRF. Thereafter, the learning unit 404 repeatedly executes the processes from S602 to S604 until it is determined in S604 that the imaging parameters of all the imaging devices 101 have been input.

[0040] When it is determined in S604 that the imaging parameters of all the imaging devices 101 have been input, in S605, the learning unit 404 determines whether or not the number of learning times of NeRF has reached the number of epochs acquired in S601. Specifically, the learning unit 404 determines whether or not the number of learning times of NeRF has reached the number of epochs for each captured image data of each imaging device 101 identified in S504. When it is determined in S605 that the number of epochs has not been reached, the learning unit 404 returns to the process of S602 and repeatedly executes the processes from S602 to S605 until it is determined in S605 that the number of epochs has been reached. Note that in this case, the learning unit 404 repeatedly executes the processes from S602 to S604 for each learning round until it is determined in S604 that the imaging parameters of all the imaging devices 101 have been input. If it is determined in S605 that the number of epochs has been reached, the learning unit 404 ends the process of the flowchart shown in FIG. 6, that is, the process of S505.

[0041] In this embodiment, the condition for ending the NeRF learning process in the learning unit 404 is that the number of learning times reaches the number of epochs, but the condition for ending the learning process is not limited to this. For example, the learning unit 404 may end the learning when the update amount of the weight parameter of the NeRF falls within a predetermined range or the change amount of the image generated by the NeRF becomes smaller than a predetermined standard.

[0042] As described above, in this embodiment, the image processing device 102 is configured to identify the image capture device 101 that is close to the attitude or position of the virtual camera when learning NeRF, and to use only the captured image data of the identified image capture device 101 as learning image data. According to the image processing device 102 configured in this manner, it is possible to estimate the radiance field of the object 106 with high accuracy while shortening the learning time of NeRF by reducing the number of learning images used in learning NeRF, and to generate a virtual viewpoint image with little deterioration in quality.

[0043] [Embodiment 2] The image processing device 102 according to the first embodiment performs learning of NeRF by using only the captured image data of the imaging device 101 close to the attitude or position of the virtual camera as learning image data of NeRF. In the learning method of NeRF described in the first embodiment, when the number of the imaging devices 101 close to the attitude or position of the virtual camera is sufficiently large, it is possible to estimate the radiance field with high accuracy. However, when the number of the imaging devices 101 close to the attitude or position of the virtual camera is small, it may not be possible to estimate the radiance field with high accuracy. Therefore, in the second embodiment, a form in which the learning time of NeRF is shortened while using the captured image data of all the imaging devices 101 as learning image data, rather than using only the captured image data of the imaging device 101 identified by the identification unit 403 as learning image data, will be described. Note that the functional configuration of the image processing device 102 according to the second embodiment (hereinafter simply referred to as "image processing device 102") is the same as that of the image processing device 102 according to the first embodiment shown as an example in FIG. 2 and FIG. 4. In the second embodiment, the difference from the first embodiment will be mainly described.

[0044] The learning unit 404 according to the second embodiment (hereinafter, simply referred to as the "learning unit 404") performs learning of the NeRF a predetermined number of times using captured image data of all the imaging devices 101 as learning image data before performing the learning process described in the first embodiment. By using the captured image data of all the imaging devices 101 as learning image data for learning, it is possible to estimate a radiance field with higher accuracy compared to a case where only captured image data of the imaging device 101 close to the attitude or position of the virtual camera is used as learning image data. Specifically, the learning unit 404 performs a learning process using the captured image data of all the imaging devices 101 as learning image data and a learning process using only captured image data of the imaging device 101 close to the attitude or position of the virtual camera as learning image data. That is, the learning unit 404 further performs learning using only captured image data of the imaging device 101 close to the attitude or position of the virtual camera as learning image data for the NeRF after learning using the captured image data of all the imaging devices 101 as learning image data.

[0045] More specifically, for example, in the conventional NeRF learning, learning is performed n times using the captured image data of all the image capturing devices 101 as learning image data. In contrast, in the NeRF learning in this embodiment, first, the captured image data of all the image capturing devices 101 is used as learning image data to perform n times of learning. D After that, only the captured image data of the image capturing device 101 close to the posture or position of the virtual camera is used as the learning image data, and n T times(nn D According to this learning method, only the captured image data of the image capturing device 101 having a posture or position close to the virtual camera is used as the learning image data. T In the n learning times, the learning image data can be reduced by the difference with respect to the captured image data of all the imaging devices 101. As a result, according to the image processing device 102 according to the present embodiment, it is possible to shorten the learning time of NeRF compared to the conventional case where the captured image data of all the imaging devices 101 are used as learning image data and learning is performed n times.

[0046] The operation of the image processing device 102 will be described with reference to FIG. 7. The image processing device 102 sequentially executes the processes from S501 to S507 shown in the flowchart of FIG. 5, like the image processing device 102 according to the first embodiment, but the process of the learning unit 404 in S505 is different from that of the learning unit 404 according to the first embodiment. The difference between the process of the learning unit 404 and the learning unit 404 according to the first embodiment will be described with reference to FIG. 7. FIG. 7 is a flowchart showing an example of the flow of the learning process in the learning unit 404 according to the second embodiment, and is a flowchart showing an example of the detailed flow of the process of S505 shown in FIG. 5. The process of the flowchart is executed after the process of S504 shown in FIG. 5. In the description of the flowchart shown in FIG. 7, the steps performing the same process as the steps shown in the flowchart of FIG. 6 are denoted by the same reference numerals, and the description thereof will be omitted.

[0047] First, in S701, the learning unit 404 acquires the number of learning epochs when learning NeRF. As described in the first embodiment, the number of epochs is a numerical value representing the number of learning times in deep learning. The learning of NeRF according to this embodiment is divided into a learning step in which captured image data of all the imaging devices 101 are used as learning image data, and a learning step in which only captured image data of the imaging device 101 close to the attitude or position of the virtual camera are used as learning image data. Therefore, the learning unit 404 acquires a first number of epochs of learning in which captured image data of all the imaging devices 101 are used as learning image data, and a second number of epochs of learning in which only captured image data of the imaging device 101 close to the attitude or position of the virtual camera are used as learning image data.

[0048] Next, in S702, the learning unit 404 selects an arbitrary imaging device 101 from among all the imaging devices 101, and inputs the imaging parameters of the selected imaging device 101 to NeRF. Next, in S703, the learning unit 404 updates the weight parameters of NeRF by using the captured image data of the imaging device 101 corresponding to the imaging parameters input to NeRF in S702 as correct learning image data (teacher image data). Specifically, the learning unit 404 updates the weight parameters of NeRF so that the difference between the image data generated and output by NeRF and the learning image data becomes small. Next, in S704, the learning unit 404 determines whether the imaging parameters of all the imaging devices 101 have been input in S702.

[0049] If it is determined in S704 that the imaging parameters of at least some of the imaging devices 101 have not been input, the learning unit 404 returns to the process of S702. At this time, in S702, the learning unit 404 inputs the imaging parameters of the imaging devices 101 whose imaging parameters have not yet been input, among all the imaging devices 101, to the NeRF. Thereafter, the learning unit 404 repeatedly executes the processes from S702 to S704 until it is determined in S704 that the imaging parameters of all the imaging devices 101 have been input. If it is determined in S704 that the imaging parameters of all the imaging devices 101 have been input, in S705, the learning unit 404 determines whether the number of learning times of the NeRF has reached the first epoch number acquired in S701. Specifically, the learning unit 404 determines whether the number of learning times of the NeRF has reached the first epoch number for each captured image data of all the imaging devices 101.

[0050] If it is determined in S705 that the first epoch number has not been reached, the learning unit 404 returns to the process of S702 and repeatedly executes the processes from S702 to S705 until it is determined in S705 that the first epoch number has been reached. Note that in this case, the learning unit 404 repeatedly executes the processes from S702 to S704 for each learning round until it is determined in S704 that the imaging parameters of all the imaging devices 101 have been input.

[0051] If it is determined in S705 that the first epoch number has been reached, the learning unit 404 executes the processes from S602 to S604. Specifically, the learning unit 404 repeatedly executes the processes from S602 to S604 until it is determined in S604 that the imaging parameters of all the imaging devices 101 have been input. If it is determined in S604 that the imaging parameters of all the imaging devices 101 have been input, the learning unit 404 determines in S706 whether the number of times NeRF has been learned has reached the second epoch number acquired in S701. Specifically, the learning unit 404 determines whether the number of times NeRF has been learned has reached the second epoch number for each captured image data of each imaging device 101 identified in S504.

[0052] If it is determined in S706 that the second number of epochs has not been reached, the learning unit 404 returns to the process of S602 and repeatedly executes the processes from S602 to S706 until it is determined in S706 that the number of epochs has been reached. In this case, the learning unit 404 repeatedly executes the processes from S602 to S604 for each learning round until it is determined in S604 that the imaging parameters of all the imaging devices 101 have been input. If it is determined in S706 that the second number of epochs has been reached, the learning unit 404 ends the process of the flowchart shown in FIG. 7, i.e., the process of S505.

[0053] In the present embodiment, the image processing device 102 may perform learning using only captured image data of the imaging device 101 close to the attitude or position of the virtual camera as learning image data after performing learning using captured image data of all the imaging devices 101 as learning image data. However, the above-mentioned learning order is merely an example, and the learning order of NeRF is not limited to the above-mentioned order. For example, the image processing device 102 may perform learning using only captured image data of the imaging device 101 close to the attitude or position of the virtual camera as learning image data, and then perform learning using captured image data of all the imaging devices 101 as learning image data.

[0054] Furthermore, for example, the image processing device 102 may perform learning of NeRF as follows. First, the image processing device 102 performs learning using captured image data of all the imaging devices 101 as learning image data, and then performs learning using only captured image data of the imaging devices 101 close to the attitude or position of the virtual camera as learning image data. Next, the image processing device 102 performs learning again using captured image data of all the imaging devices 101 as learning image data. That is, the image processing device 102 may perform learning using captured image data of all the imaging devices 101 as learning image data and learning using only captured image data of the imaging devices 101 close to the attitude or position of the virtual camera as learning image data in multiple separate times.

[0055] In the present embodiment, the learning using only the captured image data of the imaging device 101 having an attitude or position close to that of the virtual camera as learning image data has been described as being executed as one learning process. However, the image processing device 102 may execute the learning using only the captured image data of the imaging device 101 having an attitude or position close to that of the virtual camera as learning image data by dividing the learning into a plurality of learning processes.

[0056] For example, the image processing device 102 performs learning using only the captured image data of the image capturing device 101 close to the attitude or position of the virtual camera as learning image data, by dividing the learning process into a plurality of learning processes as follows. In the formula (1), the threshold value θ of the angle between the direction vector indicating the attitude of the virtual camera and the direction vector indicating the attitude of each image capturing device 101 is th Specifically, for example, a threshold value θ th As the first threshold θ th1 and the second threshold θ th2 First, the determination unit 403 of the image processing device 102 determines a first threshold value θ th1 Next, the learning unit 404 of the image processing device 102 identifies one or more image capture devices 101 using the first threshold θ th1 Then, the determination unit 403 of the image processing device 102 performs learning using the captured image data of each image capturing device 101 identified using the second threshold value θth2 The identification unit 403 identifies one or more image capture devices 101 using a second threshold value θ th2 Then, the learning unit 404 of the image processing device 102 performs learning using the captured image data of all the image capturing devices 101 as learning image data.

[0057] In addition, the image processing device 102 may set the learning weight for each piece of learning image data to be high, for example, by treating the image data captured by the imaging device 101 that is close to the attitude or position of the virtual camera as important learning image data and using the learning image data.

[0058] In this embodiment, the condition for ending the NeRF learning process in the learning unit 404 has been described as the number of learning attempts reaching the number of epochs, but the condition for ending the learning process is not limited to this. For example, as described above in the first embodiment, the learning unit 404 may end the NeRF learning when the update amount of the NeRF weight parameters falls within a predetermined range or the change amount of the image generated by the NeRF becomes smaller than a predetermined criterion.

[0059] As described above, in this embodiment, the image processing device 102 is configured so that, when learning NeRF, not only captured image data of the image capturing device 101 close to the attitude or position of the virtual camera, but also captured image data of all the image capturing devices 101 are used as learning image data. According to the image processing device 102 configured in this way, even if there are few image capturing devices 101 close to the attitude or position of the virtual camera, it is possible to estimate the radiance field of the object 106 with high accuracy while shortening the learning time of NeRF, and to generate a virtual viewpoint image with little deterioration in quality.

[0060] [Embodiment 3] The image processing device 102 according to the first and second embodiments specifies the imaging device 101 that outputs captured image data used as learning image data based on virtual camera information, i.e., the position or posture of the virtual camera. According to the image processing device 102 according to the first and second embodiments, the learning time of NeRF can be shortened by intensively learning the captured image data of the imaging device 101 thus specified as learning image data. In the third embodiment, a form will be described in which the learning conditions of NeRF are determined based on the importance of the object 106, and NeRF is learned based on the determined learning conditions.

[0061] Fig. 8 is a block diagram showing an example of a functional configuration of an image processing device 102 according to the third embodiment (hereinafter, simply referred to as "image processing device 102"). The image processing device 102 has a virtual camera information acquisition unit 400, an imaging parameter acquisition unit 401, an image acquisition unit 402, an identification unit 830, a learning unit 804, a generation unit 405, and an output unit 406. That is, compared to the image processing device 102 according to the first embodiment, the identification unit 403 and the learning unit 404 of the image processing device 102 according to the first embodiment are changed to the identification unit 830 and the learning unit 804, respectively. Hereinafter, among the units shown in Fig. 8, those that perform the same processes as the units shown in Fig. 4 are assigned the same reference numerals as those in Fig. 4, and description thereof will be omitted.

[0062] The identification unit 830 identifies, from among all the imaging devices 101, an imaging device 101 that outputs captured image data used as learning image data for NeRF learning. The identification unit 830 includes an importance determination unit 831 and a condition determination unit 832. The processing of the importance determination unit 831 and the condition determination unit 832 will be described below. The importance determination unit 831 determines the importance of the object 106 based on the appearance of the object 106 from the virtual viewpoint, that is, based on an image area corresponding to the object 106 in the virtual viewpoint image (hereinafter referred to as the "area of ​​the object 106"). When multiple objects are within the angle of view of the virtual camera, the importance determination unit 831 determines the importance of each object. The condition determination unit 832 determines the learning conditions for NeRF learning based on the importance of each object determined by the importance determination unit 831. Below, a case where three objects, object A, object B, and object C, are present within the angle of view of the virtual camera will be described as an example.

[0063] For example, when the importance of object A is determined to be higher than the importance of other objects, the learning conditions of the NeRF are determined so that a radiance field can be estimated such that object A will be a high-quality image in the virtual viewpoint image generated by the trained NeRF. The importance of an object is determined, for example, based on the size of the area of ​​the object 106 in the virtual viewpoint image. For example, the importance of an object 106 having a larger area is determined to be higher. The method of determining the importance of the object 106 is not limited to the above-mentioned method, and for example, the importance of the object 106 may be determined by the user specifying the object 106 that the user wishes to view intensively via the UI panel 103 or the like.

[0064] For example, the condition determination unit 832 determines the learning conditions so that NeRF learning is performed by using more captured image data of an image capturing device 101 capable of capturing an object 106 with high importance in its angle of view as learning image data than captured image data of other image capturing devices 101. By determining the learning conditions in this way, the captured image data used as learning image data for NeRF learning is reduced, and the radiance field of an object with high importance can be estimated with high accuracy while shortening the learning time of NeRF.

[0065] The operation of the image processing device 102 will be described with reference to Figs. 9 to 13. Fig. 9 is a flowchart showing an example of a series of processing flows of the image processing device 102 according to the third embodiment. The processing of each step shown in the flowchart in Fig. 9 is realized by the CPU 201 reading a predetermined program from the storage unit 203, developing it in the main memory 202, and executing it. In the description of the flowchart shown in Fig. 9, steps that perform the same processing as the steps shown in the flowchart in Fig. 5 are denoted by the same reference numerals, and description thereof will be omitted.

[0066] First, the image processing device 102 executes the processes from S501 to S503. After S503, in S901, the importance determination unit 831 determines the importance of one or more objects present in the imaging target area for each object. Details of the process of determining the importance of the object in S901 will be described later. After S901, in S902, the condition determination unit 832 determines the learning conditions of the NeRF based on the importance of the object determined in S901. Details of the process of determining the learning conditions in S902 will be described later. After S902, in S903, the learning unit 804 learns the NeRF based on the learning conditions determined in S902, and transmits the learned NeRF obtained as a result of the learning to the generation unit 405. Details of the learning process of the NeRF in S903 will be described later. After S903, the image processing device 102 executes the processes of S506 and S507, and after S507, the image processing device 102 ends the process of the flowchart shown in FIG. 9.

[0067] Fig. 10 is a diagram showing an example of the arrangement of a plurality of imaging devices 101 according to the third embodiment and an object 106 existing in an imaging target area. Fig. 10 shows the arrangement of ten imaging devices 101 (101a to 101j) as an example. Fig. 10 also shows three objects 106 (106a to 106c) as an example. Fig. 10 also shows the position of a virtual camera 301 in a three-dimensional space.

[0068] FIG. 11 is a diagram showing an example of a captured image obtained by imaging using each imaging device 101 according to the third embodiment. FIG. 11(a) shows an example of a virtual viewpoint image 1100 at the position and orientation of the virtual camera 301, and the virtual viewpoint image 1100 includes images corresponding to the objects 106a to 106c. FIG. 11(b) shows an example of a captured image 1101 obtained by imaging using the imaging device 101a, and the captured image 1101 includes images corresponding to the objects 106a to 106c. FIG. 11(c) to (k) show examples of captured images 1102 to 1110 obtained by imaging using the imaging devices 101b to 101j, similar to FIG. 11(b). Also, each of the captured images 1102 to 1110 includes images corresponding to the objects 106a to 106c.

[0069] For example, the importance determination unit 831 first calculates the size of each area of ​​the objects 106a to 106c in the virtual viewpoint image. Next, the importance determination unit 831 determines the importance of each object based on the calculated size of the area. Also, the importance determination unit 831 calculates the size of each area of ​​the objects 106a to 106c in each of the captured images 1101 to 1110. Next, the importance determination unit 831 determines the influence of the area of ​​each object 106 for each of the captured images 1101 to 1110 based on the determined importance of each object 106 and the size of the area of ​​each object 106 in each of the captured images 1101 to 1110. Here, the influence is an index representing the degree of influence that the learning image data has on the estimation accuracy of the radiance field of each object 106 when learning NeRF using the captured image data of each imaging device 101 as learning image data. Specifically, the influence degree is an index that represents the degree of influence that the area of ​​each object 106 in the captured image used to train the NeRF has on the image quality of the area of ​​each object 106 in the virtual viewpoint image generated by the trained NeRF.

[0070] The condition determination unit 832 determines the learning conditions so that NeRF learning is performed, for example, by using as learning image data the data of a captured image in which the influence degree of the area of ​​each object 106 in the captured image is equal to or higher than the importance of the corresponding object 106. Furthermore, the condition determination unit 832 may set the learning conditions so that the data of a captured image including an area of ​​an object 106 with a higher importance is used as learning image data more times to perform NeRF learning.

[0071] For example, when the importance of the object 106a is greatest, the importance of the object 106b is greatest, and the importance of the object 106c is greatest, the condition determining unit 832 determines whether the number of learning times is n A >n B >n C Determine the NeRF learning conditions so that: Arepresents the number of times of NeRF learning using the captured image data including the area of ​​the object 106a with the highest importance as learning image data. B represents the number of times of NeRF learning using the captured image data including the area of ​​the object 106b with the second highest importance as learning image data. C represents the number of times of NeRF learning using the captured image data including the area of ​​the object 106c with the lowest importance as learning image data. Note that when the area of ​​the object 106c in the virtual viewpoint image 1100 is very small, that is, when the importance of the object 106c is very low, the condition determination unit 832 may C You may set the value to 0.

[0072] Furthermore, the condition determination unit 832 may determine a learning condition for performing learning using captured image data of all the imaging devices 101 as learning image data (hereinafter referred to as "default learning") in addition to the above-mentioned learning based on the importance of the object 106. In this case, for example, if the total number of learning times is n, the condition determination unit 832 may determine a learning condition for performing learning using the captured image data of all the imaging devices 101 as learning image data. A , n B , n C , and n D n so that the sum of A , n B , n C , and n D Set the value of n D represents the number of times of learning in the default learning. Note that, in order to sufficiently perform learning based on the importance of the object 106, n D It is advisable to set the value as small as possible.

[0073] The process of S901 will be described with reference to Fig. 12. Fig. 12 is a flowchart showing an example of a process flow in the importance determination unit 831 according to the third embodiment, and is a flowchart showing an example of a detailed process flow in S901. The process of the flowchart is executed after the process of S503 shown in Fig. 9. First, in S1201, the importance determination unit 831 specifies the image capture device 101 having an attitude closest to that of the virtual camera 301. For example, the importance determination unit 831 specifies the image capture device 101 having an attitude closest to that of the virtual camera 301 based on the inner product value of the direction vector indicating the attitude of the virtual camera 301 and the direction vector indicating the attitude of the virtual camera 301, as in Equation (1).

[0074] Next, in S1202, the importance determination unit 831 selects two or more imaging devices 101 sparsely from among all the imaging devices 101, based on the imaging device 101 identified in S1201. Using the two or more imaging devices 101 selected in S1202, the importance determination unit 831 estimates a rough shape of the object 106 in the next step. Therefore, in S1202, it is preferable that the importance determination unit 831 selects, from among all the imaging devices 101, two or more imaging devices arranged at approximately equal intervals, such as 60 degrees, in the horizontal direction, such that the angle formed by the directional vectors of the attitudes of the imaging devices 101 is equal to or greater than 60 degrees.

[0075] Specifically, for example, the importance determination unit 831 selects the imaging device 101 having the closest orientation to the orientation of the virtual camera from among all the imaging devices 101 shown as an example in FIG. 3(a), and then selects one imaging device 101 for every four imaging devices in a clockwise direction. Through such selection, five imaging devices 101 shown as an example in FIG. 3(c) are selected. The more imaging devices 101 are selected, the higher the accuracy of estimating the shape of the object 106. However, the more imaging devices 101 are selected, the greater the amount of calculation required for the shape estimation process. Therefore, it is preferable to change the number or interval of the imaging devices 101 to be selected depending on the use case.

[0076] After S1202, in S1203, the importance determination unit 831 estimates the shape of the object 106 using the captured image data of the imaging device 101 selected in S1202 from the multi-view image data acquired in S502. For example, the importance determination unit 831 estimates the shape of the object 106 by a visual hull (hereinafter referred to as "VH") method. Specifically, the importance determination unit 831 first extracts the area of ​​the object 106 as a silhouette from each captured image. Next, the importance determination unit 831 estimates the three-dimensional shape of the object 106 from the extracted silhouette and the imaging parameters of the imaging device 101 acquired in S501 to generate three-dimensional shape data. Specifically, the importance determination unit 831 projects the extracted silhouette into a three-dimensional space based on the imaging parameters of the imaging device 101, and the importance determination unit 831 defines and estimates the intersection of the projection areas as the shape of the object.

[0077] More specifically, the importance determination unit 831 first defines a three-dimensional space in which voxels of a predetermined size are laid out. Next, the importance determination unit 831 projects each voxel laid out in the three-dimensional space onto a two-dimensional captured image of each imaging device 101 from the three-dimensional coordinates using the imaging parameters. The importance determination unit 831 determines whether each projected voxel overlaps with the silhouette of the object 106 extracted in each captured image. For a voxel determined to overlap with the silhouette of the object 106 in the number of captured images equal to or greater than a predetermined threshold, the importance determination unit 831 determines that the voxel is part of the shape of the object 106. For example, when laying out the voxels in the three-dimensional space, the importance determination unit 831 initializes the flag values ​​of all voxels to 0, and changes the flag values ​​of voxels determined to be part of the shape of the object 106 to 1. A set of voxels whose flag value is 1 becomes three-dimensional shape data that indicates the shape of the object 106 .

[0078] In this embodiment, the importance determination unit 831 is described as estimating the shape of the object 106 using the VH method, but the method of estimating the shape of the object 106 is not limited to a method using a three-dimensional shape reconstruction technique such as the VH method. For example, the importance determination unit 831 may estimate the shape of the object 106 using a trained model obtained as a result of deep learning or the like. In this case, the importance determination unit 831 does not necessarily estimate the shape of the object 106 using the captured image data of multiple imaging devices 101. For example, the importance determination unit 831 may estimate the shape of the object 106 using only the captured image data of one imaging device 101, such as the captured image data of the imaging device 101 specified in S1201.

[0079] After S1203, in S1204, the importance determination unit 831 reversely projects the shape of the object 106 estimated in S1203 onto the virtual viewpoint image to be generated based on the virtual camera information. However, when there are a plurality of objects 106, it is assumed that the virtual camera and the imaging device 101 are consistent with each other regarding which object 106 the area of ​​the shape of each object 106 corresponds to during reverse projection. Next, in S1205, the importance determination unit 831 determines the importance of the object 106 based on the size of the silhouette of the object 106 in the virtual viewpoint image reversely projected in S1204, that is, the size of the area of ​​the object 106 in the virtual viewpoint image. Hereinafter, the size of the area of ​​the object 106 will be described as being determined by the number of pixels included in the area, but the size of the area of ​​the object 106 may be determined by other indices, such as the maximum number of pixels in the vertical or horizontal direction of the area.

[0080] For example, when the regions of the shapes of the objects 106a to 106c are projected back onto the virtual viewpoint image, the size of the silhouette of the object 106a in the virtual viewpoint image is assumed to be 500 pixels. Also, the size of the silhouette of the object 106b in the virtual viewpoint image is assumed to be 300 pixels, and the size of the silhouette of the object 106c is assumed to be 200 pixels. In this case, for example, the importance determination unit 831 determines that the importance of the objects 106a, 106b, and 106c is 0.5, 0.3, and 0.2, respectively.

[0081] In the above example, the importance of the object 106 has been described as being determined based only on the size of the area of ​​the object 106 in the virtual viewpoint image. In addition, the importance may be determined based on a provisional importance previously set by the user. For example, the user sets a provisional importance for the object 106 in advance. The user can freely set the provisional importance of the object 106 based on the importance of the object 106 that the user considers, such as the type of the object 106, the user's interests or concerns, etc. In this case, for example, the importance determination unit 831 calculates the true importance by performing a predetermined weighting based on the provisional importance previously set for each object 106 on the importance determined based on the size of the area of ​​the object 106 in the virtual viewpoint image.

[0082] Specifically, for example, the user pre-sets 0.2, 0.8, and 0.0 as provisional importance for objects 106a, 106b, and 106c, respectively. The importance determination unit 831 determines the true importance by, for example, multiplying the provisional importance set in advance by the ratio of the sizes of the areas of the objects 106 in the virtual viewpoint image. That is, the importance determination unit 831 multiplies the provisional importance set in advance by the ratio of the sizes of the areas of the objects 106 in the virtual viewpoint image by I. s , the provisional importance set by the user is I u Then, the true importance I can be expressed as follows: I = Is × Iu Equation (3)

[0083] For example, in the above case, the true importance of object 106a, object 106b, and object 106c are 0.5×0.2=0.1, 0.3×0.8=0.24, and 0.2×0.0=0.00, respectively. Note that the importance determination unit 831 may correct the importance of each object 106 so that the sum of the importance of each object 106 is 1. In the above example, the true importance of object 106a, object 106b, and object 106c are 0.29, 0.71, and 0.00, respectively.

[0084] After S1205, in S1206, the importance determination unit 831 determines the influence when learning is performed using the captured image data of each imaging device 101 as learning image data, based on the size of the area of ​​each object 106 in the captured image of each imaging device 101. Specifically, for example, the importance determination unit 831 determines the influence when learning NeRF using a method similar to the method of determining the importance of the object 106 in S1205. More specifically, for example, the importance determination unit 831 determines the influence of each object 106 for each captured image data, based on the ratio of the size of the area of ​​each object 106 in the captured image of each imaging device 101.

[0085] The information on the size of the area of ​​each object 106 in the captured image of each imaging device 101 can use information on the size of the silhouette of the object 106 extracted when estimating the shape of the object 106 using the VH method in S1203. If the silhouette of the object 106 is not extracted in S1203, the size of the silhouette of the object 106 when the area of ​​the shape of the object 106 estimated in S1203 is back-projected onto the captured image of each imaging device 101 may be calculated. However, if there are multiple objects 106, it is assumed that there is consistency between the virtual camera and the imaging device 101 regarding which object 106 the area of ​​the shape of each object 106 corresponds to when back-projecting. After S1206, the importance determination unit 831 ends the process of the flowchart shown in FIG. 12, i.e., the process of S901.

[0086] The process of S902 will be described with reference to FIG. 13. FIG. 13 is a flowchart showing an example of the process flow in the condition determination unit 832 according to the third embodiment, and is a flowchart showing an example of a detailed process flow in S902. The process of the flowchart is executed after the process of S901 shown in FIG. 9. First, in S1301, the condition determination unit 832 determines the number of times of learning (the number of epochs) of default learning. Here, as described above in the second embodiment, the default learning is learning in which each of the captured image data of all the imaging devices 101 is used as learning image data in NeRF learning. Also, the number of times of learning (the number of epochs) of default learning indicates how many times learning using the captured image data as learning image data is performed for each captured image data in the default learning. Note that in the subsequent step, the condition determination unit 832 determines the number of epochs of learning using the captured image data with a high degree of influence as learning image data. Therefore, in order to increase the number of epochs, it is preferable that the condition determination unit 832 determines the number of epochs of default learning to be as small as possible.

[0087] Next, in S1302, the condition determination unit 832 determines the number of epochs for each imaging device 101 based on the importance of each object 106 and the influence of each object 106 for each captured image data determined in S901. For example, the condition determination unit 832 determines the number of epochs for the imaging device 101 so as to increase the number of learning times in which captured image data having an influence of the object 106 that is equal to or higher than the importance of the object 106 is used as learning image data. Specifically, for example, a value several percent lower than the importance of each of the objects 106a to 106c is set as a threshold value, and the number of epochs for the imaging device 101 is determined so as to increase the number of learning times in which captured image data having an influence higher than the threshold value is used as learning image data.

[0088] For example, the condition determining unit 832 may set each piece of captured image data having a higher influence level than the threshold value of the importance level of the object 106a as learning image data among all the captured image data constituting the multi-viewpoint image data, and may set n A Similarly, the condition determining unit 832 determines that each piece of captured image data having a higher influence level than the threshold value of the importance level of the object 106b is to be used as learning image data, and determines that learning is to be performed n times for each piece of learning image data. B Similarly, the condition determining unit 832 determines that each piece of captured image data having a higher influence level than the threshold value of the importance level of the object 106b is to be used as learning image data, and determines that learning is to be performed n times for each piece of learning image data. C In this case, for all of the objects 106a to 106c, the number of learning times is determined to be n for the captured image data having a degree of influence higher than the threshold value of the importance. A , n B , n C , and n D The sum of n D is the default number of epochs for training.

[0089] The condition determination unit 832 may determine the number of times of learning for each learning image data according to the importance of each object 106. In other words, the condition determination unit 832 may determine the number of times of learning for each learning image data so that the more important the object area of ​​the captured image data is, the more times the NeRF is learned by using the data as learning image data. A , n B , and n C Specifically, when the importance of each object 106 is in the order of object 106a, object 106b, and object 106c, the condition determining unit 832 may determine the value of n A , n B , n C n A , n B , and n C Determine the value of

[0090] In the present embodiment, the captured image data having a degree of influence higher than the importance threshold is used as the learning image data, and the learning image data is used a larger number of times to perform learning. However, the determination of the number of epochs is not limited to this. For example, the condition determination unit 832 may decrease the number of epochs of learning in which captured image data having a degree of influence lower than the importance threshold is used as the learning image data. Also, for example, the condition determination unit 832 may increase the number of epochs of learning in which captured image data having a degree of influence equivalent to the importance is used as the learning image data.

[0091] After S1302, the condition determination unit 832 ends the process of the flowchart shown in FIG. 13, i.e., the process of S902. After S902, the image processing device 102 executes the process of S903. Specifically, as described above, after S902, in S903, the learning unit 804 performs NeRF learning based on the learning conditions determined in S902. More specifically, In S903, the learning unit 804 performs learning of the NeRF using the captured image data of each image capturing device 101 as learning image data according to the number of epochs corresponding to each image capturing device 101 determined in S1302.

[0092] In this embodiment, the condition for terminating the learning process of NeRF in the learning unit 804 is that the number of learning times reaches the number of epochs determined in S1302. However, the condition for terminating the learning process is not limited to this. For example, as described above in the first and second embodiments, the learning unit 804 may terminate the learning of NeRF when the update amount of the weight parameter of NeRF falls within a certain range or the change amount of the image generated by NeRF becomes smaller than a certain criterion. In this case, the learning unit 804 may change the condition for terminating the learning depending on the captured image data used as the learning image data for NeRF learning. For example, the learning unit 804 may set a stricter condition for terminating the learning for learning in which captured image data having the same or higher influence as the importance is used as the learning image data. In addition, for example, the learning unit 804 may set a lenient condition for terminating the learning for learning in which captured image data having an influence lower than the importance is used as the learning image data.

[0093] The learning unit 804 may change the weight of NeRF learning for each learning image data depending on the captured image data used as learning image data in NeRF learning. For example, the learning unit 804 may set a high weight of NeRF learning for captured image data having a higher influence than the importance, assuming that the captured image data is important image data in NeRF learning, when learning using the captured image data as learning image data. Conversely, the learning unit 804 may set a low weight of NeRF learning for captured image data having a lower influence than the importance, assuming that the captured image data is not important image data in NeRF learning, when learning using the captured image data as learning image data.

[0094] In the present embodiment, the area of ​​the estimated shape of each object 106 is inversely projected onto the virtual viewpoint image to be generated, and the importance of each object 106 is determined based on the size of the silhouette of each object 106 in the virtual viewpoint image. Also, in the present embodiment, the influence of each object 106 is determined based on the size of the silhouette corresponding to each object 106 extracted from the captured image of each imaging device 101. However, the method of determining the importance or influence of the object 106 is not limited to the above-mentioned method.

[0095] For example, the importance may be determined based on the distance from the virtual camera to the object 106, which may be calculated based on the estimated three-dimensional shape of the object 106 and the position and orientation of the virtual camera. Specifically, the importance determination unit 831 may determine the importance of each object 106 according to the distance from the virtual camera, such that the importance of the object 106 is higher as the object 106 is closer to the position of the virtual camera. Similarly, the influence may be determined based on the distance from the imaging device 101 to the object 106, which may be calculated based on the estimated three-dimensional shape of the object 106 and the position and orientation of each imaging device 101. Specifically, the importance determination unit 831 may determine the influence of each object 106 according to the distance from the imaging device 101, such that the influence of the object 106 is higher as the object 106 is closer to the position of the imaging device 101. Also, for example, the importance determination unit 831 may use the volume of the estimated three-dimensional shape of each object 106 as an index for determining the importance and influence. Also, for example, the estimated volume of the three-dimensional shape of each object 106 and the distance to the object 106 may be used as an index for determining the importance and the influence.

[0096] As described above, in this embodiment, the image processing device 102 is configured to determine the learning conditions for NeRF learning based on the estimated three-dimensional shape of the object. According to the image processing device 102 configured in this manner, it is possible to reduce the number of times of NeRF learning using captured image data that does not include the object 106 with high importance as learning image data, and to shorten the learning time of NeRF. Furthermore, according to the image processing device 102, it is possible to estimate the radiance field of the object 106 with high importance with high accuracy while shortening the learning time of NeRF, and to generate a virtual viewpoint image with little deterioration in quality in the area of ​​the object 106.

[0097] [Embodiment 4] The image processing device 102 according to the third embodiment reversely projects the area of ​​the shape of the object 106 roughly estimated using the VH method onto the virtual viewpoint image, and determines the importance of the object 106 based on the size of the area of ​​the object 106 in the virtual viewpoint image. Also, the image processing device 102 according to the third embodiment determines the influence of the object 106 in the captured image data used as learning image data when learning NeRF based on the size of the area of ​​the object 106 in each captured image. Furthermore, the image processing device 102 according to the third embodiment determines the number of learning times (the number of epochs) when the captured image data is used as learning image data based on the determined importance of the object 106 and the determined influence of the object 106. Here, the estimation process of the three-dimensional shape of the object 106 using the VH method or the like requires a large amount of calculation and a long processing time. Therefore, in the fourth embodiment, a form in which the size of the area of ​​the object 106 in the virtual viewpoint image is estimated from the data of the captured image, which is a two-dimensional image, without estimating the shape of the object 106 will be described.

[0098] The functional configuration of the image processing device 102 according to the fourth embodiment is similar to that of the image processing device 102 according to the third embodiment, and only the processing of the importance determination unit 831 according to the fourth embodiment differs from the processing of the importance determination unit 831 according to the third embodiment. That is, in each step in the flowchart shown in Fig. 9, among the processing of the image processing device 102 according to the fourth embodiment, only the processing of S901 differs from the processing of the image processing device 102 according to the third embodiment. Hereinafter, the image processing device 102 according to the fourth embodiment will be simply referred to as the "image processing device 102".

[0099] Fig. 14 is a flowchart showing an example of the processing flow of the importance determination unit 831 according to the fourth embodiment, and is a flowchart showing an example of the detailed processing flow in S901 shown in Fig. 9. The importance determination unit 831 according to the fourth embodiment (hereinafter simply referred to as "importance determination unit 831") extracts the area of ​​the object 106 in the captured image by performing two-dimensional segmentation on the captured image. Specific steps are as follows.

[0100] First, in S1401, the importance determination unit 831 specifies the image capture device 101 that is closest to the attitude of the virtual camera, similarly to the importance determination unit 831 according to the third embodiment. Next, in S1402, the importance determination unit 831 extracts the area of ​​the object 106 in the captured image of the image capture device 101 specified in S1401. Specifically, the importance determination unit 831 performs two-dimensional segmentation on the captured image, and extracts the area of ​​the object 106 by separating the area of ​​the main object 106 from the background area in the captured image. For example, the importance determination unit 831 performs segmentation on the captured image using an image processing method such as the SNAKE method, the level set method, or the foreground / background separation method. The segmentation method is not limited to the image processing method as described above. For example, the importance determination unit 831 may perform segmentation on the captured image using a learned model obtained as a result of deep learning such as semantic segmentation. By using such a method, the importance determining section 831 can extract the area of ​​each object 106 in the captured image.

[0101] When multiple objects 106 exist in the imaging target area, labeling is performed for each area of ​​the object 106 after performing segmentation to determine which object 106 each of the multiple areas of the object 106 corresponds to. Furthermore, the importance determination unit 831 may perform segmentation taking into consideration known labels. However, it is assumed that there is consistency between the imaging devices 101 when segmentation is performed regarding which object 106 each area of ​​the object 106 corresponds to.

[0102] Next, in S1403, the importance determination unit 831 obtains the number of pixels included in the area of ​​each object 106 extracted in S1402, and obtains the obtained number of pixels as the size of the area of ​​the object 106 in the virtual viewpoint image. Here, in this embodiment, the importance determination unit 831 does not estimate the three-dimensional shape of the object 106. Therefore, the importance determination unit 831 cannot directly calculate the size of the area of ​​the object 106 in the virtual viewpoint image. Therefore, the importance determination unit 831 regards the size of the area of ​​the object 106 in the captured image of the imaging device 101 specified in S1401 as the size of the area of ​​the object 106 in the virtual viewpoint image.

[0103] Next, in S1404, the importance determination unit 831 determines the importance of each object 106. The process of determining the importance of the object 106 in S1404 is similar to the process of determining the importance of the object 106 in S1205 shown in Fig. 12, and therefore a description thereof will be omitted. Next, in S1405, the importance determination unit 831 extracts the area of ​​each object 106 in the captured image of each imaging device 101 other than the imaging device 101 identified in S1401, among all the imaging devices 101. The method of extracting the area of ​​the object 106 in S1405 is similar to the method of extracting the area of ​​the object 106 in S1402, and therefore a description thereof will be omitted.

[0104] Next, in S1406, the importance determination unit 831 obtains the number of pixels included in the area of ​​each object 106 extracted in S1405, and obtains the obtained number of pixels as the size of the area of ​​the object 106 in each captured image. The importance determination unit 831 also obtains the number of pixels included in the area of ​​each object 106 extracted in S1402, and obtains the obtained number of pixels as the size of the area of ​​the object 106 in the captured image of the imaging device 101 specified in S1401. Note that, as described above in the description of the process of S1402, it is assumed that there is consistency between the imaging devices 101 when performing segmentation regarding which object 106 the areas of the multiple objects 106 correspond to.

[0105] Next, in S1407, the importance determination unit 831 determines the influence of the object 106 in the captured image data used as learning image data in learning NeRF based on the importance of each object 106 determined in S1404 and the size of the area of ​​each object 106 in each captured image acquired in S1406. Specifically, the importance determination unit 831 determines the influence of the object 106 in the captured image data used as learning image data in learning NeRF based on the importance of each object 106 and the size of the area of ​​each object 106 in each captured image. The process of determining the influence of the object 106 in S1407 is the same as the process of determining the influence of the object 106 in S1206 shown in FIG. 12, so a description thereof will be omitted. After S1407, the image processing device 102 ends the process of the flowchart shown in FIG. 14, that is, the process of S901 shown in FIG. 9.

[0106] In this embodiment, as a method for identifying an imaging device 101 that captures an image in which the object 106 is captured similarly to the virtual viewpoint image, the orientation of the virtual camera is compared with the orientation of the imaging device 101 as in S1401. However, the method for identifying the imaging device 101 is not limited to the above-mentioned method. For example, when the object 106 is present at a specific position such as the center of the stage or the center of the imaging target area as in FIG. 1, the size of the area of ​​the object 106 in each captured image is determined by the position, orientation, angle of view, etc. of each imaging device 101. Therefore, the importance determination unit 831 may use at least one of these imaging parameters to identify an imaging device 101 that captures an image in which the object 106 is captured similarly to the virtual viewpoint image.

[0107] As described above, in this embodiment, the image processing device 102 is configured to acquire the size of the area of ​​the object 106 in the virtual viewpoint image and each imaging device by performing two-dimensional segmentation on the captured image, which is a two-dimensional image. In addition, the image processing device 102 is configured to determine the importance of the object 106 based on the size of the area of ​​the object 106 in the virtual viewpoint image acquired in this way. Furthermore, the image processing device 102 is configured to determine the influence of the object 106 for each captured image data based on the determined importance of the object 106 and the size of the area of ​​the object 106 in each captured image acquired in this way. According to the image processing device 102 configured in this way, since the learning conditions can be determined without estimating the three-dimensional shape of the object 106, the time required for pre-processing of NeRF learning can be shortened, and as a result, the time required for the entire NeRF learning process can be shortened.

[0108] [Other embodiments] The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) for implementing one or more functions.

[0109] It should be noted that, within the scope of the present disclosure, the embodiments may be freely combined, any component of each embodiment may be modified, or any component of each embodiment may be omitted.

[0110] [Configuration of this disclosure] The present disclosure includes the following configurations, methods, and programs.

[0111] <Configuration 1> an imaging parameter acquisition means for acquiring imaging parameters of each of a plurality of imaging devices disposed at different positions; an image acquisition means for acquiring data of captured images obtained by imaging using each of the plurality of imaging devices; a viewpoint acquisition means for acquiring virtual viewpoint information including at least one of information indicating a position of a virtual viewpoint and information indicating a line of sight from the virtual viewpoint; a determination means for determining, based on the virtual viewpoint information, a learning condition of a learning model that estimates a radiance field corresponding to an object present in an imaging region of the plurality of imaging devices; a learning means for learning the learning model based on the learning conditions, the imaging parameters, and the captured image data; 13. An image processing device comprising:

[0112] <Configuration 2> the determination means acquires a virtual viewpoint image corresponding to an appearance from the virtual viewpoint based on the virtual viewpoint information, determines the importance of the object when learning the learning model based on an object area that is an image area corresponding to the object in the acquired virtual viewpoint image, and determines the learning condition based on the determined importance of the object; 2. The image processing device according to claim 1,

[0113] <Configuration 3> the determining means estimates a three-dimensional shape of the object using at least a part of data of the plurality of captured images acquired by the image acquiring means, and specifies an object area in the virtual viewpoint image based on the virtual viewpoint information and the estimated three-dimensional shape of the object; 3. The image processing device according to configuration 2,

[0114] <Configuration 4> the determination means estimates a three-dimensional shape of the object using at least a part of data of the plurality of captured images acquired by the image acquisition means, determines importance of the object using at least one of information indicating a volume of the object acquired based on the estimated three-dimensional shape of the object and information indicating a distance from the position of the virtual viewpoint to the object acquired based on the estimated three-dimensional shape of the object, and determines the learning condition based on the determined importance of the object; 4. The image processing device according to any one of configurations 1 to 3,

[0115] <Component 5> The determination means estimates a three-dimensional shape of the object from at least a part of the data of the plurality of captured images acquired by the image acquisition means using a three-dimensional shape reconstruction technique or a trained model obtained as a result of deep learning; 5. The image processing device according to claim 3 or 4.

[0116] <Component 6> the determining means identifies the object region in the virtual viewpoint image using at least one of an image segmentation technique and an image foreground / background separation technique; 4. The image processing device according to configuration 2 or 3,

[0117] <Component 7> the determining means, based on the virtual viewpoint information and the imaging parameters, specifies, from among the plurality of imaging devices, an imaging device that captures the captured image similar to the appearance from the virtual viewpoint, and specifies, as the object region in the virtual viewpoint image, the object region in the captured image obtained by imaging using the specified imaging device; 3. The image processing device according to configuration 2,

[0118] <Component 8> The determining means determines the object region in the captured image obtained by capturing an image using the determined imaging device, using at least one of an image segmentation technique and an image foreground / background separation technique; 8. The image processing device according to configuration 7,

[0119] <Component 9> the virtual viewpoint information includes information indicating a position of the virtual viewpoint, the determining means, based on information indicating a position of the virtual viewpoint and information indicating a position of an imaging device included in the imaging parameters, specifies, from among the plurality of imaging devices, an imaging device that captures the captured image similar to an appearance from the virtual viewpoint; 9. The image processing device according to configuration 7 or 8,

[0120] <Component 10> the virtual viewpoint information includes information indicating a line of sight direction from the virtual viewpoint, the determining means identifies, from among the plurality of imaging devices, an imaging device that captures the captured image similar to the appearance from the virtual viewpoint, based on information indicating a line of sight direction from the virtual viewpoint and information indicating a direction of an optical axis of the imaging device, which is included in the imaging parameters; 10. The image processing device according to any one of configurations 7 to 9,

[0121] <Component 11> the virtual viewpoint information includes information indicating a width of a field of view from the virtual viewpoint, the determining means identifies, from among the plurality of imaging devices, an imaging device that captures the captured image similar to the appearance from the virtual viewpoint, based on information indicating a width of a field of view from the virtual viewpoint and information indicating an angle of view of the imaging device included in the imaging parameters; 11. The image processing device according to any one of configurations 7 to 10,

[0122] <Component 12> the determining means determines an importance of the object when learning the learning model based on the virtual viewpoint information and information indicating a type of the object, and determines the learning condition based on the determined importance of the object; 12. The image processing device according to any one of configurations 1 to 11,

[0123] <Component 13> the determining means determines, for each piece of captured image data, a number of times that the learning model will be trained using the captured image data based on the importance of the object; 13. The image processing device according to any one of configurations 2 to 12,

[0124] <Component 14> the determining means determines a convergence condition of a weight parameter included in the learning model, which is used as a termination condition of the learning model, based on the importance of the object; 14. The image processing device according to any one of configurations 2 to 13,

[0125] <Component 15> the determining means determines the convergence condition based on the importance of the object and a state in which the object appears in the captured image used in the learning model; 15. The image processing device according to claim 14,

[0126] <Component 16> a generating means for estimating a radiance field corresponding to the object using a trained model, the trained model being a trained model, and generating a virtual viewpoint image corresponding to an appearance from the virtual viewpoint; Further comprising: the virtual viewpoint information includes information indicating a position of the virtual viewpoint and information indicating a line of sight from the virtual viewpoint, The generating means generates the virtual viewpoint image by inputting the virtual viewpoint information into the trained model; 16. The image processing device according to any one of configurations 1 to 15,

[0127] <Component 17> an output means for controlling the display of the virtual viewpoint image and outputting the same on a display device; 17. The image processing device according to claim 16, further comprising:

[0128] <Method> an imaging parameter acquisition step of acquiring imaging parameters of each of a plurality of imaging devices arranged at different positions; an image acquisition step of acquiring data of captured images obtained by imaging using each of the plurality of imaging devices; a viewpoint acquiring step of acquiring virtual viewpoint information including at least one of information indicating a position of a virtual viewpoint and information indicating a line of sight from the virtual viewpoint; determining, based on the virtual viewpoint information, learning conditions for a learning model that estimates a radiance field corresponding to an object present in an imaging region of the plurality of imaging devices; a learning process of learning the learning model based on the learning conditions, the imaging parameters, and the captured image data; 13. An image processing method comprising:

[0129] <Method> A program for causing a computer to function as the image processing device according to any one of configurations 1 to 17. [Explanation of symbols]

[0130] 102 Image processing device 400 Virtual camera information acquisition unit 401 Imaging parameter acquisition unit 402 Image Acquisition Unit 403 Specific part 404 Learning Department

Claims

1. an imaging parameter acquisition means for acquiring imaging parameters of each of a plurality of imaging devices disposed at different positions; an image acquisition means for acquiring data of captured images obtained by imaging using each of the plurality of imaging devices; a viewpoint acquisition means for acquiring virtual viewpoint information including at least one of information indicating a position of a virtual viewpoint and information indicating a line of sight from the virtual viewpoint; a determination means for determining, based on the virtual viewpoint information, a learning condition of a learning model that estimates a radiance field corresponding to an object present in an imaging region of the plurality of imaging devices; a learning means for learning the learning model based on the learning conditions, the imaging parameters, and the captured image data; 13. An image processing device comprising:

2. the determination means acquires a virtual viewpoint image corresponding to an appearance from the virtual viewpoint based on the virtual viewpoint information, determines the importance of the object when learning the learning model based on an object area that is an image area corresponding to the object in the acquired virtual viewpoint image, and determines the learning condition based on the determined importance of the object; 2. The image processing device according to claim 1,

3. the determining means estimates a three-dimensional shape of the object using at least a part of data of the plurality of captured images acquired by the image acquiring means, and specifies an object area in the virtual viewpoint image based on the virtual viewpoint information and the estimated three-dimensional shape of the object; 3. The image processing device according to claim 2,

4. the determination means estimates a three-dimensional shape of the object using at least a part of data of the plurality of captured images acquired by the image acquisition means, determines importance of the object using at least one of information indicating a volume of the object acquired based on the estimated three-dimensional shape of the object and information indicating a distance from the position of the virtual viewpoint to the object acquired based on the estimated three-dimensional shape of the object, and determines the learning condition based on the determined importance of the object; 2. The image processing device according to claim 1,

5. The determination means estimates a three-dimensional shape of the object from at least a part of the data of the plurality of captured images acquired by the image acquisition means using a three-dimensional shape reconstruction technique or a trained model obtained as a result of deep learning; 4. The image processing device according to claim 3,

6. the determining means identifies the object region in the virtual viewpoint image using at least one of an image segmentation technique and an image foreground / background separation technique; 3. The image processing device according to claim 2,

7. the determining means, based on the virtual viewpoint information and the imaging parameters, specifies, from among the plurality of imaging devices, an imaging device that captures the captured image similar to the appearance from the virtual viewpoint, and specifies, as the object region in the virtual viewpoint image, the object region in the captured image obtained by imaging using the specified imaging device; 3. The image processing device according to claim 2,

8. The determining means determines the object region in the captured image obtained by capturing an image using the determined imaging device, using at least one of an image segmentation technique and an image foreground / background separation technique; 8. The image processing device according to claim 7,

9. the virtual viewpoint information includes information indicating a position of the virtual viewpoint, the determining means identifies, from among the plurality of imaging devices, an imaging device that captures the captured image similar to the appearance from the virtual viewpoint, based on information indicating a position of the virtual viewpoint and information indicating a position of an imaging device included in the imaging parameters; 8. The image processing device according to claim 7,

10. the virtual viewpoint information includes information indicating a line of sight direction from the virtual viewpoint, the determining means identifies, from among the plurality of imaging devices, an imaging device that captures the captured image similar to the appearance from the virtual viewpoint, based on information indicating a line of sight direction from the virtual viewpoint and information indicating a direction of an optical axis of the imaging device, which is included in the imaging parameters; 8. The image processing device according to claim 7,

11. the virtual viewpoint information includes information indicating a width of a field of view from the virtual viewpoint, the determining means identifies, from among the plurality of imaging devices, an imaging device that captures the captured image similar to the appearance from the virtual viewpoint, based on information indicating a width of a field of view from the virtual viewpoint and information indicating an angle of view of the imaging device included in the imaging parameters; 8. The image processing device according to claim 7,

12. the determining means determines an importance of the object when learning the learning model based on the virtual viewpoint information and information indicating a type of the object, and determines the learning condition based on the determined importance of the object; 2. The image processing device according to claim 1,

13. the determining means determines, for each piece of captured image data, a number of times that the learning model will be trained using the captured image data based on the importance of the object; 3. The image processing device according to claim 2,

14. the determining means determines a convergence condition of a weight parameter included in the learning model, which is used as a termination condition of the learning model, based on the importance of the object; 3. The image processing device according to claim 2,

15. the determining means determines the convergence condition based on the importance of the object and a state in which the object appears in the captured image used in the learning model; The image processing device according to claim 14 .

16. a generating means for estimating a radiance field corresponding to the object using a trained model, the trained model being a trained model, and generating a virtual viewpoint image corresponding to an appearance from the virtual viewpoint; Further comprising: the virtual viewpoint information includes information indicating a position of the virtual viewpoint and information indicating a line of sight from the virtual viewpoint, The generating means generates the virtual viewpoint image by inputting the virtual viewpoint information into the trained model; 2. The image processing device according to claim 1,

17. an output means for controlling the display of the virtual viewpoint image and outputting the same on a display device; 17. The image processing apparatus according to claim 16, further comprising:

18. an imaging parameter acquisition step of acquiring imaging parameters of each of a plurality of imaging devices arranged at different positions; an image acquisition step of acquiring data of captured images obtained by imaging using each of the plurality of imaging devices; a viewpoint acquiring step of acquiring virtual viewpoint information including at least one of information indicating a position of a virtual viewpoint and information indicating a line of sight from the virtual viewpoint; determining, based on the virtual viewpoint information, learning conditions for a learning model that estimates a radiance field corresponding to an object present in an imaging region of the plurality of imaging devices; a learning process of learning the learning model based on the learning conditions, the imaging parameters, and the captured image data; 13. An image processing method comprising:

19. A program for causing a computer to function as the image processing device according to any one of claims 1 to 17.

Images