Image processing system, image processing method, and storage medium

Abstract

To improve the quality of a virtual viewpoint image.SOLUTION: A virtual viewpoint image of a subject from a designated virtual viewpoint is generated based on a group of images of the subject from a plurality of directions. A first three dimensional model representing the subject is generated based on the image group. A second three dimensional model representing a specific part of the subject is generated based on at least two images extracted from the image group. A first virtual viewpoint image of the first three dimensional model from a virtual viewpoint is generated. A second virtual viewpoint image of the second three dimensional model from the virtual viewpoint is generated. A virtual viewpoint image indicating a correction result of the first virtual viewpoint image is generated by combining a plurality of virtual viewpoint images each representing at least a part of the object, including the second virtual viewpoint image.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to an image processing system, an image processing method, and a program, and in particular to generating a virtual viewpoint image. [Background technology]

[0002] In recent years, technology for generating virtual viewpoint images has become known. In this technology, multiple imaging devices are installed in different locations to perform synchronized shooting from multiple viewpoints. Images from the multiple viewpoints obtained by the shooting can then be used to generate virtual viewpoint images from any viewpoint, not just images from the viewpoints of the imaging devices. Services using such technology can capture soccer, baseball, or other sports. In this case, businesses and others can freely set virtual viewpoints in a three-dimensional virtual space generated based on the captured images. This makes it possible to watch a game while moving the virtual viewpoint in the virtual space. This virtual viewpoint image technology can provide a higher sense of realism than conventional technologies.

[0003] When generating a virtual viewpoint image, a three-dimensional model of a subject is generated based on images from multiple viewpoints. The virtual viewpoint image is then generated based on the three-dimensional model. Therefore, if the quality of the three-dimensional model deteriorates, the quality of the virtual viewpoint image also deteriorates. On the other hand, there may be a large difference between the shape of the generated three-dimensional model and that of the subject. In such cases, techniques for correcting the shape of the three-dimensional model to improve the quality of the three-dimensional model have been known. For example, Patent Document 1 discloses a method for correcting the shape of a three-dimensional model generated by a volume intersection method using color information. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Publication No. 2022-139243 Summary of the Invention [Problem to be solved by the invention]

[0005] The method described in Patent Document 1 has a problem in that shape correction is difficult when the color variance in a part of the three-dimensional model is small. For this reason, there has been a demand for an alternative method for improving the quality of virtual viewpoint images.

[0006] The present disclosure aims to improve the quality of virtual viewpoint images. [Means for solving the problem]

[0007] An image processing system according to an embodiment of the present disclosure includes the following configuration: 1. An image processing system that generates a virtual viewpoint image of a subject from a specified virtual viewpoint based on a group of images of the subject taken from a plurality of directions, a model generation means for generating a first three-dimensional model representing the subject based on the image group, and for generating a second three-dimensional model representing a specific portion of the subject based on at least two images extracted from the image group; an image generation means for generating a first virtual viewpoint image of the first three-dimensional model from the virtual viewpoint, and for generating a second virtual viewpoint image of the second three-dimensional model from the virtual viewpoint; a synthesis means for synthesizing a plurality of virtual viewpoint images, each representing at least a part of the subject, including the second virtual viewpoint image, to generate a virtual viewpoint image that shows a result of correcting the first virtual viewpoint image; Equipped with. [Effects of the Invention]

[0008] The quality of the virtual viewpoint image can be improved. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram showing an example of the arrangement of an image processing system according to an embodiment. [Figure 2]FIG. 1 is a diagram showing an example of installation of an imaging device according to an embodiment. [Figure 3] FIG. 2 is a diagram showing an example of the hardware configuration of an apparatus according to an embodiment. [Figure 4] FIG. 1 is a diagram showing an example of the functional configuration of an image processing system according to an embodiment. [Figure 5] FIG. 10 is a diagram showing an example of a virtual viewpoint image. [Figure 6] FIG. 1 is a diagram showing an example of the functional configuration of an image processing system according to an embodiment. [Figure 7] 1 is a flowchart showing the flow of an image processing method according to an embodiment. [Figure 8] FIG. 10 is a diagram showing an example of a UI of an operation application. [Figure 9] 1 is a flowchart showing the flow of an image processing method according to an embodiment. [Figure 10] FIG. 10 is a diagram showing an example of a UI of an operation application. [Figure 11] FIG. 10 is a diagram showing an example of a UI of an operation application. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] FIG. 1 is a diagram showing the configuration of an image processing system 100 according to an embodiment. First, components of the image processing system 100 will be described with reference to FIG. 1. The image processing system 100 includes a plurality of image capture devices 110, an image generation device 120, a display control device 130, a display device 140, and an operation terminal 150. The image capture devices 110, the image generation devices 120, the display control device 130, and the operation terminal 150 are connected to one another. In this embodiment, these devices are connected via a communication cable such as a Local Area Network (LAN) cable. Furthermore, the image generation device 120 and the display device 140 are connected via a video signal transmission cable. Alternatively, these devices may be connected to one another wirelessly.

[0012] The image processing system 100 can generate a virtual viewpoint image of an object from a specified virtual viewpoint based on a group of images of the object taken from multiple directions. The image processing system 100 performs a virtual viewpoint image correction process, which will be described later. The image processing system 100 can also perform a three-dimensional model generation process and a virtual viewpoint image generation process. The image processing system 100 can execute these three processes sequentially or in parallel. In this embodiment, the group of images of the object are collected in the image generation device 120. The image generation device 120 then generates a virtual viewpoint image by performing processes such as generating a three-dimensional model and rendering based on the virtual viewpoint. The virtual viewpoint image can be displayed on the operation terminal 150 or another user terminal.

[0013] The imaging device 110 is, for example, a digital camera capable of capturing images (still images or video). FIG. 2 is a diagram showing an example of installation of the imaging devices 110. Each imaging device 110 is installed to surround a specific area, for example, a playing field in a stadium. Each imaging device 110 captures an image of a subject within the area (or a video composed of multiple frame images). A group of images obtained by capturing the images is transmitted from each imaging device 110 to the image generation device 120. In the three-dimensional model generation process, the imaging device 110 can capture images at a frequency of, for example, 60 times per second and transmit the group of images obtained by capturing the images to the image generation device 120. Note that a unique camera ID is recorded in the imaging device 110, such as a digital camera. The imaging device 110 can add the camera ID to the metadata of the images to be transmitted.

[0014] The image generation device 120 generates a virtual viewpoint image. The image generation device 120 can also generate a three-dimensional model used to generate the virtual viewpoint image. The image generation device 120 is, for example, a server device. The image generation device 120 can have a database function and an image processing function.

[0015] A background image can be registered in advance in the database of the image generating device 120. The background image can be obtained by the imaging device 110 capturing an image of a scene in which no subject is present, such as a stadium before starting to capture the subject. The image generating device 120 can acquire such a background image from the imaging device 110.

[0016] Furthermore, the image generating device 120 can separate a foreground image from an image generated by the imaging device 110 capturing an image. The foreground image is an image of a specific object that is the subject. The image generating device 120 can perform image processing to extract the foreground image as an object image from an image captured in a scene in which a subject is present. Note that the type of the specific object is not particularly limited. For example, the specific object may be a person, a ball, or other object. The image generating device 120 can generate an object image by performing image processing to recognize the specific object. Alternatively, the image generating device 120 may generate an object image according to a background subtraction method.

[0017] In the 3D model generation process, the image generation device 120 generates a first 3D model representing an object based on a group of images of the object from multiple directions. For example, the image generation device 120 separates an object image from images acquired by each of the multiple image capture devices 110. The image generation device 120 then stores the generated object images. Alternatively, the image generation device 120 may store images acquired by each of the multiple image capture devices 110. The image generation device 120 then generates a 3D model based on the object images. The multiple image capture devices 110 can capture images synchronously. The image generation device 120 can generate an object image from each of the multiple images acquired by the multiple image capture devices 110 synchronously. The image generation device 120 then generates a 3D model based on the multiple object images thus acquired.

[0018] Furthermore, in the virtual viewpoint image generation process, the image generation device 120 generates a first virtual viewpoint image of a first three-dimensional model from a virtual viewpoint. In this embodiment, virtual viewpoint information and playback time information are input to the image generation device 120 by a user operation on the display control device 130. Then, the image generation device 120 generates a virtual viewpoint image based on the accumulated captured images, three-dimensional models, and received virtual viewpoint information. The image generation device 120 generates a virtual viewpoint image corresponding to the virtual viewpoint information based on background images, three-dimensional models, and object images managed in a database.

[0019] The virtual viewpoint information is information indicating parameters such as the position of the virtual viewpoint. The virtual viewpoint information can indicate the position, attitude angle, or gaze point position of the virtual viewpoint in a three-dimensional virtual space constructed based on the captured image. In this embodiment, the virtual viewpoint information includes information indicating at least the position and attitude of the virtual viewpoint. For example, the virtual viewpoint information can include information indicating the relative position of the virtual viewpoint with respect to a predetermined origin position such as the center of the stadium where the image was captured, i.e., the front-back, left-right, and up-down positions with respect to the origin position. The virtual viewpoint information can also include information indicating the attitude angles of the virtual viewpoint with respect to the front-back, left-right, and up-down axes.

[0020] Furthermore, the playback time information is information indicating the playback time corresponding to the shooting time of the captured image. The time information can represent, for example, hours, minutes, seconds, or frame numbers. The time information can also indicate the number of frames per second. By specifying the playback time information, a virtual viewpoint image of the subject at the corresponding shooting time is generated. For example, the number of frames per second can be 60 frames. Note that when the imaging device 110 captures a still image, the time of the virtual viewpoint image is uniquely determined. Therefore, playback time information is not necessary.

[0021] In this embodiment, model-based rendering (MBR) is used as a method for generating a virtual viewpoint image. MBR is a method for generating a three-dimensional model representing a three-dimensional shape based on multiple captured images, and then generating a virtual viewpoint image using the three-dimensional model. The multiple captured images are obtained by photographing a subject from multiple directions. A three-dimensional model of a target scene can be obtained by a three-dimensional shape reconstruction method such as a volume intersection method or multi-view stereo (MVS). Using such a three-dimensional model, a virtual viewpoint image can be generated that represents the scene from a virtual viewpoint. Specifically, the virtual viewpoint image can be generated by performing a calculation to project the three-dimensional model onto a projection surface installed at the virtual viewpoint. Note that rendering methods other than MBR may also be used to generate the virtual viewpoint image. For example, a billboard method may be used to generate the virtual viewpoint image.

[0022] The virtual viewpoint image generated by the image generation device 120 is transmitted to a display device 140 such as a liquid crystal display via a video signal transmission cable. The image generation device 120 also stores virtual viewpoint information in a database in order to generate the virtual viewpoint image.

[0023] Furthermore, in the virtual viewpoint image correction process, the image generation device 120 generates a second 3D model representing a specific part of the subject based on at least two images extracted from a group of images of the subject taken from multiple directions. Then, the image generation device 120 generates a second virtual viewpoint image of the second 3D model from a virtual viewpoint. For example, a user can specify a correction time to be corrected via the operation terminal 150. The image generation device 120 then searches for an object image and a 3D model corresponding to the correction time from among the object images and 3D models stored at multiple times. The image generation device 120 transmits the object image and the 3D model corresponding to the correction time obtained by the search to the operation terminal 150. Furthermore, the user can specify a part of the object to be corrected via the operation terminal 150. Then, the image generation device 120 generates a 3D model of the specific part in accordance with the user's instruction. The image generation device 120 then generates a virtual viewpoint image based on the stored virtual viewpoint information corresponding to the correction time and the 3D model of the specific part. Then, the image generating device 120 transmits the generated virtual viewpoint image to the operation terminal 150 via a communication cable.

[0024] The display control device 130 is used by the user to input virtual viewpoint information and time information. The display control device 130 may be, for example, a personal computer (PC) or a tablet. A viewpoint controller 131 and a time controller 132 are connected to the display control device 130. The viewpoint controller 131 may be, for example, a mouse, a keyboard, a six-axis controller, a touch panel, or a momentary rocker switch. The time controller 132 may be, for example, an operation device equipped with a turntable. The user operates the viewpoint controller 131 and the time controller 132. With such a time controller 132, the user can specify the direction in which time advances by the direction of rotation and the length of time by the amount of rotation. Using such a time controller 132, the user can intuitively operate the time.

[0025] Furthermore, the display control device 130 transmits virtual viewpoint information to the image generation device 120 in the virtual viewpoint image generation process. The display control device 130 can convert user input via the viewpoint controller 131 and the time controller 132 into virtual viewpoint information.

[0026] The user can continuously move the virtual viewpoint and the time by operating the viewpoint controller 131 and the time controller 132. On the other hand, the user may move the virtual viewpoint to a preset virtual viewpoint in the three-dimensional virtual space by operating the viewpoint controller 131. The user may also instantaneously move the playback time to a preset time by operating the time controller 132. An instruction to move the virtual viewpoint or the playback time may be recorded in a recording device (not shown) in the display control device 130. The display control device 130 may read out the recorded instruction to move the virtual viewpoint or the playback time and transmit it to the image generation device 120.

[0027] The operation terminal 150 is, for example, a PC. The operation terminal 150 may be equipped with a keyboard for accepting user operations, such as a laptop PC. The operation terminal 150 may also have a mouse 5 for accepting user operations.

[0028] In the virtual viewpoint image correction process, the operation terminal 150 receives a three-dimensional model and an object image from the image generation device 120 in response to user operation. The operation terminal 150 then displays the received three-dimensional model and object image on a screen of the operation terminal 150 via an operation application. The operation application executed on the operation terminal 150 displays the three-dimensional model and object image. The operation terminal 150 also provides a UI for user input to specify a specific part of the subject. In this embodiment, the user specifies, on the UI, a two-dimensional area related to the part of the subject whose shape is to be corrected. The operation application executed on the operation terminal 150 transmits, based on the user instruction, a request to the image generation device 120 to generate a three-dimensional model of the specific part specified by the user. The image generation device 120 then receives, in response, a three-dimensional model of the specific part and a virtual viewpoint image based on the three-dimensional model. The image generation device 120 can then display the received three-dimensional model and virtual viewpoint image.

[0029] In the virtual viewpoint image correction process, the operation terminal 150 can accumulate the virtual viewpoint images received from the image generation device 120 as a response for each trial for each specific body part. Then, the operation terminal 150 synthesizes the multiple virtual viewpoint images received from the image generation device 120. For example, the operation terminal 150 can synthesize a first virtual viewpoint image of a first three-dimensional model of the subject with a second virtual viewpoint image of a second three-dimensional model of a specific body part of the subject. The operation terminal 150 can also synthesize virtual viewpoint images of each body part of the subject. The operation terminal 150 can upload the synthesized virtual viewpoint image to the image generation device 120.

[0030] In the virtual viewpoint image correction process, the image generation device 120 accumulates the virtual viewpoint image uploaded from the operation terminal 150 in the database of the image generation device 120. In this way, the image generation device 120 can store the corrected virtual viewpoint image in the database. When receiving virtual viewpoint information in the virtual viewpoint image generation process, the image generation device 120 can determine whether the same virtual viewpoint information is recorded in the database. If the same virtual viewpoint information is recorded, the image generation device 120 may read the corrected virtual viewpoint image stored in the database according to the settings and transmit it to the display device 140.

[0031] The operation terminal 150 can be realized by a computer including a processor and a memory. Fig. 3 is a diagram showing an example of the hardware configuration of the operation terminal 150. The operation terminal 150 has a CPU 301, a ROM 302, a RAM 303, an HDD 304, a display unit 305, an input unit 306, and a communication unit 307.

[0032] The CPU 301 can read out a control program stored in the ROM 302 and execute various processes in accordance with the control program. The RAM 303 is used as a temporary storage area such as the main memory or work area of ​​the CPU 301. The HDD 304 stores various data, various programs, etc. The display unit 305 displays various information. The input unit 306 accepts various operations by the user. A keyboard, a mouse, a six-axis controller, etc. can be connected to the input unit 306. The communication unit 307 communicates with an external device via a network. An example of a network is a local area network (LAN). Alternatively, the communication unit 307 may communicate with an external device wirelessly. The hardware elements included in the operation terminal 150 are connected to each other via a bus 308.

[0033] In this way, a processor such as CPU 301 executes a program stored in a memory such as ROM 302, RAM 303, or HDD 304, thereby realizing the functions of each unit shown in FIG. 6 and the like, which will be described later.

[0034] Furthermore, the image generation device 120 and the display control device 130 can be realized by a computer including a processor and a memory. The hardware configuration of the image generation device 120 and the display control device 130 can be similar to the hardware configuration of the operation terminal 150. That is, a processor such as a CPU 301 executes a program stored in a memory such as a ROM 302, a RAM 303, or an HDD 304, thereby realizing the functions of each unit shown in FIGS. 4 and 6, which will be described later.

[0035] Fig. 4 is a diagram showing an example of the functional configuration of an image processing system 100 used in the three-dimensional model generation process and the virtual viewpoint image generation process according to this embodiment. Fig. 6 is a diagram showing an example of the functional configuration of an image processing system 100 used in the virtual viewpoint image correction process according to this embodiment. As described above, the functions of each device shown in Figs. 4 and 6 can be realized by a computer, but some or all of the functions may also be realized by dedicated hardware.

[0036] As shown in FIG. 2, the image processing system according to this embodiment is configured by a plurality of devices connected via a network. However, the functions of one device may be realized by another device. For example, the image generation device 120 may have an image synthesis unit. The image processing system according to this embodiment may also be realized by a single information processing device. For example, the operation terminal 150 may have the functions of the image generation device 120 and the display control device 130.

[0037] The three-dimensional model generation process is performed by the imaging device 110, the object image generation unit 121, the object image recording unit 122, the model generation unit 123, and the model recording unit .

[0038] In the three-dimensional model generation process, the imaging device 110 captures images at a frequency of 60 times per second. The imaging device 110 then transmits image data to the object image generation unit 121. The image data may include metadata in addition to the image. The metadata may include the camera parameters and camera ID of the imaging device 110, as well as the image capture time. The camera parameters may represent internal and external parameters of the imaging device 110. The internal parameters represent the focal length of the lens, etc. The external parameters represent the three-dimensional position, pan, tilt, and roll directions of the imaging device 110, etc.

[0039] The camera parameters of the imaging device 110 are recorded in the imaging device 110. The camera parameters of the imaging device 110 can be calculated in advance using a known camera calibration technique. For example, multiple markers placed in a stadium can be captured using the imaging device 110. Markers detected from the captured image can be associated with markers placed in the stadium. The camera parameters can then be calculated by three-dimensional geometric calculations based on the association. The camera parameters may be dynamically corrected based on the image captured by the imaging device 110. Dynamic correction can be performed, for example, to correct errors caused by camera shake. Furthermore, the imaging device may move around the stadium, such as a tally camera. In this case, the camera parameters may be dynamically calculated using a technique such as SLAM.

[0040] The object image generation unit 121 separates the object (specific object) from the received image data. The separation method may be a background subtraction method or an inference method based on a learning model. The object image generation unit 121 generates image data showing an object image, which is an image of the separated object. The object image generation unit 121 then transmits the generated image data to the object image recording unit 122.

[0041] The object image is, for example, a color image with RGBA 4-channel alpha. In the image, the alpha value of Ach in areas where a specific object exists is set to 1, and the alpha value in other areas is set to 0. The values ​​of the 3 RGB channels that represent color are the values ​​in the received image data. For example, the RGB values ​​in areas where no specific object exists can all be set to 0. Metadata from the received image data can be added to the image data of the object image.

[0042] The object image recording unit 122 records the image data of the received object image in a database. In this embodiment, the database of the image generating device 120 can be configured using the HDD 304. Note that the configuration of the database is not particularly limited. For example, the database may be configured using a network storage connected to the image generating device 120 via a network.

[0043] The model generation unit 123 generates a first three-dimensional model representing a subject that is a specific object, based on a group of images of the subject from multiple directions. In this embodiment, the model generation unit 123 reads out object images from the object image recording unit 122. Then, the model generation unit 123 generates a three-dimensional model of the specific object based on the object images. As a method for generating the three-dimensional model, a three-dimensional shape reconstruction method such as the volume intersection method or Multi-View-Stereo (MVS) can be used, as described above.

[0044] Furthermore, the model generation unit 123 can generate data indicating the generated three-dimensional model. Then, the model generation unit 123 transmits the data of the three-dimensional model to the model recording unit 124. Note that the method of converting the three-dimensional model into data is not particularly limited. In this embodiment, the three-dimensional model is represented by voxels. Alternatively, the three-dimensional model may be mesh data. Furthermore, the three-dimensional model may be converted into data using a method such as a neural network or Gaussian splatting.

[0045] The model recording unit 124 records the three-dimensional model data received from the model generating unit 123 in a database.

[0046] The virtual viewpoint image generation process is performed by the viewpoint controller 131, the time controller 132, the conversion unit 133, the viewpoint receiving unit 125, the viewpoint recording unit 126, the image generating unit 127, the image output unit 128, and the display device 140.

[0047] In this embodiment, the viewpoint controller 131 is composed of a six-axis controller having two joysticks 1 and 2, and a momentary rocker switch 3, as shown in Fig. 1. The viewpoint controller 131 periodically outputs movement amount information of the joysticks 1 and 2 and the momentary rocker switch 3 to the conversion unit 133. For example, the viewpoint controller 131 can output the movement amount information at a cycle of 60 times per second.

[0048] 1, the time controller 132 is an operation device having a turntable 4. The time controller 132 outputs, to the conversion unit 133, information on the amount of movement of the turntable 4 according to the user's rotation operation.

[0049] The conversion unit 133 accumulates movement amount information input by user operations for each of the connected viewpoint controllers 131 and time controllers 132. The conversion unit 133 also converts the accumulated movement amount information into virtual viewpoint information. The conversion unit 133 then outputs the virtual viewpoint information to the viewpoint receiving unit 125. In this embodiment, the origin of the three-dimensional virtual space is the center of the imaging range surrounded by the multiple imaging devices 110. The origin can be set in advance, for example, to the center of a stadium. The position and direction of the virtual viewpoint are defined by this coordinate system.

[0050] The conversion unit 133 reflects the movement amount information of joystick 1 in the translational movement of the virtual viewpoint. The conversion unit 133 also reflects the movement amount information of joystick 2 in the rotational movement of the virtual viewpoint. At this time, the conversion unit 133 can update the virtual viewpoint information so that the three-dimensional orientation of the virtual viewpoint changes. The conversion unit 133 also reflects the movement amount information of momentary rocker switch 3 in the angle of view of the virtual viewpoint. In other words, operating joystick 1 of the viewpoint controller 131 causes the virtual viewpoint to move translationally. Operating joystick 2 changes the orientation of the virtual viewpoint. Operating momentary rocker switch 3 also changes the angle of view of the virtual viewpoint.

[0051] Furthermore, the conversion unit 133 reflects the movement amount information of the time controller 132 in the time indicated by the virtual viewpoint information. By operating the time controller 132, the user can move time forward from the shooting start time of each image capture device 110. Furthermore, by operating the time controller 132, the user can move time back.

[0052] The viewpoint receiving unit 125 outputs the virtual viewpoint information received from the display control device 130 to the viewpoint recording unit 126 and the image generating unit 127.

[0053] The viewpoint recording unit 126 records the input virtual viewpoint information in a database.

[0054] The image generation unit 127 generates a virtual viewpoint image of a three-dimensional model from a virtual viewpoint. The image generation unit 127 can generate the virtual viewpoint image based on a three-dimensional model representing a subject and a group of images of the subject from multiple directions. In the virtual viewpoint image generation process, the image generation unit 127 generates a first virtual viewpoint image of a first three-dimensional model representing the subject. The image generation unit 127 also transmits the generated virtual viewpoint image to the image output unit 128.

[0055] In this embodiment, the image generation unit 127 acquires, from a database, an object image corresponding to the time indicated by the input virtual viewpoint information. The image generation unit 127 also acquires, from the database, a three-dimensional model corresponding to the time indicated by the input virtual viewpoint information. The image generation unit 127 can generate a virtual viewpoint image from the three-dimensional model and the object image using model-based rendering, as described above. Note that if any data corresponding to the time indicated by the input virtual viewpoint information does not exist in the database, the virtual viewpoint image generation process can be terminated without generating the virtual viewpoint image.

[0056] The image output unit 128 records the virtual viewpoint image transmitted from the image generation unit 127 in a database. The image output unit 128 can record the virtual viewpoint image in the database in association with virtual viewpoint information. The image output unit 128 can also transmit the virtual viewpoint image transmitted from the image generation unit 127 to the display device 140 or an external device. The image output unit 128 may output the virtual viewpoint image recorded in the database, which corresponds to a time specified by the user via the time controller 132 or the like, to the display device 140 or an external device.

[0057] The display device 140 displays the virtual viewpoint image received from the image output unit 128 .

[0058] FIG. 5(A) shows an example of a virtual viewpoint image displayed on the display device 140 after the above three-dimensional model generation process and virtual viewpoint image generation process. In this example, a baseball player in a stadium is the subject (specific object). As shown by the dashed line for illustrative purposes, the accuracy of the image around the player's arm is low. This low accuracy results from excess shape of the three-dimensional model that was not removed by the visual volume intersection method or MVS. Therefore, in this embodiment, the virtual viewpoint image is corrected in the following virtual viewpoint image correction process.

[0059] An example of the functional configuration of the image processing system 100 used in the virtual viewpoint image correction process according to this embodiment will be described with reference to Fig. 6. The virtual viewpoint image correction process is performed by a time determination unit 151, an object image receiving unit 152, a model receiving unit 153, a display control unit 154, a part determination unit 155, and an image synthesis unit 156. In the virtual viewpoint image correction process, the object image recording unit 122, the model recording unit 124, the model generation unit 123, the viewpoint recording unit 126, and the image generation unit 127 also operate.

[0060] The time determination unit 151 determines the time to be corrected. The time determination unit 151 receives input from the mouse 5 operated by the user while the operation application is being executed. The time determination unit 151 may also receive input by user operation via another input device, such as a keyboard connected to the input unit 306. The time determination unit 151 then determines the time to be corrected based on the received input. The time determination unit 151 also transmits a data download request related to the determined time to the object image recording unit 122 and the model recording unit 124.

[0061] The object image recording unit 122 searches the database for data of the object image corresponding to the time included in the request, and then transmits the data of the found object image to the object image receiving unit 152.

[0062] The model recording unit 124 searches the database for data of the three-dimensional model corresponding to the time included in the request. Then, the object image recording unit 122 transmits data of the found three-dimensional model to the model receiving unit 153.

[0063] The object image receiving unit 152 transmits the received object image to the display control unit 154 and the part determining unit 155 .

[0064] The model receiving unit 153 transmits the received three-dimensional model to the display control unit 154 and the part determining unit 155 .

[0065] The display control unit 154 causes the display unit 305 of the operation terminal 150 to display a user interface. Here, the display control unit 154 can cause the display unit 305 to display a user interface including an image of the subject. The display control unit 154 can also cause the display unit 305 to display a user interface including a three-dimensional model of the subject. For example, the display control unit 154 causes the display unit 305 of the operation terminal 150 to display the received object image and three-dimensional model. The display control unit 154 can cause the display unit 305 to display a UI of the operation application showing the object image and three-dimensional model.

[0066] The part determination unit 155 determines a specific part of the subject to be corrected. The part determination unit 155 can acquire a user input indicating the specific part of the subject. The part determination unit 155 can receive an input by a user operation via an input device such as a mouse 5. In this embodiment, the specific part of the subject is indicated by specifying an area on the image of the subject. Such area specification can be performed on the object image displayed on the display unit 305 or on the captured image obtained by the imaging device 110. That is, in this embodiment, the user input can indicate an area in the image of the subject that corresponds to the specific part of the subject. Furthermore, the user input can be an input indicating an area on the image of the subject in a user interface displayed by the display control unit 154.

[0067] Furthermore, the part determination unit 155 can extract at least two images to be used for correcting the virtual viewpoint image from the group of images of the subject. For example, the part determination unit 155 can select at least two images from the group of images of the subject. Then, the part determination unit 155 can extract a partial image from each of the selected images of the subject.

[0068] In this embodiment, the part determination unit 155 selects two or more of the multiple object images corresponding to the time to be corrected. The part determination unit 155 can select an image from the group of subject images that shows a specific part of the subject. In this embodiment, the part determination unit 155 selects at least two images from the group of subject images in accordance with a user specification. The user can select an image from the group of subject images or the group of object images that shows a specific part of the subject. In this way, by using an image selected so that the specific part of the subject is shown in the image, as described below, a three-dimensional model can be generated, thereby improving the accuracy of the three-dimensional model of the specific part. Furthermore, the part determination unit 155 extracts a partial area in the area corresponding to the specific part of the subject from each of the selected object images in accordance with the user input as described above.

[0069] For example, the part determination unit 155 can receive an input specifying a specific part in each of a plurality of object images, specified by a user operation. Then, the part determination unit 155 can send a request to update the display of the display unit 305 to the display control unit 154. The part determination unit 155 can also generate an object image of a specific part, which is a partial image showing the specific part. Furthermore, the part determination unit 155 sends a request to the model generation unit 123 to generate a three-dimensional model of the specific part and generate a virtual viewpoint image of the specific part. In this way, by the user specifying the shape of the specific part of the subject in the image, it is possible to improve the accuracy of the three-dimensional model of the specific part generated based on the partial image.

[0070] In response to the request, the model generation unit 123 generates a second three-dimensional model representing a specific region of the subject. The model generation unit 123 can generate a three-dimensional model of the specific region based on at least two images extracted from a group of images of the subject. For example, as described above, the model generation unit 123 can generate a three-dimensional model of the specific region based on at least two images selected from a group of images of the subject. The model generation unit 123 can also generate a three-dimensional model of the specific region based on at least two images selected from a group of images of the subject in accordance with a user specification. The model generation unit 123 can also generate a three-dimensional model of the specific region based on at least two images in which the specific region of the subject is captured.

[0071] Furthermore, the model generation unit 123 can generate a three-dimensional model of a specific part of a subject based on a partial image of the specific part of the subject extracted from a group of images of the subject. For example, the model generation unit 123 can generate a three-dimensional model of the specific part based on a partial image of an area corresponding to the specific part of the subject indicated by a user input. In this embodiment, the model generation unit 123 generates a three-dimensional model of the specific part based on the partial image indicating the specific part transmitted from the part determination unit 155.

[0072] In this embodiment, the model generation unit 123 generates three-dimensional model data of a region around an arm, for example, using the volume intersection method as described above, based on multiple object images showing the region around the arm included in the received request. The model generation unit 123 can add metadata including time information indicating the time of the target correction to the three-dimensional model data. The model generation unit 123 then transmits the generated three-dimensional model data to the model recording unit 124.

[0073] The model recording unit 124 records the received three-dimensional model data of the specific part in a database. The model recording unit 124 also transmits the three-dimensional model data of the specific part to the image generating unit 127.

[0074] The image generation unit 127 generates a second virtual viewpoint image of the three-dimensional model of the specific part from a virtual viewpoint. The image generation unit 127 can acquire virtual viewpoint information corresponding to the time to be corrected from the viewpoint recording unit 126 based on time information included in the metadata of the received three-dimensional model data of the specific part. The image generation unit 127 can also acquire an object image corresponding to the time to be corrected from the object image recording unit 122. The image generation unit 127 then generates a virtual viewpoint image of the specific part based on the three-dimensional model, virtual viewpoint information, and object image. The image generation unit 127 transmits the generated virtual viewpoint image of the specific part to the image synthesis unit 156.

[0075] The image synthesis unit 156 synthesizes multiple virtual viewpoint images, each representing at least a portion of the subject, including a second virtual viewpoint image of a three-dimensional model of a specific part of the subject. In this embodiment, each of the synthesized virtual viewpoint images is a virtual viewpoint image from the same virtual viewpoint. In this manner, the image synthesis unit 156 generates a virtual viewpoint image that shows the result of correcting the first virtual viewpoint image of the three-dimensional model of the subject. For example, the image synthesis unit 156 can synthesize multiple virtual viewpoint images of a three-dimensional model, each representing a different specific part of the subject, from the same virtual viewpoint. Furthermore, the image synthesis unit 156 can synthesize the first virtual viewpoint image of the three-dimensional model of the subject and the second virtual viewpoint image of the three-dimensional model of the specific part of the subject. In this case, the image synthesis unit 156 can generate the virtual viewpoint image by superimposing the second virtual viewpoint image on the first virtual viewpoint image.

[0076] The image synthesis unit 156 can transmit the virtual viewpoint image obtained by synthesis to the display control unit 154 for display on the display unit 305 of the operation terminal 150. The image synthesis unit 156 may also transmit the virtual viewpoint image obtained by synthesis to the display device 140 for display. In addition, the image synthesis unit 156 may also transmit the virtual viewpoint image obtained by synthesis to the image output unit 128 for recording in the database of the image generation device 120.

[0077] The operation terminal 150 may specify each of multiple specific parts of the subject. In this case, the image generating device 120 can generate a three-dimensional model of each of the multiple specific parts. Furthermore, the image generating device 120 can generate a virtual viewpoint image of the three-dimensional model of each of the multiple specific parts. The image synthesis unit 156 can receive the virtual viewpoint image generated for each specific part in this manner. For example, the image synthesis unit 156 can perform a receiving operation to receive virtual viewpoint images multiple times. In this case, the image synthesis unit 156 can synthesize multiple virtual viewpoint images of the three-dimensional model, each representing a different specific part of the subject, from a virtual viewpoint. For example, the image synthesis unit 156 can overlay multiple virtual viewpoint images, each corresponding to a specific part. In this case, multiple specific parts may be selected so that the combination of the multiple specific parts represents the entire subject. On the other hand, the image synthesis unit 156 may synthesize multiple second virtual viewpoint images, each corresponding to a specific part, with the first virtual viewpoint image of the three-dimensional model of the subject. Furthermore, the image synthesis unit 156 may perform a synthesis process for the virtual viewpoint images based on a user operation.

[0078] Next, a description will be given of the processing in each process performed by the image processing system 100. Fig. 7 is a flowchart showing the operation of the image processing system 100 in each process according to this embodiment. The following processing can be performed by, for example, the CPU 301 reading and executing a program stored in the ROM 302 or the HDD 304.

[0079] First, the three-dimensional model generation process will be described with reference to FIG.

[0080] In S701, the multiple imaging devices 110 capture images. For example, each of the imaging devices 110 can capture images at a frequency of 60 times per second. Each of the multiple imaging devices 110 transmits the captured image to the object image generation unit 121.

[0081] In S702, the object image generating unit 121 separates an image of a specific object from the received captured image, and the object image recording unit 122 records the separated object image in a database.

[0082] In S703, the model generation unit 123 acquires an object image from the object image recording unit 122. Then, the model generation unit 123 generates a three-dimensional model of a specific object based on the object image. Furthermore, the model recording unit 124 records the generated three-dimensional model in a database.

[0083] Next, the virtual viewpoint image generation process will be described with reference to FIG.

[0084] In S711, the viewpoint receiving unit 125 receives virtual viewpoint information according to a user operation from the display control device 130. Then, the viewpoint receiving unit 125 outputs the virtual viewpoint information to the viewpoint recording unit 126 and the image generating unit 127.

[0085] In S712, the image generation unit 127 acquires an object image and a three-dimensional model from the database based on the time indicated by the input virtual viewpoint information.

[0086] In S713, the image generation unit 127 determines whether or not the object image and the three-dimensional model have been successfully acquired. For example, if the database does not contain data corresponding to the time indicated by the virtual viewpoint information, the image generation unit 127 cannot acquire the object image or the three-dimensional model. If the image generation unit 127 determines that it has failed to acquire this data, the virtual viewpoint image generation process ends without generating a virtual viewpoint image. If not, the process proceeds to S714.

[0087] In S714, the image generation unit 127 generates a virtual viewpoint image using the acquired object image and three-dimensional model data.

[0088] In S715, the image output unit 128 displays the virtual viewpoint image generated by the image generation unit 127 on the display device 140. In addition, the image output unit 128 records the virtual viewpoint image generated by the image generation unit 127 in a database.

[0089] In S716, the viewpoint recording unit 126 records the virtual viewpoint information received in S711 in the database.

[0090] Finally, the virtual viewpoint image correction process will be described with reference to FIG.

[0091] In S721, the user operates the UI of the operation application displayed on the display unit 305 of the operation terminal 150 to determine the time to be corrected.

[0092] In S722, the object image recording unit 122 and the model recording unit 124 search the database for data of the object image and the three-dimensional model corresponding to the time determined in S711. The object image recording unit 122 and the model recording unit 124 transmit the found data to the object image receiving unit 152 and the model receiving unit 153.

[0093] In S723, the object image receiving unit 152 and the model receiving unit 153 determine whether or not they have successfully acquired the object image and the three-dimensional model. For example, if the database does not contain data corresponding to the time determined in S721, the object image recording unit 122 or the model recording unit 124 cannot acquire the object image or the three-dimensional model. If the object image receiving unit 152 and the model receiving unit 153 determine that they have failed to acquire this data, the process proceeds to S731. In this case, in S731, the display control unit 154 displays an error message in the operation application. The virtual viewpoint image correction process then ends. If the object image receiving unit 152 and the model receiving unit 153 determine that they have successfully acquired this data, the process proceeds to S724.

[0094] In S724, the display control unit 154 displays the received object image and three-dimensional model on the UI of the operation application.

[0095] Fig. 8 shows an example of the UI of the operation application. The UI shown in Fig. 8 is displayed on the display unit 305 of the operation terminal 150. The UI is made up of several windows.

[0096] The window shown in FIG. 8(A) is a window for determining the time to be corrected. This window has a seek bar 802 at the top for determining the time to be corrected. This window also has a three-dimensional model display portion 801 at the bottom. The three-dimensional model 803 acquired in S722 is displayed in the three-dimensional model display portion 801. In this example, based on the time indicated by the seek bar 802, the three-dimensional model 803 corresponding to this time is displayed. When the user performs a movement operation on the seek bar 802 via an input device such as the mouse 5 or a keyboard, the time to be corrected is determined. Then, the time determination unit 151 transmits a data download request for the determined time to the object image recording unit 122 and the model recording unit 124.

[0097] The window shown in FIG. 8(B) is a window for displaying multiple object images corresponding to different imaging devices. An object image 804 is displayed in this window. If there are a large number of object images, it may not be possible to fit all of the object images in the window. For this reason, an object image seek bar 805 can be placed in the window as shown in FIG. 8(B). The user can slide the object image in the window by operating the object image seek bar 805. Note that the method of displaying the object image shown in FIG. 8(B) is merely an example. For example, the angle of view or magnification of the image may be changeable. Furthermore, when multiple three-dimensional models exist, the UI may provide a function for cropping an image area corresponding to a specific three-dimensional model.

[0098] In S725, the user selects an object image on the UI displayed on the display unit 305 in which the specific part is not occluded.

[0099] In S726, Yusa specifies, on the UI displayed on the display unit 305, an area on the selected object image that corresponds to a specific part.

[0100] In S725 and S726, the user can input to designate a specific portion on the object image while referring to the shape of the three-dimensional model 803 displayed in FIG. 8(A) and the virtual viewpoint image generated in S714. The user can designate a specific portion of the subject that is desired to be modified in the virtual viewpoint image or the three-dimensional model. In the example shown in FIG. 8, an unnecessary shape around the subject's left arm is modified. For this purpose, the virtual viewpoint image modification process is repeatedly executed for each of three portions of the three-dimensional model corresponding to the left arm, the head and torso, and other portions.

[0101] The virtual viewpoint image correction process for the left arm region will be described below. In S725, to specify the left arm region, the user first selects an object image on the UI shown in FIG. 8(B) in which the left arm is not occluded. In the example of FIG. 8(B), three images are selected. Then, the user presses button 806. Then, the window shown in FIG. 8(C) is displayed on the display unit 305.

[0102] Fig. 8(C) is a UI for specifying an area corresponding to a specific part on an object image. Fig. 8(C) displays the object image selected in Fig. 8(B). In S726, the user specifies the area of ​​the left arm part on the object image. The user inputs the area 809 of the left arm part for each of the three object images selected in S725.

[0103] The user can switch the editing mode by pressing the region addition button 807 and the region subtraction button 808. Pressing the region addition button 807 switches the input mode to the region addition mode. In the region addition mode, a region can be added according to user input. For example, the user uses an input device such as a mouse 5 to input a line indicating the outer edge of the region of the left arm. When a line indicating a closed region is input, the closed region is added to the specified region. As shown in FIG. 8(C), the texture of the specified region 809 can be changed to clarify the input specified region. As another example, a region of a predetermined range around the input point or line (for example, a region within a radius of 10 pixels) may be added to the specified region.

[0104] Pressing the part subtraction button 808 switches the input mode to part subtraction mode. Contrary to part addition mode, part subtraction mode allows regions to be excluded according to user input. For example, when a line indicating a closed region is input, the closed region is excluded from the specified region. At this time, the texture of the closed region can be restored to the texture of the original object image. Also, a predetermined range of region around the input point or line (for example, a region within a radius of 10 pixels) may be excluded from the specified region.

[0105] The input method of the specified area may be switchable or changeable on a setting screen (not shown). For example, the method of specifying the area or the size of the predetermined range may be changeable on the setting screen.

[0106] The part determination unit 155 receives user input via a mouse, etc. Then, the part determination unit 155 can update the texture display as needed to show the specified area, as shown in Fig. 8(C).

[0107] 8(C) may have an object image seek bar 810. Even if there are a large number of object images and it is not possible to arrange all of the object images within the window, the object images can be slid within the window by operating the object image seek bar 810.

[0108] When the user has finished inputting, the user presses button 811. Then, the part determination unit 155 extracts an object image of the left arm part from the area specified by the user in each of the three object images selected in S725. The part determination unit 155 also transmits to the model generation unit 123 a request to generate a three-dimensional model and a virtual viewpoint image of the left arm part, along with the three object images of the left arm part.

[0109] In S727, the model generation unit 123 generates a three-dimensional model of the specific body part based on the received two or more object images of the specific body part. In this example, the model generation unit 123 generates a three-dimensional model of the left arm part based on three object images of the left arm part. The model generation unit 123 records data of the generated three-dimensional model of the specific body part in a database. In addition, the model generation unit 123 transmits data of the three-dimensional model of the specific body part to the display control unit 154 via the model receiving unit 153. As shown in FIG. 8(E), the three-dimensional model of the specific body part is displayed on the display unit 305.

[0110] In S728, the image generation unit 127 generates a virtual viewpoint image of the specific part based on the three-dimensional model of the specific part generated in S727. In this example, the image generation unit 127 acquires three-dimensional model data of the left arm part from the database. Furthermore, the image generation unit 127 acquires virtual viewpoint information corresponding to the time to be corrected from the database. Then, the image generation unit 127 generates a virtual viewpoint image of the left arm part according to the three-dimensional model and the virtual viewpoint information.

[0111] The image generation unit 127 transmits the virtual viewpoint image of the specific part to the image synthesis unit 156. In this example, the image generation unit 127 transmits the generated virtual viewpoint image of the left arm part to the image synthesis unit 156. The virtual viewpoint image of the specific part is also transmitted to the display control unit 154 via the image synthesis unit 156. The display control unit 154 can display the virtual viewpoint image of the specific part on the display unit 305, as shown in FIG. 8(G).

[0112] Here, the virtual viewpoint image of the specific part generated by the image generation unit 127 can be an RGBAZ 5ch image. That is, the virtual viewpoint image can have an alpha ch (A) and a depth ch (Z) used for image synthesis in addition to the RGB color values. In the virtual viewpoint image, the alpha value of Ach in the area where the specific part exists is set to 1, and the alpha value in the other area is set to 0. The depth value is a real value that represents the distance from the virtual viewpoint to the surface of the three-dimensional model of the specific part. The image generation unit 127 can calculate the depth value by geometric calculation based on the shape of the three-dimensional model and the position, direction, and angle of view of the virtual viewpoint.

[0113] In S729, the image synthesis unit 156 synthesizes a plurality of virtual viewpoint images. The plurality of virtual viewpoint images includes the virtual viewpoint image transmitted by the image generation unit 127 in S728. The image synthesis unit 156 can synthesize the virtual viewpoint images taking into consideration the depth value and the alpha value. Note that, as an initialization process before synthesis, a virtual viewpoint image in which no subject exists and only the background is drawn may be generated. The image synthesis unit 156 may store such a virtual viewpoint image in advance. In such an image, the depth value of each pixel may be set to any value, such as the maximum value.

[0114] In S730, the image synthesis unit 156 determines whether the process of generating virtual viewpoint images for all specific portions to be modified has been completed. If the image synthesis unit 156 determines that the process of generating and synthesizing virtual viewpoint images for all specific portions has been completed, this process ends. If not, the process returns to S725. Then, in S725, the user specifies another specific portion. For example, the image synthesis unit 156 can acquire a user input indicating whether or not to continue specifying specific portions. If the image synthesis unit 156 determines that an input indicating the end of processing has been received, this process can end.

[0115] S725 to S730 are repeatedly executed for each body part. In this example, in S725 for the second time, the user can select three object images showing the head and torso parts, as in the first time, and press button 806. In subsequent S726, the user specifies an area 812 indicating the head and torso parts on the three object images, as shown in FIG. 8(D). When the user presses button 811, in S727, the display of the three-dimensional model of the head and torso parts is updated as shown in FIG. 8(F). Furthermore, in S728, a virtual viewpoint image of the head and torso parts is generated as shown in FIG. 8(H).

[0116] In S729, the image synthesis unit 156 synthesizes virtual viewpoint images of the head and torso regions with the virtual viewpoint image obtained by synthesizing the virtual viewpoint image in which the background is drawn and the virtual viewpoint image of the upper left portion in the first S729. Here, the image synthesis unit 156 can synthesize multiple virtual viewpoint images based on depth values ​​and alpha values ​​so that the virtual viewpoint image in the foreground is visible. For example, the image synthesis unit 156 can synthesize the virtual viewpoint image generated in S728 with the virtual viewpoint image to be synthesized. At this time, the image synthesis unit 156 can identify pixels in the virtual viewpoint image generated in S728 that have an alpha value of 1 and a depth value smaller than that of the virtual viewpoint image to be synthesized. Then, for the identified pixels, the image synthesis unit 156 can set the color value, depth value, and alpha value of the virtual viewpoint image to be synthesized to the color value, depth value, and alpha value of the virtual viewpoint image generated in S728. However, the method for synthesizing the virtual viewpoint images is not limited to this method. For example, the image synthesis unit 156 may synthesize multiple virtual viewpoint images according to weights set by any method.

[0117] Furthermore, in the third and subsequent processing iterations, the user can sequentially specify all parts of the subject. In this manner, the processing of S725 to S730 is repeatedly executed so that a virtual viewpoint image of the entire subject is generated.

[0118] As described above, by generating and synthesizing virtual viewpoint images for each body part, a corrected virtual viewpoint image can be obtained as shown in Fig. 5(B). In Fig. 5(B), the parts of the shape around the player's arms shown in Fig. 5(A), which are less accurate, have been corrected more accurately.

[0119] The display control unit 154 may cause the display unit 305 to display a UI of an operation application for referring to and selecting a virtual viewpoint image of a specific part before composition. In this case, the image composition unit 156 can change whether or not to compose a virtual viewpoint image of a specific part in accordance with a user instruction. The user may also modify the area designation of a specific part by referring to the virtual viewpoint image of the specific part. Furthermore, the display control unit 154 may cause the display unit 305 to display a UI of an operation application for referring to and selecting virtual viewpoint images before and after composition. The image composition unit 156 can change whether or not to save the virtual viewpoint image after composition in accordance with a user instruction.

[0120] In one embodiment, the display control device 130, the image generation device 120, and the operation terminal 150 have the same hardware configuration. Therefore, a single information processing device may perform the functions of each of these devices through multitasking. That is, in the above-described embodiment, after an object image is selected and a three-dimensional model is generated for one specific portion, an object image for the next specific portion is selected. However, the processing order is not limited to this order. For example, after an object image for one specific portion is selected, an object image for the next specific portion may be selected in the UI in parallel with the generation of a three-dimensional model for this specific portion in a separate process.

[0121] In addition, in the present embodiment, the display device 140 and the display unit 305 have been described as separate devices. However, the specific configuration of the image processing system 100 is not limited to this example. For example, the UI of the operation application may be displayed on the display device 140. Furthermore, the UI of the operation application may be configured as multiple windows.

[0122] (Object image selection assistance) In the above embodiment, the user selected an image showing a specific part of the subject from among multiple images of the subject. In such a configuration, the user may need prior knowledge of the processing overview or the criteria for selecting object images. Furthermore, when the number of imaging devices and the number of object images are large, the user's burden in selecting an appropriate image increases. In the following variant, the operation terminal 150 can select an image showing a specific part of the subject from among multiple images of the subject. For example, based on a user input indicating a specific position on a three-dimensional model, the operation terminal 150 can select an image showing the position indicated by the user input. In the following variant, the user specifies a point on a first three-dimensional model representing the subject, which is generated based on a group of images of the subject. The operation terminal 150 then semi-automatically selects an object image. This configuration reduces the user's operational burden and the burden of acquiring prior knowledge.

[0123] In this modified example, the operation terminal 150 further includes an occlusion calculation unit 157 and an object image selection unit 158 ​​.

[0124] The occlusion calculation unit 157 can determine whether or not each part of the subject is occluded in an image included in the image group of the subject. For example, the occlusion calculation unit 157 can determine whether or not the three-dimensional coordinates on the surface of the three-dimensional model are not occluded and can be observed on the object image.

[0125] The occlusion calculation unit 157 can make this determination based on a first 3D model representing the subject generated based on the group of images of the subject and position and orientation information of the image capture device that acquired each of the group of images of the subject. For example, the occlusion calculation unit 157 can make this determination based on camera parameters and a 3D model included in the metadata of the object image. This 3D model may be the 3D model generated in S703.

[0126] For example, the occlusion calculation unit 157 can generate information indicating whether or not each point on the surface of the three-dimensional model is occluded. For example, if a three-dimensional model exists between a specific point and the virtual viewpoint, the occlusion calculation unit 157 can determine that this point is occluded. The information indicating whether or not there is occlusion can have a value of 1 if the corresponding point is not occluded, and 0 if the corresponding point is occluded.

[0127] Furthermore, the occlusion calculation unit 157 can generate, for each part of the first three-dimensional model, information indicating the imaging device that captured the image in which this part appears. As described above, information indicating the presence or absence of occlusion can be described using bits of 0 / 1. Furthermore, the metadata of the three-dimensional model can include occlusion information for each voxel of the three-dimensional model. The occlusion information for one voxel can include information indicating the presence or absence of occlusion for each of the multiple imaging devices 110. In other words, the occlusion information for one voxel is data having a bit width corresponding to the number of imaging devices 110.

[0128] Then, the occlusion calculation unit 157 can add information indicating the presence or absence of occlusion, or the above-mentioned occlusion information, to the metadata of the three-dimensional model. In this embodiment, the amount of data of the occlusion information included in the metadata of the three-dimensional model is the number of image capture devices 110 x the number of voxels (bits). Note that if the data format of the three-dimensional model is mesh data, the occlusion calculation unit 157 can similarly generate information indicating the presence or absence of occlusion for each vertex of the mesh.

[0129] The occlusion calculation unit 157 transmits the occlusion information thus generated, the metadata of the three-dimensional model, or the data of the three-dimensional model to which the metadata has been added, to the display control unit 154.

[0130] The display control unit 154 can display, for each part of the first three-dimensional model, information about the imaging devices that captured images showing each part on the display unit 305. In this modification, the display control unit 154 can display, on the display unit 305, information indicating the number of imaging devices that captured images showing each part. For example, the display control unit 154 can analyze occlusion information included in the metadata of the three-dimensional model data. In this manner, the display control unit 154 can calculate the number of imaging devices that captured images showing each part, i.e., the number of observations. The display control unit 154 can then draw information indicating the number of observations on the three-dimensional model in a window displaying the three-dimensional model of the operation application. For example, the display control unit 154 can visualize the number of observations by drawing a heat map on the three-dimensional model that indicates the number of observations at each position. The user can refer to the number of observations and specify a point on the three-dimensional model, as described below. The display control unit 154 may also display, on the display unit 305, information identifying each of the imaging devices that captured images showing each part. Furthermore, it is not essential to display information about the imaging device that captured the image showing each part.

[0131] The object image selection unit 158 ​​can receive an input indicating a position on the first three-dimensional model from the user interface. For example, the object image selection unit 158 ​​can receive an operation in which the user designates a point on the three-dimensional model via an input device such as the mouse 5. The user can designate a point on a specific portion that the user wishes to modify.

[0132] Then, the object image selection unit 158 ​​selects an image from the group of images of the subject in which the point specified by the user appears. As described above, when the user specifies a point on a specific part, the object image selection unit 158 ​​can select an image in which the specific part of the subject appears. The object image selection unit 158 ​​can select an image in which the specific part of the subject appears, based on whether or not the specific part of the subject is occluded in an image included in the group of images of the subject. The object image selection unit 158 ​​can determine whether or not the specific part of the subject is occluded in the object image, based on the occlusion information described above.

[0133] For example, the object image selection unit 158 ​​can refer to the occlusion information and select an image in which the point specified by the user is observed without being occluded. The object image selection unit 158 ​​can refer to the occlusion information corresponding to the voxel described in the metadata based on the index of the voxel corresponding to the specified point. The object image selection unit 158 ​​can then determine the camera ID of the imaging device that captured the image in which the point specified by the user appears by referring to the bit data included in the occlusion information. Note that the bit position and the camera ID of each of the multiple imaging devices 110 are associated in advance. The object image selection unit 158 ​​then selects an object image corresponding to the determined camera ID. The corresponding object image, which has metadata including this camera ID, appears in the point specified by the user. The object image selection unit 158 ​​can transmit information identifying the selected object image to the display control unit 154.

[0134] 9 shows the operation of the image processing system 100 in the virtual viewpoint image correction process according to this modified example. Description of the same processes as those in FIG. 7 will be omitted.

[0135] In S901, the occlusion calculation unit 157 determines whether each point on the surface of the 3D model is observed in the image without being occluded. As described above, this determination can be made based on the camera parameters and the 3D model included in the metadata of the received object image. The occlusion calculation unit 157 describes the determination result in the metadata of the 3D model.

[0136] In S902, the display control unit 154 analyzes the metadata described in the three-dimensional model. Then, the display control unit 154 draws a heat map on the three-dimensional model as described above in a window displaying the three-dimensional model in the operation application. FIG. 10 shows an example of a window displaying a three-dimensional model. As shown in FIG. 10, the fewer the number of images observing each position in the three-dimensional model 803, the bluer (or darker) the color can be applied. Furthermore, the more the number of images observing each position in the three-dimensional model 803, the redder (or lighter) the color can be applied. Furthermore, as shown in FIG. 10, an indicator 1001 indicating the number of observations or a histogram 1002 of the number of observations may be displayed in the window.

[0137] Note that the presence or absence of a histogram or the like showing the occlusion determination result, the presence or absence of an indicator display, and the presence or absence of a histogram display may be switchable within the window. For example, the presence or absence of display of this information may be changeable on a setting screen. Also, a UI component such as a check box for switching the presence or absence of display may be provided for each item within the window.

[0138] In S903, the object image selection unit 158 ​​acquires information indicating the three-dimensional coordinates of a point on the three-dimensional model 803 based on the operation of the mouse 5 by the user.

[0139] In S904, the object image selection unit 158 ​​selects an object image that is not occluded from the acquired point based on the three-dimensional coordinates of the point. As described above, the object image selection unit 158 ​​can select an object image by referring to the metadata attached to the three-dimensional model 803. Then, the object image selection unit 158 ​​transmits the ID of the selected object image to the display control unit 154.

[0140] The display control unit 154 can update the display on the display unit 305 so that a check mark is placed on the object image selected by the object image selection unit 158. The user can confirm the selection result and correct the selection result as necessary. Thereafter, the user presses the button 806.

[0141] According to the above configuration, an object image is selected semi-automatically when the user designates a point on the three-dimensional model, thereby reducing the operational burden on the user.

[0142] Note that the specific configuration of the image processing system 100 is not limited to the above example. For example, when generating a three-dimensional model, the model generation unit 123 of the image generation device 120 may determine whether or not each point of the three-dimensional model is occluded. In this case, the model generation unit 123 may record metadata of the three-dimensional model including occlusion information.

[0143] (Assistance in selecting specific parts) In the above embodiment, the user specifies the area of ​​a specific part on the object image. In this case, the user specifies an area equal to the number of images multiplied by the number of specific parts. In this modified example, the input load on the user is further reduced by automatically identifying the part of the subject from the image. Below, a configuration different from the above modified example will be described.

[0144] In this modification, the part determination unit 155 extracts a partial image of a specific part of the subject from the image of the subject. To this end, the part determination unit 155 can identify areas on the object image that correspond to each part of the three-dimensional model based on at least one of the received object image and the three-dimensional model. Then, the part determination unit 155 can extract a partial image of the specific part from the area on the object image that corresponds to the part specified by the user.

[0145] The identification method is not particularly limited. For example, the part determination unit 155 can extract a partial image of a specific part of the subject from the image of the subject based on inference using a machine learning model. As a specific example, by using BodyPix (Google, 2019), it is possible to identify areas on the object image corresponding to each part of the three-dimensional model. Furthermore, the part determination unit 155 can identify each part of the three-dimensional model based on posture estimation of the subject based on the three-dimensional model. In this case, the part determination unit 155 can identify areas on the object image corresponding to each part of the three-dimensional model. The three-dimensional model generated in S703 can be used as the three-dimensional model.

[0146] At this time, the display control unit 154 can superimpose information indicating the classification result by the part determination unit 155 on the object image or three-dimensional model displayed on the display unit 305. Fig. 11 shows an example of displaying the classification result. In Fig. 11(A), the classification result is displayed on a three-dimensional model. In Fig. 11(B), the classification result is displayed on an object image.

[0147] A list of identified body parts is displayed in the windows shown in FIGS. 11(A) and (B). Also displayed in the windows are check boxes for selecting each identified body part. For example, if the user selects the check box 1101 corresponding to the left arm body part, the body part determination unit 155 cuts out a partial image from the region 809 corresponding to the left arm body part in each object image. As shown in FIG. 11(B), the user can further edit the automatically selected specific body part region. At this time, the user can switch editing modes by pressing the body part addition button 807 or the body part subtraction button 808.

[0148] The virtual viewpoint image correction process according to this modification can be performed according to the flowchart shown in Fig. 9. For example, when button 811 is pressed in S725, the part determination unit 155 generates an object image by logical AND processing of the designated area and the object image as shown in Fig. 11(B), and transmits the object image to the model generation unit 123.

[0149] In this way, by using a method for identifying each part of a subject, the input load on the user can be further reduced.

[0150] (Other Examples) The present disclosure can also be realized by a process in which a program that realizes one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device read and execute the program, or by a circuit (e.g., ASIC) that realizes one or more functions.

[0151] The disclosure of this specification includes the following image processing system, image processing method, and program. (Item 1) 1. An image processing system that generates a virtual viewpoint image of a subject from a specified virtual viewpoint based on a group of images of the subject taken from a plurality of directions, a model generation means for generating a first three-dimensional model representing the subject based on the image group, and for generating a second three-dimensional model representing a specific portion of the subject based on at least two images extracted from the image group; an image generation means for generating a first virtual viewpoint image of the first three-dimensional model from the virtual viewpoint, and for generating a second virtual viewpoint image of the second three-dimensional model from the virtual viewpoint; a synthesis means for synthesizing a plurality of virtual viewpoint images, each representing at least a part of the subject, including the second virtual viewpoint image, to generate a virtual viewpoint image that shows a result of correcting the first virtual viewpoint image; An image processing system comprising: (Item 2) 2. The image processing system according to claim 1, wherein the model generation means generates the second three-dimensional model based on at least two images selected from the group of images. (Item 3) 3. The image processing system according to claim 2, wherein the model generation means generates the second three-dimensional model based on at least two images showing a specific portion of the subject. (Item 4) 4. The image processing system according to item 3, further comprising a selection unit for selecting an image from the group of images that shows a specific part of the subject. (Item 5) 5. The image processing system according to item 4, wherein the selection means selects an image in which a specific part of the subject is captured based on whether the specific part of the subject is occluded in the images included in the image group. (Item 6) Item 6. The image processing system according to item 5, wherein the selection means determines whether a specific part of the subject is occluded in an image included in the group of images based on the first three-dimensional model and position and orientation information of the imaging device that acquired each of the group of images. (Item 7) 7. The image processing system according to any one of items 4 to 6, wherein the selection means generates, for each part of the first three-dimensional model, information indicating the imaging device that captured the image in which the part appears, based on the first three-dimensional model and position and orientation information of the imaging device that captured the image to obtain the image included in the image group. (Item 8) 8. The image processing system according to item 7, further comprising a display control means for displaying, on a display device, information about the imaging device that captured the image of each part of the first three-dimensional model. (Item 9) 9. The image processing system according to any one of items 4 to 8, further comprising an acquisition unit that acquires a user input indicating a specific part of the subject. (Item 10) a display control means for causing a display device to display a user interface on which the first three-dimensional model is displayed; 10. The image processing system according to item 9, wherein the user input is an input in the user interface indicating a position on the first three-dimensional model. (Item 11) 11. The image processing system according to any one of items 2 to 10, wherein the model generation means generates the second three-dimensional model based on at least two images selected from the group of images according to user input. (Item 12) 12. The image processing system according to any one of items 1 to 11, wherein the model generation means generates the second three-dimensional model based on a partial image of a specific part of the subject extracted from the image group. (Item 13) Item 13. The image processing system according to item 12, wherein the model generation means generates the second three-dimensional model based on a partial image of an area corresponding to a specific part of the subject indicated by a user input. (Item 14) a display control means for causing a display device to display a user interface including the image of the subject; Item 14. The image processing system according to item 13, wherein the user input is an input indicating an area on the image of the subject in the user interface. (Item 15) 15. The image processing system according to any one of items 12 to 14, further comprising an extraction unit that extracts a partial image of a specific part of the subject from the image of the subject. (Item 16) Item 16. The image processing system according to item 15, wherein the extraction means extracts a partial image of a specific part of the subject from the image of the subject based on inference using a machine learning model or based on posture estimation of the subject based on the first three-dimensional model. (Item 17) 17. The image processing system according to any one of items 1 to 16, wherein the synthesizing means superimposes the second virtual viewpoint image on the first virtual viewpoint image. (Item 18) 18. The image processing system according to any one of items 1 to 17, wherein the synthesis means synthesizes virtual viewpoint images of a plurality of three-dimensional models each representing a different specific part of the subject from the virtual viewpoint. (Item 19) 1. An image processing method performed by an image processing system that generates a virtual viewpoint image of a subject from a specified virtual viewpoint based on a group of images of the subject taken from a plurality of directions, comprising: generating a first three-dimensional model representing the subject based on the images; generating a first virtual viewpoint image of the first three-dimensional model from the virtual viewpoint; generating a second three-dimensional model representing a specific portion of the subject based on at least two images extracted from the set of images; generating a second virtual viewpoint image of the second three-dimensional model from the virtual viewpoint; generating a virtual viewpoint image that shows a result of correcting the first virtual viewpoint image by synthesizing a plurality of virtual viewpoint images, each of which represents at least a portion of the subject, including the second virtual viewpoint image; An image processing method comprising: (Item 20) 19. A program for causing a computer to function as the image processing system according to any one of items 1 to 18.

[0152] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0153] 100: Image processing system, 110: Imaging device, 120: Image generation device, 130: Display control device, 140: Display device, 150: Operation terminal, 121: Object image generation unit, 122: Object image recording unit, 123: Model generation unit, 124: Model recording unit, 125: Viewpoint receiving unit, 126: Viewpoint recording unit, 127: Image generation unit, 128: Image output unit, 151: Time determination unit, 152: Object image receiving unit, 153: Model receiving unit, 154: Display control unit, 155: Part determination unit, 156: Image synthesis unit

Claims

1. 1. An image processing system that generates a virtual viewpoint image of a subject from a specified virtual viewpoint based on a group of images of the subject taken from a plurality of directions, a model generation means for generating a first three-dimensional model representing the subject based on the image group, and for generating a second three-dimensional model representing a specific portion of the subject based on at least two images extracted from the image group; an image generation means for generating a first virtual viewpoint image of the first three-dimensional model from the virtual viewpoint, and for generating a second virtual viewpoint image of the second three-dimensional model from the virtual viewpoint; a synthesis means for synthesizing a plurality of virtual viewpoint images, each representing at least a part of the subject, including the second virtual viewpoint image, to generate a virtual viewpoint image that shows a result of correcting the first virtual viewpoint image; and An image processing system comprising:

2. 2. The image processing system according to claim 1, wherein said model generating means generates said second three-dimensional model based on at least two images selected from said image group.

3. 3. The image processing system according to claim 2, wherein said model generating means generates said second three-dimensional model based on at least two images showing a specific portion of said subject.

4. 4. The image processing system according to claim 3, further comprising a selection unit for selecting an image from the group of images in which a specific part of the subject is captured.

5. 5. The image processing system according to claim 4, wherein the selection means selects an image in which a specific part of the subject is captured based on whether the specific part of the subject is occluded in an image included in the image group.

6. 6. The image processing system according to claim 5, wherein the selection means determines whether a specific part of the subject is occluded in an image included in the group of images based on the first three-dimensional model and position and orientation information of an imaging device that acquired each of the group of images.

7. 5. The image processing system according to claim 4, wherein the selection means generates, for each part of the first three-dimensional model, information indicating the imaging device that captured the image in which the part appears, based on the first three-dimensional model and position and orientation information of the imaging device that captured the image to obtain the image included in the image group.

8. 8. The image processing system according to claim 7, further comprising a display control means for causing a display device to display, for each part of the first three-dimensional model, information about an imaging device that captured an image in which the part appears.

9. 5. The image processing system according to claim 4, further comprising an acquisition unit that acquires a user input indicating a specific part of the subject.

10. a display control means for causing a display device to display a user interface on which the first three-dimensional model is displayed; 10. The image processing system according to claim 9, wherein the user input is an input in the user interface indicating a position on the first three-dimensional model.

11. 3. The image processing system according to claim 2, wherein said model generating means generates said second three-dimensional model based on at least two images selected from said image group in accordance with a user input.

12. 2. The image processing system according to claim 1, wherein said model generating means generates said second three-dimensional model based on a partial image of a specific part of said subject extracted from said image group.

13. 13. The image processing system according to claim 12, wherein the model generating means generates the second three-dimensional model based on a partial image of a region corresponding to a specific part of the subject indicated by a user input.

14. a display control means for causing a display device to display a user interface including the image of the subject; 14. The image processing system according to claim 13, wherein the user input is an input indicating an area on the image of the subject in the user interface.

15. 13. The image processing system according to claim 12, further comprising an extracting means for extracting a partial image of a specific part of the subject from the image of the subject.

16. 16. The image processing system according to claim 15, wherein the extraction means extracts a partial image of a specific part of the subject from the image of the subject based on inference using a machine learning model or based on posture estimation of the subject based on the first three-dimensional model.

17. 2. The image processing system according to claim 1, wherein said synthesizing means superimposes said second virtual viewpoint image on said first virtual viewpoint image.

18. 2. The image processing system according to claim 1, wherein said synthesizing means synthesizes a plurality of virtual viewpoint images of a three-dimensional model, each representing a different specific portion of said subject, from said virtual viewpoint.

19. 1. An image processing method performed by an image processing system that generates a virtual viewpoint image of a subject from a specified virtual viewpoint based on a group of images of the subject taken from a plurality of directions, comprising: generating a first three-dimensional model representing the subject based on the images; generating a first virtual viewpoint image of the first three-dimensional model from the virtual viewpoint; generating a second three-dimensional model representing a specific portion of the subject based on at least two images extracted from the set of images; generating a second virtual viewpoint image of the second three-dimensional model from the virtual viewpoint; generating a virtual viewpoint image that shows a result of correcting the first virtual viewpoint image by synthesizing a plurality of virtual viewpoint images, each of which represents at least a portion of the subject, including the second virtual viewpoint image; An image processing method comprising:

20. A program for causing a computer to function as the image processing system according to any one of claims 1 to 18.

Images