Image processing system, information processing method, and program

Abstract

To control a virtual camera corresponding to a virtual viewpoint image generated based on multiple captured images captured by multiple imaging devices; set, for the virtual camera, first imaging parameters and second imaging parameters different from the first imaging parameters; generate a first virtual viewpoint image based on the first imaging parameters, and a second virtual viewpoint image based on the second imaging parameters; combine the first and second virtual viewpoint images to generate a composite image; acquire user operation content on the first or second imaging parameters; and display the generated composite image.SOLUTION: A virtual viewpoint image is generated by combining virtual viewpoint images created using multiple imaging parameters.SELECTED DRAWING: Figure 8

Description

[Technical Field]

[0001] The present invention relates to an image processing system, an information processing method, and a program. [Background technology]

[0002] In recent years, a technology that generates virtual viewpoint images from which viewpoints can be freely changed has been attracting attention. This technology generates virtual viewpoint images by continuously generating virtual viewpoint images while changing the position and posture of a virtual camera, which is the virtual viewpoint from which the generated model is viewed. This technology also makes it possible to generate virtual viewpoint images (free viewpoint images). This technology also makes it possible to express an object model generated in a certain frame as an afterimage by continuing to display it for a certain period of time.

[0003] This video representation using afterimages can be useful for motion analysis, for example, when used to display a series of movements of an athlete all at once. Furthermore, video representations that display afterimages of people as an effect are often used in commercials or music videos, and free viewpoint video can also generate such images in real time. Patent Document 1 describes a method for displaying virtual viewpoint images of a specific subject, which are continuously captured, corresponding to different points in time, allowing the specific subject to be compared from a desired viewpoint. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent Publication No. 2021-13095 Summary of the Invention [Problem to be solved by the invention]

[0005] The technique in Patent Document 1 enables the expression of afterimages in free viewpoint video by continuously displaying object models generated by capturing images of an object at different times for a certain period of time. In the above technique, an image viewed from a single virtual camera is generated and superimposed on each object model generated at different times. Therefore, the technique described in Patent Document 1 does not allow for video expression such as sliding afterimages or gradually enlarging them in real time.

[0006] An object of the present invention is to synthesize a virtual viewpoint image created using a plurality of imaging parameters. [Means for solving the problem]

[0007] In order to achieve the object of the present invention, for example, an image processing system according to one embodiment has the following configuration: That is, the image processing device controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, the image processing device including: first setting means for setting first imaging parameters of the virtual camera and second imaging parameters different from the first imaging parameters, generation means for generating the first virtual viewpoint image based on the first imaging parameters and the second virtual viewpoint image based on the second imaging parameters, and synthesis means for synthesizing the first virtual viewpoint image and the second virtual viewpoint image to generate a composite image, and an information processing device including: acquisition means for acquiring user operation details for the first imaging parameters or the second imaging parameters, and display means for displaying the generated composite image. [Effects of the Invention]

[0008] In the virtual viewpoint image, a virtual viewpoint image created using a plurality of imaging parameters is synthesized. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a block diagram showing an example of the configuration of an image processing system according to a first embodiment. [Figure 2] FIG. 1 is a block diagram showing an example of a hardware configuration of an information processing apparatus according to a first embodiment. [Figure 3] FIG. 1 is a diagram for explaining an information processing device according to a first embodiment. [Figure 4] FIG. 2 is a diagram for explaining a GUI of the information processing device according to the first embodiment. [Figure 5] 5A to 5C are diagrams for explaining a process of determining virtual imaging parameters according to the first embodiment. [Figure 6] FIG. 1 is a block diagram showing an example of the functional configuration of an image processing system according to a first embodiment. [Figure 7] 5A to 5C are diagrams for explaining a process for determining parameters for an afterimage according to the first embodiment. [Figure 8] FIG. 2 is a diagram for explaining processing between devices according to the first embodiment. [Figure 9] 4 is a flowchart showing an example of an information processing method according to the first embodiment. [Figure 10] FIG. 10 is a diagram for explaining a GUI of an information processing device according to a second embodiment. [Figure 11] FIG. 10 is a diagram for explaining a camera path file according to the second embodiment. [Figure 12] FIG. 10 is a block diagram showing an example of the functional configuration of a system according to a second embodiment. [Figure 13] 10 is a flowchart showing an example of an information processing method according to the second embodiment. [Figure 14] FIG. 10 is a diagram for explaining a GUI of an information processing apparatus according to another embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the scope of the invention claimed. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0011] [Embodiment 1] An image processing device included in the image processing system according to this embodiment sets first and second imaging parameters for a virtual camera corresponding to a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices. The image processing device then generates virtual viewpoint images based on these respective imaging parameters and synthesizes the virtual viewpoint images. Hereinafter, a virtual viewpoint image (especially a foreground object therein) generated based on the second imaging parameters and synthesized with a virtual viewpoint image generated based on the first imaging parameters will be referred to as an "afterimage." Furthermore, a "virtual camera corresponding to a virtual viewpoint image" refers to a virtual camera that performs processing to generate (capture) the virtual viewpoint image in a virtual space.

[0012] The afterimage according to this embodiment is a virtual viewpoint image generated based on a second imaging parameter and synthesized with a reference virtual viewpoint image, or a part of the virtual viewpoint image. Here, the afterimage may be a virtual viewpoint image of a previous frame relative to the reference virtual viewpoint image, or may be a virtual viewpoint image generated at a different timing in the same frame as the reference virtual viewpoint image. By synthesizing virtual viewpoint images generated in this way, it is possible to perform afterimage representation in which images generated at different timings are simultaneously displayed in the virtual viewpoint image.

[0013] For example, the image processing device according to this embodiment uses, as the first imaging parameters, imaging parameters of a virtual camera set in a first frame (reference frame) that serves as a reference, and uses, as the second imaging parameters, imaging parameters of a virtual camera set for a second frame (past frame) that precedes the reference frame. Hereinafter, the reference frame to which the afterimage is to be tracked may be referred to as the "current time," for example.

[0014] Hereinafter, a detailed description will be given of the image processing system 10 according to the embodiment 1. In this embodiment, an afterimage is rendered and displayed based on afterimage parameters that are different from real-time camera parameters and that are input by the user on the GUI.

[0015] 1 is a block diagram showing an example of the system configuration of an image processing system 10 that generates a virtual viewpoint image, including an image processing device 102 and an information processing device 103 according to this embodiment. The image processing system 10 includes an imaging system 101, an image processing device 102, and an information processing device 103.

[0016] The imaging system 101 is a system that uses multiple imaging devices (cameras) installed at different positions surrounding an imaging area to capture images with time synchronization between the cameras. Hereinafter, imaging performed by the multiple cameras of the imaging system 101 with time synchronization between the cameras may be simply referred to as "synchronized imaging." The imaging system 101 transmits multiple images captured synchronously from multiple viewpoints to the image processing device 102. At this time, the transmitted multiple images are assumed to be transmitted via a communication medium such as a LAN cable. Note that, here, the image processing device 102 is assumed to be connected to the imaging system 101 and the information processing device 103 so as to be able to transmit and receive information, but the type of connection between the devices is not particularly limited. For example, each connection may be made via wired communication via the above-mentioned LAN cable or wireless communication, or the type of connection between the devices may be different. In the following description of this embodiment, the imaging area captured by the imaging system 101 is assumed to be a photography studio where photography is performed to create virtual viewpoint images, a stadium where a sports competition is held, or a stage where a performance is held.

[0017] The image processing device 102 generates a virtual viewpoint image using a technique such as Model-Based Rendering (MBR). The processing performed by the image processing device 102 will be specifically described below. First, the image processing device 102 generates three-dimensional shape data of a subject based on multiple images captured synchronously using multiple cameras. In this embodiment, the processing of "generating three-dimensional shape data of a subject" refers to processing of generating a 3D point cloud (a set of points having three-dimensional coordinates) that represents the three-dimensional shape of the subject.

[0018] The method for generating three-dimensional shape data based on multiple images is not particularly limited, and for example, a visual hull technique may be used. Here, a CG (Computer Graphics) model of a stadium or the like where the physical cameras of the imaging system 101 are installed is assumed as the background model, and three-dimensional shape data of the subject is used in such a background model. This background model is, for example, created in advance and stored in the image processing device 102 (for example, stored in the ROM 203 of FIG. 2 , which will be described later), but may also be generated or edited by user operation, for example. The image processing device 102 according to this embodiment renders and generates a virtual viewpoint image viewed from a virtual camera by placing the three-dimensional shape data of the subject and a specified background model in the same virtual space. In this embodiment, parameters used to generate the virtual viewpoint image, such as the position or orientation of the virtual camera, are set as virtual imaging parameters by the information processing device 103. Through the above processing, an image of three-dimensional shape data viewed from the position and line of sight of the virtual camera can be generated.

[0019] Fig. 2 is a block diagram showing an example of the hardware configuration of the image processing device 102. The image processing device 102 includes a CPU 201, a RAM 202, a ROM 203, a communication unit 204, and an input / output unit 205. The CPU 201 controls the entire information processing device using control programs or data stored in the RAM 202 or the ROM 203, and controls each function of the image processing device 102 shown in Fig. 4. That is, in this embodiment, each function of the image processing device 102 is realized by the CPU 201 executing each program.

[0020] The RAM 202 temporarily stores programs or data read from the ROM 203. The ROM 203 holds programs or data that do not require modification. The data that does not require modification may include the posture parameters of the imaging camera or a background model.

[0021] The communication unit 204 communicates with external devices. Here, it is assumed that the image processing device 102 is connected to the imaging system 101 and the information processing device 103 so as to be able to send and receive information, but the type of connection used for communication between the devices is not particularly limited. For example, each connection may be made by wired communication via the above-mentioned LAN cable or the like, or by wireless communication, and the type of connection between the devices may be different.

[0022] The input / output unit 205 acquires an input signal to the image processing device 102 and outputs an output signal to the outside. The input / output unit 205 according to this embodiment acquires, as input signals, a plurality of images from the imaging system 101, an operation amount of a controller from the information processing device 103, or a signal indicating an input to an input device, and can output a virtual viewpoint image generated as a processing result to the display unit of the information processing device 103.

[0023] The image processing device 102 may also have the same hardware configuration as the information processing device 103 shown in Fig. 2. The image processing device 102 according to this embodiment will be described below as including a CPU, RAM, ROM, a communication unit, and an input / output unit, which are not shown, just like the information processing device 103.

[0024] The image processing device 102 sets first and second virtual imaging parameters as imaging parameters (virtual imaging parameters) of the virtual camera. Here, the following description will be given assuming that a foreground object extracted from a virtual viewpoint image generated based on the second virtual imaging parameters is composited with a virtual viewpoint image generated based on the first virtual imaging parameters. Hereinafter, the first virtual imaging parameters may be referred to as "reference parameters," and the second virtual imaging parameters may be referred to as "parameters for afterimage." Furthermore, a virtual camera that generates a virtual viewpoint image using such parameters for afterimage may be referred to as a "virtual camera for afterimage."

[0025] The virtual imaging parameters according to this embodiment are parameters for capturing a virtual viewpoint image, and include position parameters indicating the position of the virtual camera and attitude parameters indicating the attitude of the virtual camera. The virtual imaging parameters also include parameters for rendering a foreground object in a virtual space. The image processing device 102 according to this embodiment may set the virtual imaging parameters based on, for example, a user input acquired by the information processing device 103. User input in the information processing device 103 will be described below with reference to FIG. 3.

[0026] 3 is a diagram illustrating an example of the information processing device 103 according to this embodiment. The information processing device 103 according to this embodiment is, for example, a controller that accepts operations by a user and transmits the operation contents as output to the image processing device 102. In the example of FIG. 3, the information processing device 103 is a controller equipped with joysticks 301a and 301b for controlling a virtual camera, and determines a camera path representing the viewpoint of the virtual camera based on the amount of operation by the user. The joysticks 301a and 301b each acquire input regarding parameters (x, y, z) that indicate the position of the virtual camera in three-dimensional coordinates and pan, tilt, and roll parameters (Pan, Tilt, Roll) that represent the attitude of the virtual camera.

[0027] Note that the inputs used to set the virtual imaging parameters are not limited to inputs via the joysticks 301a and 301b, and any input that can be used to set each virtual imaging parameter can be used. For example, the image processing device 102 may set the virtual imaging parameters based on values ​​input by a user. In the example of FIG. 3, the information processing device 103 has two or more display units, including a display unit 302 that acquires and displays a virtual viewpoint video generated by the image processing device 102, and a display unit 303 that displays a GUI for inputting information about the virtual imaging parameters. In the example of FIG. 3, a GUI for inputting information about each item of the virtual imaging parameters is displayed on the display unit 303, and the virtual imaging parameters are set based on user operations acquired via the GUI, and an afterimage is displayed on the display unit 302.

[0028] 3, as long as the information processing device 103 has a configuration capable of similarly acquiring the amount of user operation. For example, the information processing device 103 may be a PC having a keyboard and a mouse instead of the joysticks 301a and 301b, or may be a mobile terminal such as a smartphone. Even in such a case, the information processing device 103 can display the same information as that displayed on the display unit 302 or the display unit 303 on a display unit (e.g., an organic EL display) included in the information processing device 103 or connected to the information processing device 103.

[0029] In this embodiment, virtual imaging parameters of the virtual camera for a reference frame (reference frame) may be used as the reference parameters, and virtual imaging parameters of the virtual camera for a frame (past frame) preceding the reference frame may be used as the afterimage parameters. In this embodiment, the current time is used as the reference frame, and information indicating the past frame is input by the information processing device 103 together with information regarding the virtual imaging parameters. Fig. 4 is a diagram showing an example of a GUI 401 displayed on the display unit 303, which is a GUI for performing such inputs.

[0030] The button 402 is a button for switching the afterimage display ON / OFF. The input space 403 is a field for inputting the number of afterimages to be displayed per frame for each subject. Here, only integers are accepted as input. The information processing device 103 acquires foreground models corresponding to the numerical value input in the input space 403 and displays them as afterimages. For example, when an integer N is input in the input space 403, the information processing device 103 displays N pieces of foreground data from time codes before the current time as afterimages. Here, the N pieces of foreground data do not necessarily refer to foreground data at time codes N consecutive frames counting backward from the current time. The specific time code for which foreground data is acquired may be determined by numerical values ​​input in the input space 403 and an input space 404 (described later). In this way, the number of afterimages synthesized by the image processing device 102 is not limited to one, and multiple afterimages may be generated and synthesized using additional virtual imaging parameters different from the second virtual imaging parameters. The time code according to this embodiment is information indicating the image capturing timing (the generation timing of the virtual viewpoint image) that is recorded in association with the virtual viewpoint image when the virtual viewpoint image is generated by the virtual camera. Here, the time code is recorded so that the order in which each virtual viewpoint image is generated and the frame interval at which the virtual viewpoint images are generated can be known.

[0031] The input space 404 is a field for inputting the frame interval of the time code of the foreground data to be displayed as an afterimage. Here, only integers are accepted as input. The information processing device 103 retrieves a foreground model to be displayed as an afterimage by going back in time from the current time by the number of frames input in the input space 404. For example, if an integer I is input in the input space 404, the information processing device 103 displays the foreground I frames before the current time as the first afterimage, the foreground I frames before that as the second afterimage, the foreground I frames before that as the third afterimage, and so on. If N is input in the input space 403, the information processing device 103 displays the foreground up to N×I frames before as an afterimage. For example, if 3 is input in the input space 403 and 2 in the input space 404, the foreground 2 frames, 4 frames, and 6 frames before the current time are displayed as afterimages.

[0032] Furthermore, the image processing device 102 according to this embodiment can set the transparency of the afterimages to be combined. The input space 405 is a field for inputting the transparency of the afterimage. Here, the numerical values ​​input into the input space 405 are assumed to be integers and real numbers between 0.0 and 1.0. An example of the process for actually determining the transparency of the afterimages will be described below. When the real number a is input into the input space 405, the information processing device 103 sets the alpha value of the first afterimage to a multiplied by the alpha value of the foreground at the current time. Furthermore, the alpha value of the second afterimage is set to a multiplied by the alpha value of the first afterimage, the alpha value of the third afterimage is set to a multiplied by the alpha value of the second afterimage, the alpha value of the fourth afterimage, and so on. In this way, the image processing device 102 can set the display transparency of the afterimages to be superimposed (individually for each afterimage). This process makes it possible to display afterimages that gradually become more transparent from the foreground at the current time.

[0033] In this embodiment, the afterimage is displayed by tracking the foreground at the current time. Therefore, the virtual camera capturing the virtual viewpoint image for displaying the afterimage also needs to move to track the virtual camera at the current time. From this perspective, the image processing device 102 may calculate the virtual imaging parameters (afterimage parameters) for the past frame based on the virtual imaging parameters (reference parameters) for the reference frame. For example, the image processing device 102 may set the position parameters of the virtual camera in the past frame, i.e., the position coordinates of the virtual camera for displaying the afterimage, to values ​​obtained by adding a certain offset value to the coordinates of the virtual camera at the current time. Input spaces 406, 407, and 408 in FIG. 4 are fields for inputting the offset position of the virtual camera for afterimage relative to the position of the virtual camera generating the real-time video. Here, the numerical values ​​input into the input spaces 406, 407, and 408 are assumed to be integers or real numbers. Note that such offset values ​​may hereinafter be simply referred to as "virtual camera offsets."

[0034] In this example, the position coordinates of the virtual camera for rendering the afterimage are calculated by adding the values ​​entered in input spaces 406, 407, and 408 to the coordinates of the virtual camera at the current time. A specific method for determining the parameters for the afterimage based on the values ​​entered in input spaces 406, 407, and 408 will be described below.

[0035] Here, with reference to FIG. 5 , a method for determining virtual imaging parameters of a virtual camera that displays an afterimage in a past frame as parameters for afterimage based on numerical values ​​input into input spaces 406, 407, and 408 will be described. FIG. 5 shows an example in which a user inputs 1 into input space 403, I into input space 404, and (dx, dy, dz) into input spaces 406, 407, and 408. At a certain time t, a free viewpoint image (virtual viewpoint image) of object model 501 viewed from the position of virtual camera 502 is image 503. At this time, parameters obtained by adding three-dimensional coordinate offset values ​​(dx, dy, dz) to the virtual imaging parameters of virtual camera 502 are assigned to virtual camera 504 for afterimage. Next, image processing device 102 uses virtual camera 504 to render object model 505 at time t′, which is one frame before time t, and generates foreground image 506. Furthermore, the image processing device 102 alpha-blends the foreground image 506 as an afterimage into the image 503, and generates a free viewpoint image 507 as a composite image.

[0036] In this way, the image processing device 102 generates a composite image by combining a virtual viewpoint image generated based on the reference parameters with a virtual viewpoint image generated based on the afterimage parameters. Here, as described with reference to Fig. 5, a composite image is generated by combining a foreground object extracted from a virtual viewpoint image generated based on the afterimage parameters with a virtual viewpoint image generated based on the reference parameters. This foreground object compositing process will be described here as a process of superimposing and displaying the foreground object, but it is not particularly limited to this as long as it is an image compositing process.

[0037] The above is an explanation of an example of GUI 401, but the form of GUI 401 is not limited to the one explained here. For example, a file containing information to be input into input spaces 403 to 408 may be created in advance, and the parameters for afterimages may be set by reading the file.

[0038] 6 is a block diagram showing an example of the functional configuration of the image processing device 102 and the information processing device 103 included in the image processing system 10 according to the embodiment 1. The information processing device 103 according to the present embodiment accepts a user input, generates a reference parameter and an afterimage parameter for the current time based on the acquired user input, and transmits them to the image processing device 102.

[0039] First, we will explain each functional unit of the information processing device 103. An input acquisition unit 601 acquires information input by a user on the GUI 401. The acquired information is used in an afterimage switching unit 602, an afterimage number determination unit 603, an interval determination unit 604, a transparency determination unit 605, and a position determination unit 606, which will be described later.

[0040] The afterimage switching unit 602 switches the display (superimposed display) of the afterimage between ON and OFF. Here, the afterimage switching unit 602 acquires ON / OFF information of the button 402 from the input information acquired by the input acquisition unit 601, converts the acquired information into information on whether to display or hide the afterimage, and transmits it to the afterimage parameter determination unit 609.

[0041] The afterimage number determination unit 603 determines the number of afterimages to be combined in generating a composite image. The afterimage number determination unit 603 according to this embodiment extracts the number of afterimages input to the input space 403 from the input information acquired by the input acquisition unit 601, and transmits the number to the afterimage parameter determination unit 609.

[0042] The interval determination unit 604 obtains the frame interval of the afterimage to be synthesized. Here, the interval determination unit 604 extracts the frame interval of the afterimage input to the input space 404 from the input information obtained by the input obtaining unit 601, and transmits it to the afterimage parameter determination unit 609.

[0043] The transparency determination unit 605 sets the transparency when the afterimage is superimposed and displayed. Here, the transparency determination unit 605 extracts the transparency of the afterimage input to the input space 405 from the input information acquired by the input acquisition unit 601, and transmits it to the afterimage parameter determination unit 609.

[0044] The position determination unit 606 acquires information for setting the position of a virtual camera for displaying an afterimage. Here, the position determination unit 606 extracts the offset of the virtual camera position for afterimage input into the input spaces 406, 407, and 408 from the input information acquired by the position parameter input acquisition unit 601, and transmits it to the afterimage parameter determination unit 609.

[0045] The operation information acquisition unit 607 acquires information on user operations. Here, the operation information acquisition unit 607 acquires operation information such as the degree of tilt or direction of the joysticks 301a and 301b operated by the user, and transmits the information to the imaging parameter determination unit 608.

[0046] The imaging parameter determination unit 608 generates imaging parameters such as the position and orientation of the virtual camera based on the operation information acquired from the operation information acquisition unit 607. The imaging parameter determination unit 608 transmits the generated imaging parameters to the afterimage parameter determination unit 609 and the parameter transmission unit 610.

[0047] The afterimage parameter determination unit 609 sets afterimage parameters. Here, the afterimage parameter determination unit 609 determines the afterimage parameters according to information acquired from the afterimage number determination unit 603, the interval determination unit 604, the transparency determination unit 605, the position determination unit 606, and the afterimage parameter determination unit 609. For example, the afterimage parameter determination unit 609 acquires information on whether to display or hide afterimages from the afterimage switching unit 602, and when hiding afterimages, transmits blank information to the parameter transmission unit 610. Here, when displaying afterimages, the afterimage parameter determination unit 609 acquires the number of afterimages from the afterimage number determination unit 603, the frame interval from the interval determination unit 604, the transparency from the transparency determination unit 605, and the offset of the virtual camera position for afterimages from the position determination unit 606. In addition, the afterimage parameter determination unit 609 acquires information on virtual imaging parameters for the current time from the imaging parameter determination unit 608.

[0048] A method for determining parameters for afterimages according to this embodiment will be described with reference to Fig. 7. In Fig. 7, the alpha value of the foreground is set as a and the coordinates of the virtual camera are set as p = (xc, yc, zc) for the time code 701 of the current time. The afterimage parameter determination unit 609 determines the time code of the foreground to be displayed as an afterimage based on the number of afterimages obtained from the afterimage number determination unit 603 and the frame interval obtained from the interval determination unit 604. Here, if the number of afterimages is N and the frame interval is I, the afterimage parameter determination unit 609 obtains time codes for N frames every I frames, counting from the time code 701 of the current time.

[0049] The afterimage parameter determination unit 609 determines an alpha value for each acquired time code based on the transparency acquired from the transparency determination unit 605. Here, if the transparency is T, the alpha value for the nth time code excluding the time code at the current time is the value obtained by multiplying the alpha value for the (n-1)th time code by T. Therefore, the alpha value for the nth time code is the value obtained by multiplying the alpha value a of the foreground model at the current time by Tn.

[0050] Furthermore, the afterimage parameter determination unit 609 determines an afterimage parameter for each acquired time code based on the offset of the virtual camera position acquired from the position determination unit 606. Here, if the offset is d=(dx, dy, dz), the afterimage parameter for the nth time code excluding the current time is p+nd.

[0051] The afterimage parameter determination unit 609 generates rendering parameters 702 having a structure in which an alpha value for each time code for rendering an afterimage and an afterimage parameter are set, and transmits the rendering parameters 702 to the parameter transmission unit 610. Here, the rendering parameters 702 are registered by associating the time code, the alpha value, and the afterimage parameter.

[0052] The parameter transmission unit 610 acquires the reference parameters from the imaging parameter determination unit 608 and the afterimage parameters from the afterimage parameter determination unit 609. The parameter transmission unit 610 transmits the acquired virtual imaging parameters to a virtual viewpoint image generation unit 611 in the image processing device 102.

[0053] The image processing device 102 according to this embodiment includes a virtual viewpoint image generating device, a 3D model generating device, and a 3D model storage device. The processing by the virtual viewpoint image generating device will be described below. The virtual viewpoint image generating unit according to this embodiment includes a virtual viewpoint image generating unit 611 and a synthesis unit 612.

[0054] The 3D model storage device stores a virtual viewpoint image generated from a 3D model such as the subject model 501 using a virtual camera, or a foreground object (afterimage) extracted from the virtual viewpoint image, in association with a time code. The 3D model storage device may store a 3D model used in this embodiment, such as the subject model 501, or may store objects other than the virtual camera in the virtual space, such as a background model, that are arranged when generating the virtual viewpoint image.

[0055] The 3D model generation device generates each 3D model to be stored in the 3D model storage device (generates three-dimensional shape data) and stores it in the 3D model storage device as 3D model information. 3D models can be generated using any general 3D model creation method. The 3D model generation device can also generate a virtual viewpoint image based on the 3D model and afterimage parameters, and a foreground object extracted from the virtual viewpoint image, using processing similar to that of the virtual viewpoint image generation device described below. Each piece of information generated by the 3D model generation device is stored in the 3D model storage device.

[0056] The virtual viewpoint image generation unit 611 acquires the current time and afterimage parameters from the parameter transmission unit 610, and acquires 3D model information of a time code corresponding to the acquired afterimage parameters from a 3D model storage device. Here, the virtual viewpoint image generation unit 611 acquires 3D model information of multiple time codes to acquire a time code corresponding to each afterimage parameter. For each of the acquired 3D models, the unit 611 renders a foreground image seen by a virtual camera at the position and orientation indicated by the corresponding virtual imaging parameters. Here, the rendering of the multiple 3D models is performed in parallel by threads equal to the number of acquired 3D models (= the number of acquired virtual imaging parameters). Therefore, in this example, multiple virtual viewpoint images are generated simultaneously. The generated multiple virtual viewpoint images are transmitted to the synthesis unit 612.

[0057] The composition unit 612 composites a virtual viewpoint image generated based on the afterimage parameters with a virtual viewpoint image generated based on the reference parameters. Here, the composition unit 612 overlays the foregrounds of multiple virtual viewpoint images simultaneously generated by the virtual viewpoint image generation unit 611 through parallel processing to generate a composite image. This composition process may be, for example, a process of overlaying a foreground object of a virtual viewpoint image generated based on the afterimage parameters onto a virtual viewpoint image generated based on the reference parameters, or a process of overlaying the foreground objects of all virtual viewpoint images and arranging the overlaid foreground objects on a prepared background to generate a composite image, but is not limited to these.

[0058] Next, processing between the 3D model storage device, the parameter transmission unit 610, and the virtual viewpoint image generation unit 611 will be described with reference to Fig. 8. The virtual viewpoint image generation unit 611 acquires the time code of the foreground to be displayed as an afterimage from the parameter transmission unit 610. Here, time code 801 is the current time, and time codes 802 to 803 are time codes before time code 801. The virtual viewpoint image generation unit 611 acquires (N+1) time codes, which are the sum of time code 801 and the number N of afterimages acquired from the afterimage number determination unit 603, at frame intervals I acquired from the interval determination unit 604. Time code 802 is I frames before time code 801, and time code 803 is the time code I x N frames before time code 801.

[0059] Next, the virtual viewpoint image generation unit 611 acquires foreground 3D models 805, 806...807 corresponding to the acquired time codes 801, 802 to 803 from among the foreground 3D models in the 3D model storage device 804. At the same time, the virtual viewpoint image generation unit 611 acquires virtual imaging parameters 808, 809 to 810 corresponding to the time codes 801, 802 to 803. Furthermore, the virtual viewpoint image generation unit 611 generates virtual viewpoint images 811, 812 to 813 by rendering the 3D models 805, 806 to 807 with the virtual imaging parameters 808, 809 to 810 for each time code, and transmits the generated images to the synthesis unit 612.

[0060] The composition unit 612 composes the foreground images 811, 812, and 813 of the virtual viewpoint images to generate a composite image 814. Here, the foreground virtual viewpoint image captured in real time is superimposed so that it is displayed in the foreground, followed by the virtual viewpoint image at time code 801, and the virtual viewpoint image at time code 803 at the back (the earlier the generation timing, the closer it is displayed). Note that this order of composition is an example and is not limited to this. For example, depth information for each virtual viewpoint image may be acquired, and composition may be performed so that the virtual viewpoint image closest to the virtual camera is displayed in the foreground and the virtual viewpoint image farthest from the virtual camera is displayed in the back. This depth information can be referenced, for example, by saving the distance from the virtual viewpoint to the 3D model calculated when generating the virtual viewpoint images and acquiring it when compositing.

[0061] In this example, the synthesis process of the virtual viewpoint images is performed by extracting afterimages from all the virtual viewpoint images and superimposing them. However, as described above, the synthesis process of the virtual viewpoint images is not limited to this, and may be, for example, a process of superimposing afterimages generated from virtual viewpoint images generated based on afterimage parameters onto virtual viewpoint images (for which foreground objects have not been extracted) generated based on reference parameters.

[0062] Next, a processing flow for displaying an afterimage according to this embodiment will be described with reference to Fig. 9. Fig. 9 is a flowchart showing an example of an afterimage display process performed in the image processing system 10 according to this embodiment.

[0063] In S901, the input acquisition unit 601 acquires information input on the GUI 401. The input acquisition unit 601 transmits the acquired information to the afterimage switching unit 602, the afterimage number determination unit 603, the interval determination unit 604, the transparency determination unit 605, and the position determination unit 606.

[0064] In S902 , the afterimage switching unit 602 acquires ON / OFF information of the button 402 from the input information acquired by the input acquisition unit 601 , converts it into information on whether to display or hide the afterimage, and transmits it to the afterimage parameter determination unit 609 .

[0065] In S903 , the afterimage number determination unit 603 obtains the number of afterimages input to the input space 403 from the input information obtained by the input obtaining unit 601 , and transmits it to the afterimage parameter determination unit 609 .

[0066] In S904 , the interval determination unit 604 obtains the frame interval of the afterimage input to the input space 404 from the input information obtained by the input obtaining unit 601 , and transmits it to the afterimage parameter determination unit 609 .

[0067] In S905 , the transparency determination unit 605 obtains the transparency of the afterimage input in the input space 405 from the input information obtained by the input obtaining unit 601 , and transmits it to the afterimage parameter determination unit 609 .

[0068] In S906, the position determination unit 606 obtains the offset of the virtual camera position for afterimage input into the input spaces 406, 407, and 408 from the input information obtained by the input obtaining unit 601, and transmits it to the afterimage parameter determination unit 609.

[0069] In S907 , the operation information acquisition unit 607 acquires operation information such as the degree of tilt or direction of the joysticks 301 a and 301 b operated by the user, and transmits it to the imaging parameter determination unit 608 .

[0070] In S908, the imaging parameter determination unit 608 acquires operation information from the operation information acquisition unit 607, converts it into virtual imaging parameters indicating the position and orientation of the virtual camera, and transmits it to the afterimage parameter determination unit 609 and the parameter transmission unit 610.

[0071] In S909, the afterimage parameter determination unit 609 sets afterimage parameters. Here, the afterimage parameter determination unit 609 determines the afterimage parameters according to information acquired from the afterimage number determination unit 603, the interval determination unit 604, the transparency determination unit 605, the position determination unit 606, and the afterimage parameter determination unit 609. For example, the afterimage parameter determination unit 609 acquires information on whether to display or hide afterimages from the afterimage switching unit 602, and if the afterimage is to be hidden, transmits blank information to the parameter transmission unit 610. Here, if the afterimage is to be displayed, the afterimage parameter determination unit 609 acquires the number of afterimages from the afterimage number determination unit 603, the frame interval from the interval determination unit 604, the transparency from the transparency determination unit 605, and the offset of the virtual camera position for afterimages from the position determination unit 606. In addition, the afterimage parameter determination unit 609 acquires information on virtual imaging parameters for the current time from the imaging parameter determination unit 608.

[0072] In S910, the parameter transmission unit 610 transmits the reference parameters determined by the imaging parameter determination unit 608 and the afterimage parameters determined by the afterimage parameter determination unit 609 to the virtual viewpoint image generation unit 611 included in the image processing device 102.

[0073] In S911, the virtual viewpoint image generation unit 611 acquires the reference parameters and the afterimage parameters from the parameter transmission unit 610. The virtual viewpoint image generation unit 611 acquires 3D model information of the time code corresponding to the acquired virtual imaging parameters from the 3D model storage device. Next, the virtual viewpoint image generation unit 611 renders foreground images seen by a virtual camera at the position and orientation indicated by the corresponding virtual imaging parameters for each of the acquired 3D models. Here, it is assumed that the foreground object portion of the image captured by the virtual camera is generated as a virtual viewpoint image. The generated multiple virtual viewpoint images are transmitted to the synthesis unit 612.

[0074] In S912, the composition unit 612 generates a composite image by superimposing the foregrounds of the multiple virtual viewpoint images simultaneously generated by parallel processing from the virtual viewpoint image generation unit 611. Through the above processing, a composite image is generated in which the foreground objects of the virtual viewpoint images of the past frames are superimposed as afterimages.

[0075] In this embodiment, an example has been described in which virtual imaging parameters are managed for each time code, and an afterimage is rendered and displayed based on such virtual imaging parameters (based on parameters different from real-time camera parameters). This type of processing allows for the display of an afterimage that is independent of the virtual camera parameters at the current time. Therefore, even if the subject in the foreground is stationary, for example, it is possible to impart an afterimage effect, such as displaying an afterimage by sliding it horizontally in real time.

[0076] 9 has been described as being performed by the information processing device 103 and the image processing device 102 in cooperation with each other. However, as long as similar processes can be executed, the configuration is not particularly limited to this. For example, the information processing device 103 may have a functional unit having the same functions as the image processing device 102, or the image processing device 102 may have an input unit and a display unit and perform some or all of the functions described as being performed by the information processing device 103.

[0077] [Embodiment 2] In the first embodiment, an example was described in which the position of a virtual camera that displays an afterimage is calculated from x, y, and z values ​​in three-dimensional coordinates input on a GUI, and the afterimage is rendered. In the process described in the first embodiment, the virtual camera for afterimage is positioned on a straight line from the virtual camera at the current time. On the other hand, in the second embodiment, a camera path file indicating the trajectory of the virtual camera for afterimage is input, and the virtual camera for afterimage is positioned on an arbitrary trajectory in the virtual space.

[0078] The information processing device 1203 according to the second embodiment receives as input a camera path file that describes relative parameters of all virtual cameras for afterimages with respect to the virtual imaging parameters of real-time video as the coordinates of the virtual cameras included in the virtual imaging parameters. This makes it possible to display afterimages not only on straight lines but also on curved lines, for example.

[0079] The image processing device 1202 and information processing device 1203 according to this embodiment have the same functions as the image processing device 102 and information processing device 103 of embodiment 1, respectively, and can execute the same processes. Differences in the functional units between the image processing device and the information processing device according to embodiment 1 will be described later with reference to FIG.

[0080] Fig. 10 is a diagram showing an example of a GUI 1001 displayed on the display unit 303 according to this embodiment. The information entered via the GUI 1001 and the processes executed thereby will be described below. Note that, among the components shown in Fig. 10, components having the same functions as those in the GUI 401 of Fig. 4 are given the same reference numerals, and duplicated descriptions will be omitted.

[0081] The input space 1002 is a field for inputting the path of a camera path file that represents the trajectory of a virtual camera that renders an afterimage. The user inputs the camera path file in the input space 1002 and clicks the button 1003. This operation causes the information processing device 1203 to acquire information about the camera path file.

[0082] FIG. 11 shows an example of the file structure of a camera path file 1101 input into the input space 1002. The camera path file 1101 is expressed as a table, for example, as shown in FIG. 11. In this embodiment, the virtual camera for afterimage moves in accordance with the real-time virtual camera, as in the first embodiment. Therefore, the parameters for afterimage are expressed as parameters obtained by adding relative parameters (offset values) to real-time virtual imaging parameters (reference parameters). In the example of FIG. 11, such offset values ​​are stored as descriptions from the second line onward in the camera path file 1101. Here, if N is input into the input space 403 on the GUI 1001 and I is input into the input space 404, the virtual imaging parameters 1102 on the second line become offset values ​​of the virtual camera for afterimage that renders the foreground I frames before the current time code. Furthermore, the virtual imaging parameters 1103 on the third line become offset values ​​of the parameters for afterimage that render the foreground 2×I frames before the current time code. The virtual imaging parameters 1104 on the nth row are offset values ​​of parameters for afterimages used to render the foreground that is (n-1)×I frames before the current time code.

[0083] Here, the virtual imaging parameters include (x, y, z) representing the coordinates of the virtual camera in three-dimensional space, (pan, tilt, roll) representing the orientation of the virtual camera, and a zoom value Zoom. If the offset number of the virtual camera parameters for afterimages written in the camera path file 1101 does not match the number of afterimages N to be displayed, the offset number written in the camera path file 1101 is used as the number of afterimages to be actually displayed. This is just an example, and the method for determining the number of afterimages is not limited to this. For example, the number of afterimages N may be compared with the offset number, and the smaller value may be set as the number of afterimages. Alternatively, the number of afterimages N may be set as the offset number. In such a method, the image processing device 102 interpolates the discrete offset values ​​and converts them into continuous values ​​connecting the first offset value to the last offset value. The image processing device 102 then acquires values ​​at equal intervals equal to the number of afterimages N and sets these as new offset values. The above is an example of the file structure of the camera path file 1101. The structure of the camera path file 1101 is not limited to this, as long as offset values ​​can be set in a similar manner.

[0084] Furthermore, the image processing device 102 may set the afterimage parameters to values ​​obtained by adding the camera parameters described in the afterimage camera path file 1101 to the parameters of the real-time virtual camera. According to this processing, the virtual camera for afterimage moves following the real-time virtual camera, and the afterimage can also be displayed in the output video so as to follow the foreground in real time.

[0085] Next, the functional configurations of the image processing device 1202 and information processing device 1203 included in the image processing system according to this embodiment will be described with reference to Fig. 12. Fig. 12 is a block diagram showing an example of the functional configurations of the image processing device 1202 and information processing device 1203 included in the image processing system according to the second embodiment. The information processing device 1203 according to this embodiment accepts user input, generates reference parameters and afterimage parameters for the current time based on the acquired user input, and transmits them to the image processing device 102. Of the configurations shown in Fig. 12, components similar to those shown in Fig. 6 of the first embodiment are denoted by the same reference numerals, and redundant description will be omitted.

[0086] The trajectory information acquisition unit 1201 reads the file path of the camera path file entered by the user in the input space 1002 on the GUI 1001, and acquires information on a 3D trajectory generation file such as the example shown in the camera path file 1101. The acquired file information is sent to the afterimage trajectory determination unit 1202.

[0087] The afterimage trajectory determination unit 1202 acquires file information from the trajectory information acquisition unit 1201 and reads the information. The afterimage trajectory determination unit 1202 acquires virtual imaging parameters such as the position or direction of the virtual camera from the file and sets them as offset values ​​of the virtual camera parameters for rendering afterimages. This set of offset values ​​is sent to the afterimage parameter determination unit 1204.

[0088] The afterimage parameter determination unit 1204 sets afterimage parameters. Here, the afterimage parameter determination unit 1204 determines the afterimage parameters according to information acquired from the afterimage number determination unit 603, the interval determination unit 604, the transparency determination unit 605, the afterimage trajectory determination unit 1202, and the position determination unit 606. For example, the afterimage parameter determination unit 1204 acquires information on whether to display or hide afterimages from the afterimage switching unit 602, and transmits blank information to the parameter transmission unit 610 when hiding afterimages. Here, when displaying afterimages, the afterimage parameter determination unit 1204 acquires the number of afterimages from the afterimage number determination unit 603, the frame interval from the interval determination unit 604, the transparency from the transparency determination unit 605, a group of offsets for afterimage parameters from the afterimage trajectory determination unit 1202, and the offset of the virtual camera position for afterimages from the position determination unit 606. Furthermore, the afterimage parameter determination unit 1204 acquires information on the camera parameters at the current time from the imaging parameter determination unit 608 .

[0089] Here, if the number of afterimages acquired from the afterimage number determination unit 603 does not match the offset number of the virtual camera parameter for afterimages acquired from the afterimage trajectory determination unit 1202, the offset number of the virtual imaging parameter is set as the number of afterimages. Note that this is just an example, and the method for determining the number of afterimages is not limited to this. For example, the number of afterimages acquired from the afterimage number determination unit 603 may be compared with the offset number acquired from the afterimage trajectory determination unit 1202, and the smaller value may be set as the number of afterimages. Alternatively, for example, the number of afterimages may be set as the offset number. When such a method is used, the image processing device 102 interpolates the offset values, which are discrete values, and converts them into continuous values ​​connecting the first offset value to the last offset value. Thereafter, the image processing device 102 acquires values ​​at equal intervals corresponding to the number of afterimages N and sets these as new offset values.

[0090] The afterimage parameter determination unit 1204 generates rendering parameters having a structure in which the alpha values ​​at the respective time codes for rendering afterimages and the virtual imaging parameters are set, and transmits the generated rendering parameters to the parameter transmission unit 610 .

[0091] Next, a processing flow for displaying an afterimage according to this embodiment will be described with reference to Fig. 13. Fig. 13 is a flowchart showing an example of afterimage display processing performed in the image processing system according to this embodiment. The processing according to Fig. 13 is the same as that shown in Fig. 9 except that S1301 to S1302 are performed between S906 and S907, and S1303 is executed instead of S909, and therefore a duplicated description will be omitted.

[0092] In S1301, the trajectory information acquisition unit 1201 acquires information on a three-dimensional trajectory generation file such as the example of the camera path file 1101. The acquired file information is sent to the afterimage trajectory determination unit 1202.

[0093] In S1302, the afterimage trajectory determination unit 1202 acquires file information from the trajectory information acquisition unit 1201 and reads the described information. The afterimage trajectory determination unit 1202 acquires virtual imaging parameters such as the position or orientation of the virtual camera from the file and sets them as virtual afterimage parameters for rendering an afterimage. The afterimage trajectory determination unit 1202 transmits a group of camera parameters, which are virtual imaging parameters for multiple frames, to the afterimage parameter determination unit 1204.

[0094] In S1303, the afterimage parameter determination unit 1204 sets afterimage parameters. Here, the afterimage parameter determination unit 1204 determines the afterimage parameters according to information acquired from the afterimage number determination unit 603, the interval determination unit 604, the transparency determination unit 605, the afterimage trajectory determination unit 1202, and the position determination unit 606. For example, the afterimage parameter determination unit 1204 acquires information on whether to display or hide afterimages from the afterimage switching unit 602, and if the afterimage is to be hidden, transmits blank information to the parameter transmission unit 610. Here, if the afterimage is to be displayed, the afterimage parameter determination unit 1204 acquires the number of afterimages from the afterimage number determination unit 603, the frame interval from the interval determination unit 604, the transparency from the transparency determination unit 605, a group of offsets for afterimage parameters from the afterimage trajectory determination unit 1202, and the offset of the virtual camera position for afterimages from the position determination unit 606. The afterimage parameter determination unit 1204 also acquires information on camera parameters at the current time from the imaging parameter determination unit 608. Furthermore, the afterimage parameter determination unit 1204 generates rendering parameters having a structure in which alpha values ​​for each time code for rendering afterimages and virtual imaging parameters are set, and transmits the rendering parameters to the parameter transmission unit 610.

[0095] In this embodiment, a method for inputting a 3D trajectory generation file and determining virtual imaging parameters for a virtual camera that renders an afterimage has been described. This process makes it possible to display an afterimage of the foreground on any trajectory in 3D space. Therefore, for example, it is possible to display an afterimage not only in a straight line but also in a curved line.

[0096] In the first and second embodiments, a configuration has been described in which a user inputs a numerical value into an input space on the GUI 401 or the GUI 1001, and parameters for displaying an afterimage are determined based on the input. However, the form of the GUI is not limited to this. An example of a GUI different from those in the first and second embodiments will be described below with reference to FIG. 14.

[0097] FIG. 14 shows a display unit 1401 that displays a three-dimensional virtual space, which is different from the GUI 401, the GUI 1001, and the display unit 302. In the example of FIG. 14, a simple dummy 3D model 1402 that imitates an actual subject and a camera model 1403 that represents a virtual camera capturing real-time video are pre-displayed on the display unit 1401. The user can view the 3D model 1402 and the camera model 1403 from any position and posture. Here, it is assumed that the 3D model 1402 and the camera model 1403 can be freely positioned and moved by user operation. The 3D model 1402 and the camera model 1403 are moved based on mouse operation (e.g., drag-and-drop on the model). The viewpoint can also be moved by drag-and-drop from any position other than the model on the window.

[0098] The following describes the operations and processing for determining the offset value of the virtual camera for afterimages. The user first presses button 1404. This operation switches the display on display 1401 from a mode for moving the viewpoint and model during mouse operation to a mode for determining virtual camera parameters. After pressing button 1404, the user drags and drops camera model 1403 from position 1405 as the starting point to any position 1406. While the mouse is in operation, a trajectory 1407 is displayed in three-dimensional space. When the mouse is released, camera models 1408...1409, the same number as the number of afterimages entered in the input space 403 on GUI 401 or GUI 1001, are displayed at equal intervals. In this example, these camera models 1408...1409 become virtual cameras for rendering afterimages. Trajectory 1407 is the trajectory of the camera model's movement and is drawn in three-dimensional virtual space. Changing the viewpoint position of the virtual camera allows for changes in depth, etc. Once the positions of the camera models have been determined, the mode for determining virtual imaging parameters can be switched to the mode for moving the viewpoint and model by pressing button 1404 again. The positions of the models of the multiple virtual cameras determined in this way are set as the relative positions (offset values) of the virtual cameras for afterimages with respect to the real-time virtual camera.

[0099] With this configuration, in the virtual viewpoint image displayed on the display unit 302, the virtual camera for rendering afterimages continues to track the real-time virtual camera while maintaining the relative position determined by operations on the display unit 1401. The user can adjust the positions of multiple camera models while checking the virtual viewpoint image displayed on the display unit 302. Furthermore, the user can fine-tune the afterimage virtual camera by moving any of the camera models 1408...1409 on the display unit 1401 while viewing the display unit 302. As described above, rather than directly inputting numerical values, the user can draw the trajectory of the afterimage virtual camera to generate a path, which allows the user to intuitively understand the trajectory on which the virtual cameras are positioned. This has the advantage of allowing the user to roughly predict how the subject and its afterimage will appear in the actually generated video, compared to inputting numerical values.

[0100] The disclosure of this specification includes the following image processing system, information processing method, and program. (Item 1) An image processing device that controls a virtual camera that captures a virtual viewpoint image from a virtual viewpoint based on a plurality of captured images captured by a plurality of imaging devices, a first setting means for setting a first imaging parameter of the virtual camera and a second imaging parameter different from the first imaging parameter; a generating means for generating a first virtual viewpoint image based on the first imaging parameters and a second virtual viewpoint image based on the second imaging parameters; a synthesis means for synthesizing the first virtual viewpoint image and the second virtual viewpoint image to generate a synthesized image; an image processing device comprising: an acquisition means for acquiring a user's operation content for the first imaging parameter or the second imaging parameter; a display means for displaying the generated composite image; an information processing device comprising: An image processing system comprising: (Item 2) Item 1, the image processing system characterized in that the first imaging parameters are imaging parameters of the virtual camera for a first frame, and the second imaging parameters are imaging parameters of the virtual camera for a second frame prior to the first frame. (Item 3) 3. The image processing system according to item 2, further comprising a second setting means for setting an interval between the first frame and the second frame. (Item 4) 4. The image processing system according to item 3, wherein the second setting means sets the interval between the first frame and the second frame based on a user input. (Item 5) The image processing system according to any one of items 1 to 4, characterized in that the synthesis means generates the synthesized image by synthesizing the first virtual viewpoint image with a foreground object extracted from the second virtual viewpoint image. (Item 6) the composite image is an image in which the foreground object is superimposed on the first virtual viewpoint image, 6. The image processing system according to item 5, further comprising a switching means for switching ON / OFF the superimposed display of the foreground object in the composite image. (Item 7) 7. The image processing system according to item 6, wherein the switching means switches the superimposed display of the foreground object ON / OFF based on a user input. (Item 8) 8. The image processing system according to any one of items 5 to 7, wherein the second imaging parameters include parameters for rendering a foreground object at the virtual viewpoint. (Item 9) 9. The image processing system according to any one of items 5 to 8, further comprising a fourth setting means for setting a position at which the foreground object is superimposed and displayed in the first virtual viewpoint image. (Item 10) further comprising a first acquiring means for acquiring a user input; 10. The image processing system according to any one of items 1 to 9, wherein the first setting means sets the second imaging parameters based on the first imaging parameters and the user input. (Item 11) the first setting means further sets a predetermined number of imaging parameters different from the first imaging parameters and the second imaging parameters; The generating means further generates virtual viewpoint images based on the predetermined number of imaging parameters. The image processing system according to any one of items 1 to 10, characterized in that the synthesis means generates a synthetic image by further synthesizing virtual viewpoint images based on the predetermined number of imaging parameters in addition to the first virtual viewpoint image and the second virtual viewpoint image. (Item 12) Item 12. The image processing system according to item 11, wherein the predetermined number is set based on user input. (Item 13) 13. The image processing system according to any one of items 1 to 12, further comprising a third setting means for setting the transparency of the second virtual viewpoint image to be synthesized by the synthesis means. (Item 14) 14. The image processing system according to any one of items 1 to 13, wherein the second imaging parameters include information indicating the position of a virtual camera in a virtual space that generates the second virtual viewpoint image. (Item 15) Item 15. The image processing system according to item 14, wherein the information indicating the position in the virtual space of the virtual camera that generates the second virtual viewpoint image is information indicating the trajectory of the virtual camera. (Item 16) a presentation means for presenting the generated composite image; The image processing system described in any one of items 1 to 15, further comprising a third acquisition means for acquiring the user's operation content for the first imaging parameter or the second imaging parameter. (Item 17) Item 17. The image processing system according to item 16, wherein the first setting means sets the first imaging parameter or the second imaging parameter based on the operation content. (Item 18) An information processing method using an image processing device that controls a virtual camera that captures a virtual viewpoint image from a virtual viewpoint based on a plurality of captured images captured by a plurality of imaging devices, comprising: setting first imaging parameters of the virtual camera and second imaging parameters different from the first imaging parameters; generating a first virtual viewpoint image based on the first imaging parameters and a second virtual viewpoint image based on the second imaging parameters; generating a composite image by combining the first virtual viewpoint image and the second virtual viewpoint image; acquiring user operation details for the first imaging parameter and the second imaging parameter; displaying the generated composite image; An information processing method comprising: (Item 19) A program for causing a computer to function as each means of the image processing device described in any one of items 1 to 17.

[0101] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0102] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0103] 10: Image processing system, 101: Imaging system, 102: Image processing device, 103: Information processing device

Claims

1. An image processing device that controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, a first setting means for setting a first imaging parameter of the virtual camera and a second imaging parameter different from the first imaging parameter; a generating means for generating a first virtual viewpoint image based on the first imaging parameters and a second virtual viewpoint image based on the second imaging parameters; a synthesis means for synthesizing the first virtual viewpoint image and the second virtual viewpoint image to generate a synthesized image; an image processing device comprising: an acquisition unit that acquires a user's operation content for the first imaging parameter or the second imaging parameter; a display means for displaying the generated composite image; an information processing device comprising: An image processing system comprising:

2. 2. The image processing system according to claim 1, wherein the first imaging parameters are imaging parameters of the virtual camera for a first frame, and the second imaging parameters are imaging parameters of the virtual camera for a second frame preceding the first frame.

3. 3. The image processing system according to claim 2, further comprising second setting means for setting the interval between the first frame and the second frame.

4. 4. The image processing system according to claim 3, wherein said second setting means sets the interval between said first frame and said second frame based on a user input.

5. 2. The image processing system according to claim 1, wherein the synthesis means generates the synthesized image by synthesizing the first virtual viewpoint image with a foreground object extracted from the second virtual viewpoint image.

6. the composite image is an image in which the foreground object is superimposed on the first virtual viewpoint image, 6. The image processing system according to claim 5, further comprising a switching means for switching ON / OFF the superimposed display of the foreground object on the composite image.

7. 7. The image processing system according to claim 6, wherein the switching means switches the superimposed display of the foreground object between ON and OFF based on a user input.

8. 6. The image processing system of claim 5, wherein the second imaging parameters include parameters for rendering a foreground object in the virtual viewpoint image.

9. 6. The image processing system according to claim 5, further comprising a fourth setting means for setting a position where the foreground object is to be superimposed and displayed on the first virtual viewpoint image.

10. The device further comprises a first acquiring means for acquiring a user input; 2. The image processing system according to claim 1, wherein the first setting means sets the second imaging parameters based on the first imaging parameters and the user input.

11. the first setting means further sets a predetermined number of imaging parameters different from the first imaging parameter and the second imaging parameter; The generating means further generates virtual viewpoint images based on the predetermined number of imaging parameters.

2. The image processing system according to claim 1, wherein the synthesis means generates a synthetic image by further synthesizing virtual viewpoint images based on the predetermined number of imaging parameters in addition to the first virtual viewpoint image and the second virtual viewpoint image.

12. 12. The image processing system of claim 11, wherein the predetermined number is set based on a user input.

13. 2. The image processing system according to claim 1, further comprising a third setting unit for setting a transparency of the second virtual viewpoint image to be synthesized by the synthesis unit.

14. 2. The image processing system according to claim 1, wherein the second imaging parameters include information indicating a position of a virtual camera in a virtual space for generating the second virtual viewpoint image.

15. 15. The image processing system according to claim 14, wherein the information indicating the position of a virtual camera in the virtual space for generating the second virtual viewpoint image is information indicating a trajectory of the virtual camera.

16. a presentation means for presenting the generated composite image; 2. The image processing system according to claim 1, further comprising: a third acquisition unit that acquires a user's operation on the first imaging parameter or the second imaging parameter.

17. 17. The image processing system according to claim 16, wherein the first setting means sets the first imaging parameter or the second imaging parameter based on the operation content.

18. An information processing method for an image processing system that controls a virtual camera corresponding to a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, comprising: setting first imaging parameters of the virtual camera and second imaging parameters different from the first imaging parameters; generating a first virtual viewpoint image based on the first imaging parameters and a second virtual viewpoint image based on the second imaging parameters; generating a composite image by combining the first virtual viewpoint image and the second virtual viewpoint image; acquiring user operation details for the first imaging parameter and the second imaging parameter; displaying the generated composite image; An information processing method comprising:

19. A program for causing a computer to function as each of the means of the image processing system according to any one of claims 1 to 17.

Images