Image processing device, image processing method, and program

Abstract

To enable easy grasping of positions and postures of switchable virtual cameras.SOLUTION: An image generation device 10 is configured to provide control to acquire three or more virtual viewpoint images containing the same object, and display a specific virtual viewpoint image, among the three or more virtual viewpoint images displayed in a first display area, in a second display area that is larger than the first display area according to a user operation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to techniques for controlling the display of virtual viewpoint images. [Background technology]

[0002] A technology that generates an image (virtual viewpoint image) from any virtual camera (virtual viewpoint) based on a user's operation by using the multiple images captured by installing multiple cameras at different positions and capturing images synchronously has been attracting attention. With such a technology, for example, it becomes possible to watch highlight scenes of soccer or basketball from various angles, which can give the user a high sense of realism compared to normal images.

[0003] Patent Document 1 discloses a technology in which viewpoint information such as the position and orientation of a virtual camera is registered, and the registered viewpoint information is read based on user operation, thereby switching to a virtual camera with the registered position and orientation at any timing. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-114869 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, there is a concern that the position and orientation of the switchable virtual camera may not be easily grasped. For example, when a basketball is captured as an image target and the switchable virtual camera follows a player, the switchable virtual camera moves in accordance with the player's movements, and therefore, there is a concern that the position and orientation of the switchable virtual camera may not be easily grasped.

[0006] The present disclosure makes it easy to grasp the position and orientation of a switchable virtual camera. [Means for solving the problem]

[0007] The image processing device disclosed herein is characterized by having an acquisition means for acquiring three or more virtual viewpoint images including the same subject, and a display control means for controlling, based on user operation, the display of a specific virtual viewpoint image from the three or more virtual viewpoint images displayed in a first display area in a second display area larger than the first display area. Effect of the Invention

[0008] According to the present disclosure, the position and orientation of a switchable virtual camera can be easily grasped. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 illustrates an example of a configuration of an image generating device 10. [Diagram 2] FIG. 1 is a diagram showing an example of installation of an imaging system 101. [Diagram 3] FIG. 2 is a diagram illustrating tracking information. [Figure 4] FIG. 2 is a diagram illustrating a tracking viewpoint. [Diagram 5] 5 is a diagram illustrating a display unit 109 and an input device 510. FIG. [Figure 6] FIG. 2 is a diagram illustrating an example of a world coordinate system. [Figure 7] 1 is a diagram illustrating an example of a hardware configuration of an image generating device 10. FIG. [Figure 8] FIG. 2 is a diagram illustrating a functional configuration of a virtual viewpoint generation unit 105. [Figure 9] FIG. 2 is a diagram illustrating a functional configuration of a display control unit 108. [Figure 10] 1A to 1C are diagrams illustrating display contents displayed on a display unit 109. [Figure 11] 13 is a flowchart showing a process in which the image generating device 10 displays an operation virtual viewpoint image and a switching virtual viewpoint image on the display unit 109. [Figure 12] 10 is a diagram illustrating the display order of switching virtual viewpoint images by the display control unit 108. FIG. [Figure 13] 13 is a flowchart showing a process flow in which the display control unit 108 switches between a selected switch virtual viewpoint image and an operation virtual viewpoint image. [Figure 14] 13 is a diagram showing a display screen displayed by the display unit 109 in the process of switching between a selected switch virtual viewpoint image and an operation virtual viewpoint image. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the present invention will be described in detail based on preferred embodiments thereof with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present disclosure is not limited to the configurations shown in the drawings.

[0011] The image processing system is a system that generates a virtual viewpoint image that represents a scene from a specified virtual viewpoint based on a plurality of images based on images captured by a plurality of imaging devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by a user, and includes, for example, an image corresponding to a viewpoint selected by a user from a plurality of candidates. In addition, although the present embodiment will mainly describe a case where the virtual viewpoint image is a moving image, the virtual viewpoint image may also be a still image.

[0012] The viewpoint information used to generate the virtual viewpoint image is information indicating the position and direction (line of sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter indicating the three-dimensional position of the virtual viewpoint and a parameter indicating the direction of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter indicating the size of the visual field (angle of view) of the virtual viewpoint. In addition, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting a moving image of the virtual viewpoint image, and may be information indicating the position and direction of the virtual viewpoint at each of a plurality of consecutive time points.

[0013] In this embodiment, the multiple imaging devices are cameras each having an independent housing and capable of capturing images from a single viewpoint. However, this is not limited to this, and two or more imaging devices may be configured in the same housing. For example, a single camera having multiple lens groups and multiple sensors and capable of capturing images from multiple viewpoints may be installed as the multiple imaging devices.

[0014] 1 is a configuration diagram of an image generating device 10. The image generating device 10 has a three-dimensional model generating unit 102, a three-dimensional model tracking unit 103, a storage unit 104, a virtual viewpoint generating unit 105, a virtual viewpoint image generating unit 106, an input unit 107, a display control unit 108, and a display unit 109. Note that the configuration is not limited to the above, and a plurality of units may be included in another device. For example, the three-dimensional model tracking unit 103, the virtual viewpoint generating unit 105, the input unit 107, the display control unit 108, and the display unit 109 may be included in another image processing device.

[0015] 2, the imaging system 101 has multiple physical cameras 201 installed at different positions surrounding an imaging area 202, and captures images in a time-synchronized manner. A plurality of images captured synchronously from multiple viewpoints are transmitted to the three-dimensional model generating unit 102. The imaging area 202 may be an imaging studio where imaging is performed to generate a virtual viewpoint image, a stadium where a sporting event is held, or a stage where a concert or a play is held. The imaging system 101 may be a device that captures not only video but also audio and other sensor information.

[0016] The three-dimensional model generating unit 102 extracts the subject as a foreground from the multiple captured images sent from the imaging system 101, and generates a three-dimensional model of the subject from the extracted foreground image. A method of extracting the foreground includes a method using background difference information. For example, a state in which the foreground does not exist is captured in advance as a background image, and a difference between the image in which the foreground exists and the background image is calculated. If the difference value is greater than a threshold value, the pixel position is determined to be the foreground. There are various other methods for extracting the foreground, such as a method using image features related to the subject or a method using machine learning, but the present proposal does not limit the method for extracting the foreground. The three-dimensional model may be generated by a volume intersection method or may be generated using depth data obtained from stereo image processing. In this case, the method of generating the three-dimensional model is not limited. The generated three-dimensional model is transmitted to the three-dimensional model tracking unit 103 and the storage unit 104.

[0017] The three-dimensional model tracking unit 103 assigns an identifier to each of the three-dimensional models generated by the three-dimensional model generating unit 102, and assigns the position information of the three-dimensional model as tracking information and transmits it to the storage unit 104. The identifier has unique ID information for identifying each subject, and attribute information for identifying the attributes of the subject, such as the team to which each subject belongs. As shown in FIG. 3, the tracking information has position information of the three-dimensional model for each time (time code), and is stored in association with the identifier. The position of the subject is approximated, for example, by the center of gravity of a bounding box surrounding the subject. Note that the position information may be associated with a wireless tag (for example, GPS) attached to the subject and the three-dimensional model. In this embodiment, the method for tracking the three-dimensional model uses a wireless tag, but is not limited to this. For example, a representative position of the subject may be determined using a part of the three-dimensional model of the subject, and tracking information may be generated from the movement range of the representative position of the subject in successive frames.

[0018] The storage unit 104 saves and accumulates data (material data) used to generate a virtual viewpoint image and tracking information. Specifically, the material data includes the captured image and three-dimensional model received from the three-dimensional model generation unit 102, and the camera parameters of each imaging device. Note that a background model and a background texture image are stored in advance in the storage unit 104 as data used to generate the background of the virtual viewpoint image. The background model may be data of a studio set or a field of a stadium captured in advance, or may be data of a fictitious space generated by CG or the like. In this embodiment, it is assumed that a three-dimensional model of a basketball goal is stored as the background model. The material data group and tracking information stored in the storage unit 104 are transmitted to the virtual viewpoint generation unit 105, the virtual viewpoint image generation unit 106, and the display control unit 108 according to the processing of the image generation processing device 10.

[0019] The virtual viewpoint generating unit 105 generates camera parameters of a virtual camera that is a viewpoint (following viewpoint) that follows the three-dimensional model based on the tracking information received from the storage unit 104. The camera parameters of the virtual camera include parameters for specifying a position and attitude, a focal length, an angle of view, and a time. The camera parameters may include parameters that define other elements, or may not include some of the above-mentioned parameters. The generated camera parameters are transmitted to the virtual viewpoint image generating unit 106. For example, as shown in FIG. 4, the camera parameters are controlled so that the virtual camera 401 is located 3 meters away from the player 402 to be followed and on a straight line connecting the player 402 to be followed and the goal 403. The position of the camera parameters of the following viewpoint is not limited to this. For example, the camera may be provided at a position a predetermined distance away from the straight line connecting the player 402 to be followed and the goal 403. It is sufficient that the following target and the object of interest are included within the angle of view of the virtual camera. The method of determining the camera parameters of the following viewpoint is not limited to this. In this embodiment, the image generating device 10 has a plurality of virtual viewpoint generating units 105, which generate camera parameters of virtual cameras that track three-dimensional models of different subjects, respectively. The camera parameters of the virtual cameras generated by the plurality of virtual viewpoint generating units 105 are transmitted to different virtual viewpoint image generating units 106. However, this is not limited thereto, and one virtual viewpoint generating unit 105 may generate camera parameters of multiple virtual cameras, or the camera parameters of multiple virtual cameras may be transmitted to one virtual viewpoint image generating unit 106.

[0020] The virtual viewpoint image generating unit 106 acquires material data from the storage unit 104 and camera parameters of the virtual camera from the virtual viewpoint generating unit 105, and generates a virtual viewpoint image corresponding to the virtual camera. For example, model-based rendering can be used as a method for generating the virtual viewpoint image. This process makes it possible to generate a virtual viewpoint image viewed from the position and attitude of the virtual camera. Note that the method for generating the virtual viewpoint image is not limited to this. Note that, unless otherwise specified, the term "image" in this disclosure will be described as including the concepts of both moving images and still images. The generated virtual viewpoint image is transmitted to the display control unit 108.

[0021] Fig. 5 is a diagram for explaining the display unit 109 and the input device 510. Fig. 5(a) shows the layout of an operation virtual viewpoint image and a switching virtual viewpoint image displayed on the display unit 109, and Fig. 5(b) shows an example of the input device 510 connected to the input unit 107.

[0022] The input unit 107 acquires input information based on a user operation from the input device 510, as in the example of FIG. 5(b). The input device 510 has a stick 511a, a stick 511b, a seesaw switch 512, and a button group 513. The user operates them to change the camera parameters of the virtual camera. The stick 511a and the stick 511b each have an operation axis with three degrees of freedom, and the position of the virtual camera is operated by the stick 511a, and the attitude of the virtual camera is operated by the stick 511b. In addition, the focal length and the angle of view of the virtual camera are changed by tilting the seesaw switch 512 to the plus side or the minus side. The button group 513 has arrow buttons indicating up, down, left, right, and right directions, and the user switches between the virtual viewpoint for operation and the virtual viewpoint for switching. Note that the configuration of the input device 510 is not limited to this, and may be, for example, a tablet terminal, a keyboard, a mouse, or the like. The input information is information for changing the position and attitude of the virtual camera, the focal length and the angle of view, and information for switching between the virtual viewpoint for operation and the virtual viewpoint for switching.

[0023] The display control unit 108 sets virtual viewpoint images to be displayed in an operation virtual viewpoint display area 501 (first display area) and a switching virtual viewpoint display area 502 (second display area). As shown in FIG. 5(a), the operation virtual viewpoint display area 501 and the switching virtual viewpoint display area 502 are arranged on the display screen of the display unit 109. In the operation virtual viewpoint display area 501, a virtual viewpoint image of an operation virtual viewpoint that accepts a user operation to change the position and attitude of a virtual camera is displayed. In the switching virtual viewpoint display area 502, a virtual viewpoint image of a virtual viewpoint (switching virtual viewpoint) that can be switched as the operation virtual viewpoint is displayed. The order of the virtual viewpoint images to be displayed as the switching virtual viewpoint is determined based on the position information of the three-dimensional model of the tracking target in each viewpoint. A plurality of switching virtual viewpoint display areas 502 are provided, and three are provided in the example of FIG. 5(a). Note that a plurality of switching virtual viewpoint images may be arranged and displayed in one display area. When the user selects one of the switching virtual viewpoints via the input unit 107, the selected switching virtual viewpoint is set as the operation virtual viewpoint and is displayed in the operation virtual viewpoint display area 501. Furthermore, the display control unit 108 sets an image to be displayed in the third display area 503. In this embodiment, an overhead image is displayed in the third display area 503, but is not limited to this. For example, a virtual advertisement, the score of a match, or the like may be displayed.

[0024] The display unit 109 displays the operation virtual viewpoint image and the switching virtual viewpoint image set by the display control unit 108 .

[0025] FIG. 6 is a diagram showing a world coordinate system (x, y, z) in the imaging space of the virtual camera. The world coordinate system is used to represent the camera parameters of the virtual camera and the position of the subject. Here, the subject is a tangible object existing in the imaging space, such as a field 601, a ball 602, a player 402, etc. In the world coordinate system, the center of the field 601 is set as the origin (0, 0, 0). Also, the x-axis is set in the direction of the long side of the field 601, the y-axis is set in the direction of the short side of the field 601, and the z-axis is set in the vertical direction with respect to the field 601. Note that the method of setting the world coordinate system is not limited to this.

[0026] 7 is a diagram showing an example of a hardware configuration of the image generating device 10. The image generating device 10 includes a CPU 701, a RAM 702, a ROM 703, an auxiliary storage device 704, and a communication I / F 705.

[0027] The CPU 701 realizes each function of the image generating device 10 shown in FIG. 1 by controlling the entire image generating device 10 using computer programs and data stored in the RAM 702 and the ROM 703. The image generating device 10 may have one or more dedicated hardware devices different from the CPU 701, and at least a part of the processing by the CPU 701 may be executed by the dedicated hardware devices. Examples of the dedicated hardware devices include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), and a DSP (Digital Signal Processor). The RAM 702 temporarily stores programs and data supplied from the auxiliary storage device 704, and data supplied from the outside via the communication I / F 705. The ROM 703 stores programs that do not require modification. The auxiliary storage device 704 is composed of, for example, a hard disk drive, and stores various data such as image data and audio data. The communication I / F 705 is used for communication with an external device of the image generating device 10. For example, when the image generating device 10 is connected to an external device by wire, a communication cable is connected to the communication I / F 705. When the image generation device 10 has a function for wireless communication with an external device, the communication I / F 705 includes an antenna.

[0028] 8 is a block diagram showing an example of the functional configuration of the virtual viewpoint generating unit 105. The virtual viewpoint generating unit 105 has a three-dimensional model designating unit 801, an object of interest setting unit 802, a virtual viewpoint setting unit 803, and a camera parameter calculating unit 804.

[0029] The three-dimensional model designation unit 801 designates a three-dimensional model for generating a following viewpoint from among the three-dimensional models tracked by the three-dimensional model tracking unit 103. The three-dimensional model for generating a following viewpoint is designated by the user via the input unit 107. Alternatively, it may be designated from attribute information included in the tracking information. For example, a three-dimensional model having "Player (Team A)" as attribute information is designated as the three-dimensional model for generating a following viewpoint. The three-dimensional model designation unit 801 is provided in each of the virtual viewpoint generation units 105, and each designates a different three-dimensional model.

[0030] The attention object setting unit 802 sets an attention object for determining the camera parameters of the following viewpoint. The attention object is a three-dimensional model that is a reference for the position and attitude of the virtual camera, and is set by the user via the input unit 107. The user does not necessarily need to set an attention object. If the user does not set an attention object, a three-dimensional model of the subject set for each sport as a default becomes the attention object. For example, in the case of basketball, a goal is set as the attention object as a default. Note that the attention object is assumed to be a stationary object, but is not limited to this. For example, in a race such as a horse race or a bicycle race, a player running in the lead may be set as the attention object. In that case, in the following viewpoint described later, a following viewpoint can be generated in which the player being watched faces the leading player and follows the leading player. Each following viewpoint is a viewpoint that includes the subject to be followed and the attention object within the angle of view of the virtual camera. The attention object is a three-dimensional model common to each following viewpoint, and is a three-dimensional model other than the subject to be followed in each following viewpoint. In other words, the virtual viewpoints generated by the multiple virtual viewpoint generators 105 each include a different subject while including the same subject within the angle of view.

[0031] The virtual viewpoint setting unit 803 specifies conditions for determining the position and orientation of the virtual camera at the tracking viewpoint. As the conditions for the position and orientation of the virtual camera, the distance between the subject to be tracked and the virtual camera, and the relationship between the subject to be tracked and the object of interest and the position and orientation of the virtual camera are set. For example, the virtual camera position is specified so that the virtual camera is located 5 m away from the subject position of the subject to be tracked, at a height z=2.0 m, and on a straight line connecting the virtual camera, the subject to be tracked, and the object of interest in this order. The orientation of the virtual camera is specified so that the center of the subject to be tracked is always on the optical axis of the virtual camera. Note that the conditions for the position and orientation of the virtual camera at the tracking viewpoint are not limited to this. It is sufficient that the subject to be tracked and the object of interest are within the angle of view of the virtual camera. Alternatively, it is sufficient that the distance between the subject to be tracked and the virtual camera is smaller than the distance between the object of interest and the virtual camera. For example, the virtual camera position may be a straight line that is rotated by a predetermined angle around the z axis around the subject to be tracked, which is the straight line connecting the virtual camera, the subject to be tracked, and the object of interest in this order. The orientation of the virtual camera may be set so that the center of the subject to be tracked and the center of the object of interest are on the optical axis.

[0032] A camera parameter calculation unit 804 acquires position information of the three-dimensional model of the subject to be tracked and the object of interest, and calculates the camera parameters of the virtual camera at the tracking viewpoint based on the conditions specified by the virtual viewpoint setting unit 803 .

[0033] Fig. 10 is a diagram for explaining the display contents displayed on the display unit 109. In the example of Fig. 10, the object of interest is the goal 403, and each viewpoint following the player 402 to be followed is displayed in the operation virtual viewpoint display area 501 and the switching virtual viewpoint display area 502. Here, the subscript a indicates that the virtual viewpoint image is displayed in the operation virtual viewpoint display area 501, and b, c, and d indicate that the virtual viewpoint images are displayed in the switching virtual viewpoint display area 502. Meanwhile, both the virtual viewpoint image displayed in the operation virtual viewpoint display area 501 and the virtual viewpoint image displayed in the switching virtual viewpoint display area 502 include the same object of interest, the goal 403.

[0034] FIG. 9 is a block diagram showing an example of the functional configuration of the display control unit 108. As shown in FIG.

[0035] The display viewpoint control unit 901 acquires information on which subject is to be tracked and position information of the three-dimensional model of the subject to be tracked from the virtual viewpoint image generation unit 106. Then, based on the relationship between the three-dimensional model of the subject to be tracked in the virtual viewpoint set as the operation virtual viewpoint and the subject to be tracked set as the other virtual viewpoint, it determines the tracking viewpoint to be displayed in the switching virtual viewpoint display area 502. The display viewpoint control unit 901 sets the virtual viewpoint determined to be displayed as the switching virtual viewpoint. A method for determining the virtual viewpoint to be displayed will be described in detail later.

[0036] The display order control unit 902 determines the order of displaying the virtual viewpoint images of the virtual viewpoint set as the switching virtual viewpoint by the display viewpoint control unit 901 in the switching virtual viewpoint display area 502 based on the position information of the three-dimensional model of each tracking target. The method of determining the order will be described in detail later. The information on the order of the tracking viewpoints is transmitted to the display switching unit 903.

[0037] The display switching unit 903 arranges the virtual viewpoint images displayed in the switching virtual viewpoint display area 502 according to the order determined by the display order control unit 902. When a user operation for switching the switching virtual viewpoint image to the operation virtual viewpoint image is acquired, the switching virtual viewpoint selected by the user is set as the operation virtual viewpoint. The display switching unit 903 also sets a screen layout such as the size and display position of the image to be displayed on the display unit 109. Based on the set layout, the screen configuration of the operation virtual viewpoint display area 501 and the switching virtual viewpoint display area 502 configured on the display unit 109 is determined. In this embodiment, the operation virtual viewpoint display area 501 is set to be larger than the switching virtual viewpoint display area 502. In addition, when other display images, for example, an overhead image showing the entire field, are displayed on the display unit, the display switching unit 903 also sets the layout of the other display images. When the overhead image is displayed, one of the virtual viewpoint image generating units 106 generates a virtual viewpoint image of the overhead image.

[0038] When the marker control unit 904 acquires a user operation to select a switching virtual viewpoint image to be displayed in the switching virtual viewpoint display area 502, the marker control unit 904 highlights the subject corresponding to the selected switching virtual viewpoint image. Specifically, a selection marker is superimposed on the position of the three-dimensional model of the subject to be tracked in the selected virtual viewpoint image on the virtual viewpoint image displayed in the operation virtual viewpoint display area 501. The selection marker moves in accordance with the movement of the player based on the tracked position information.

[0039] FIG. 12 is a diagram for explaining a process of determining the display viewpoint and the display order to be displayed on the display unit 109. When two subjects are imaged and tracked as the imaging target, the number of virtual viewpoints is small, so even if two switching virtual viewpoint images are displayed in the switching virtual viewpoint display area 502, the user can easily find the desired switching virtual viewpoint image. On the other hand, when a game played by a large number of people such as basketball or baseball is imaged as the imaging target, the virtual viewpoint is set according to the number of players, so the user may not be able to easily identify the desired virtual viewpoint. Therefore, in this embodiment, an example will be described in which three or more virtual viewpoint images are displayed as switching virtual viewpoint images from among six or more virtual viewpoint images generated. Note that the number of images to be displayed as switching virtual viewpoint images is not limited to five, and is changed depending on the size of the display device, the size of the switching virtual viewpoint image, and the like. First, among the multiple virtual viewpoint images, the switching virtual viewpoint image is determined based on the distance between the three-dimensional model of the subject to be tracked in the operation virtual viewpoint image and the three-dimensional model of the subject to be tracked in the switching virtual viewpoint image. In this case, the display viewpoint control unit 901 calculates the distance between the three-dimensional model of the subject to be followed in the virtual viewpoint image for operation and the three-dimensional model of the subject to be followed in the virtual viewpoint image for switching. Then, it is determined that a preset number of virtual viewpoint images are to be displayed as the virtual viewpoint for switching in order of the shortest distance. The method of determining the virtual viewpoint image for switching to be displayed is not limited to this, and it may be determined that a following viewpoint is displayed that targets a three-dimensional model of a player of the same team (same attribute) as the three-dimensional model of the subject to be followed in the virtual viewpoint for operation. In this case, the display viewpoint control unit 901 refers to attribute information included in the identifier of each three-dimensional model tracked from the storage unit 104. Then, it is determined that a following viewpoint that follows a three-dimensional model having attributes matching the three-dimensional model of the subject to be followed in the virtual viewpoint for operation is to be displayed. In the example of FIG. 12, in the virtual viewpoint display area 502 for switching, a player of the same team as the three-dimensional model set as the subject to be followed in the virtual viewpoint display area 501 for operation is displayed in a shading pattern.The method of displaying subjects having the same attribute information is not limited to this, and for example, the contour of the three-dimensional model of the subject may be emphasized, or subjects not having the same attribute information may be made semi-transparent. In this embodiment, it is assumed that which virtual viewpoint image is to be displayed as the operation virtual viewpoint image is specified by a user operation.

[0040] The order of the switching virtual viewpoint images to be displayed next is determined based on, for example, the distance between the three-dimensional model of the tracking target at each tracking viewpoint and the target object. In this case, the display viewpoint control unit 901 calculates the distance between each three-dimensional model and the target object from the position information of each three-dimensional model acquired, and determines the order in ascending order of distance value. In the example of FIG. 12, the switching virtual viewpoint display area 502 displays the switching virtual viewpoint images from the left in ascending order of distance value. The order of the switching virtual viewpoint images may be determined by information obtained from outside the image generating device 10. For example, the field goal success rate of each player may be acquired, and the switching virtual viewpoint images corresponding to the players may be arranged in descending order of success rate. Alternatively, the position information of each player may be acquired, and the switching virtual viewpoint images corresponding to the players may be arranged in descending order of field goal success rate in the area where the player is standing.

[0041] FIG. 11 is a flowchart showing a process in which the image generating device 10 displays a virtual viewpoint image for operation and a virtual viewpoint image for switching on the display unit 109. This process is executed for the update interval of the virtual viewpoint image for operation. In this embodiment, it is assumed that the virtual viewpoint image displayed as the virtual viewpoint image for operation and the virtual viewpoint image for switching are images of 60 frames per second. Therefore, this process (S1101 to S1109) is executed for each frame. Note that this is not limited to this, and this process may be executed for each set of frames. Alternatively, the update interval may be different between the virtual viewpoint image for operation and the virtual viewpoint image for switching. Also, it is assumed that the initial value (virtual viewpoint) of the virtual viewpoint for operation is determined in advance.

[0042] In step S1101, the three-dimensional model generation unit 102 acquires material data of a virtual viewpoint image. Specifically, a three-dimensional model of a subject is generated based on parameters such as multiple images received from the imaging system 101 and the positions and orientations of each imaging device. Note that this is not limiting, and the three-dimensional model of a subject may be acquired from another device. In this case, an identifier including the position information and attribute information of the three-dimensional model is also acquired, and step S1102 is skipped.

[0043] In step S1102, the three-dimensional model tracking unit 103 acquires tracking information of each subject. It tracks the position information of each three-dimensional model generated by the three-dimensional model generation unit 102. Then, it assigns an identifier to each three-dimensional model and transmits it to the storage unit 104 together with the position information (tracking information).

[0044] In step S1103, the virtual viewpoint generating unit 105 generates a virtual viewpoint (tracking viewpoint). Specifically, the multiple virtual viewpoint generating units 105 generate camera parameters that become a tracking viewpoint for each specified subject based on the position information of the three-dimensional model received from the storage unit 104. In this embodiment, camera parameters are generated such that the tracking target and the target object are included within the angle of view of the virtual camera. As a result, the virtual viewpoint images of all the tracking viewpoints each include a different subject, but include the same target object.

[0045] In step S1104, the virtual viewpoint image generating unit 106 acquires the material data from the storage unit 104 and the camera parameters from the virtual viewpoint generating unit 105. Then, the virtual viewpoint image generating unit 106 performs rendering of a three-dimensional model of the subject as viewed from the position and attitude of the virtual camera to generate a virtual viewpoint image. Note that one set of the virtual viewpoint generating unit 105 and the virtual viewpoint image 106 is provided for each tracking target, so that the same number of virtual viewpoint images as the number of tracking targets are generated.

[0046] In step S1105, the display viewpoint control unit 901 determines whether or not the user has performed an operation to select one of the switching virtual viewpoint images displayed in the switching virtual viewpoint display area 502 via the input unit 107. If an operation to select one of the switching virtual viewpoint images is acquired, the process proceeds to step S1106; if not, the process proceeds to step S1107.

[0047] In step S1106, the display switching unit 903 sets the virtual viewpoint of the switch virtual viewpoint image selected in step S1105 as the operation virtual viewpoint.

[0048] In step S1107, the display viewpoint control unit 901 determines the tracking viewpoint to be displayed in the switching virtual viewpoint display area 502 based on the relationship between the three-dimensional model of the subject to be tracked in the operation virtual viewpoint and the three-dimensional model of the subject to be tracked in the other virtual viewpoints. Based on the determination result, a predetermined number of tracking viewpoints are set as switching virtual viewpoints out of all the tracking viewpoints.

[0049] In step S1108, the display order control unit 902 determines the order in which the tracking viewpoints set as the switching virtual viewpoint in step S1107 are to be displayed in the switching virtual viewpoint display area 502 based on the position information of the three-dimensional models of the subjects to be tracked.

[0050] In step S1109, the display switching unit 903 arranges the virtual viewpoint image of the following viewpoint set as the operation virtual viewpoint in the operation virtual viewpoint display area 501. The display switching unit 903 arranges the virtual viewpoint image of the following viewpoint set as the switching virtual viewpoint in the switching virtual viewpoint display area 502.

[0051] By repeating the above process for each frame, the switching virtual viewpoint image and the operation virtual viewpoint image each contain a different subject and the same object of interest, which makes it easier for the user to grasp the relationship between the position and attitude of the operation virtual viewpoint and the switching virtual viewpoint.

[0052] Fig. 13 is a flowchart showing the flow of a process for switching the selected switching virtual viewpoint as the operation virtual viewpoint. This process is also executed for the update interval of the operation virtual viewpoint image, similar to the process shown in Fig. 11, and is executed for each frame in this embodiment.

[0053] In step S1301, the display switching unit 903 determines whether or not the user has acquired an operation to select one of the switching virtual viewpoint images via the input unit 107. If the user has acquired an operation to select one of the switching virtual viewpoint images, the process proceeds to step S1302. If the user has not acquired an operation to select one of the switching virtual viewpoint images, the process proceeds to step S1307.

[0054] In step S1302, the marker control unit 904 determines whether or not a selection marker is displayed in the selected switching virtual viewpoint image. If the selection marker is displayed, the process proceeds to step S1303, and if the selection marker is not displayed, the process proceeds to step S1304. The selection marker is a circular icon as shown by 1403 in FIG.

[0055] In step S1303, the display switching unit 903 sets the virtual viewpoint corresponding to the selected switching virtual viewpoint image as the operation virtual viewpoint, and cancels the selection marker displayed on the three-dimensional model of the subject to be tracked in the selected switching virtual viewpoint image and the thick frame displayed in the selected switching virtual viewpoint image.

[0056] In step S1304, the marker control unit 904 superimposes a selection marker on the position of the three-dimensional model of the subject to be tracked in the switching virtual viewpoint image selected by the user on the virtual viewpoint image displayed in the operation virtual viewpoint display area 501. Furthermore, a thick frame 1402 is displayed on the selected switching virtual viewpoint image displayed in the switching virtual viewpoint display area 502.

[0057] In step S1305, the marker control unit 904 determines whether or not a selection marker is displayed on the 3D model of the subject to be tracked in the switching virtual viewpoint image different from the switching virtual viewpoint image selected by the user and acquired in step S1301. If the selection marker is displayed, the process proceeds to step S1306, and if the selection marker is not displayed, the process proceeds to step S1307.

[0058] In step S1306, the marker control unit 904 cancels (deletes) the selection marker displayed on the three-dimensional model of the subject to be tracked in the switching virtual viewpoint image different from the switching virtual viewpoint image selected by the user. Furthermore, if the thick frame 1402 is displayed in the switching virtual viewpoint image different from the switching virtual viewpoint image selected by the user, the thick frame is canceled (deleted).

[0059] In step S1307, the display switching unit 903 displays the switching virtual viewpoint image and the operation virtual viewpoint image on the display unit 109.

[0060] The above process allows the user to easily grasp the subject to be tracked by the switching virtual viewpoint image. In particular, when there are many subjects to be tracked, the number of switching virtual viewpoint images also increases, and there is a risk that the user cannot easily grasp the switching virtual viewpoint image desired by the user. In that case, by performing the above process, it becomes possible to easily grasp which switching virtual viewpoint image corresponds to each subject in the operation virtual viewpoint image. In this embodiment, a selection marker is displayed, but this is not limited to this, and a display may be performed in which the color of the three-dimensional model of the target subject is changed or the contour is emphasized. In addition, as a modified example, the user may select a subject on the operation virtual viewpoint image, and the corresponding switching virtual viewpoint image may be highlighted, or both the subject on the operation virtual viewpoint image and the switching virtual viewpoint image may be highlighted. In this modified example, the position of the subject corresponding to the switching virtual viewpoint image can be intuitively grasped, so that the position and attitude of the virtual camera of the switching virtual viewpoint image can be more easily grasped.

[0061] FIG. 14 is a diagram showing a display screen displayed in the process of switching between the selected switch virtual viewpoint image and the operation virtual viewpoint image.

[0062] In the example of FIG. 14(a), the first switching virtual viewpoint image from the left is selected by the user among the switching virtual viewpoint images displayed in the switching virtual viewpoint display area 502, and is surrounded by a thick frame indicating the selected state. When any of the switching virtual viewpoint images is in the selected state, a selection marker 1401 is superimposed and displayed at the position of the three-dimensional model of the subject to be tracked in the selected switching virtual viewpoint image in the operation virtual viewpoint display area 501. The selection marker 1401 is displayed at the position of the feet (z=0 m) of the player 402b based on the tracked position information. Here, the suffix a of the player 402 indicates the following viewpoint displayed in the operation virtual viewpoint display area 501, and b indicates the player displayed in the selected switching virtual viewpoint display area 502.

[0063] Fig. 14(a) shows the result of a user performing an operation to switch the selected switching virtual viewpoint to an operation virtual viewpoint via the input unit 107. The switching virtual viewpoint that was selected in Fig. 14(a) is switched to the operation virtual viewpoint, and is displayed in the operation virtual viewpoint display area 501. When the operation virtual viewpoint is switched, a new following viewpoint to be displayed in the switching virtual viewpoint display area is set by the display viewpoint control unit 901, and the following viewpoints are sorted in the order set by the display order control unit 902. Furthermore, the selection marker 1401 displayed in the operation virtual viewpoint image and the thick frame displayed in the switching virtual viewpoint image, which were displayed in Fig. 14(a), are released.

[0064] In this embodiment, a switching virtual viewpoint image that can be switched at any timing is displayed as a candidate for an operation virtual viewpoint from which a user performs an operation. Among the multiple virtual viewpoint images, multiple switching virtual viewpoint images are determined based on the relationship with the three-dimensional model being tracked in the operation virtual viewpoint image. Then, the display order of the switching virtual viewpoint images is determined based on the position information of the three-dimensional model of the subject to be tracked in each virtual viewpoint image. These operation virtual viewpoint images and switching virtual viewpoint images are tracking viewpoints that include the same object of interest, so that the position and attitude of the switchable virtual camera can be easily grasped.

[0065] In this embodiment, when a switching virtual viewpoint image displayed in the switching virtual viewpoint display area 502 is selected, the corresponding virtual viewpoint is set as the operation virtual viewpoint. In other words, the selected switching virtual viewpoint image is replaced with the operation virtual viewpoint image. However, without being limited to this, the selected switching virtual viewpoint image may be enlarged and displayed as the operation virtual viewpoint image. In that case, the switching virtual viewpoint image enlarged and displayed as the operation virtual viewpoint image is highlighted. This modified example is effective when the number of following viewpoints is small.

[0066] In this embodiment, each virtual viewpoint image is an image seen from a tracking viewpoint that tracks a specific subject. Therefore, in order to easily grasp the subject corresponding to each virtual viewpoint image, the corresponding subject may be displayed in each virtual viewpoint image so as to be distinguishable from other subjects. For example, the subject corresponding to the virtual viewpoint image may be highlighted by changing the color or displaying a circular icon at the position of the subject. In this case, a highlight different from the selection marker in FIG. 14 is used. [Explanation of symbols]

[0067] 102 3D model generation unit 103 3D model tracking unit 105 Storage Unit 107 Virtual viewpoint generation unit 111 Display control unit

Claims

1. An acquisition means for acquiring three or more virtual viewpoint images including the same subject; a display control means for controlling display of a specific virtual viewpoint image, among the three or more virtual viewpoint images displayed in the first display area, in a second display area larger than the first display area based on a user operation; 13. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the three or more virtual viewpoint images correspond to different subjects different from the same subject.

3. The image processing device according to claim 2 , wherein each of the three or more virtual viewpoint images includes a corresponding subject and the same subject.

4. 3. The image processing apparatus according to claim 2, wherein the three or more virtual viewpoint images each display a corresponding subject in a manner that allows discrimination between the corresponding subject and other subjects.

5. The image processing apparatus according to claim 1 , wherein the acquisition unit acquires a user operation for selecting the specific virtual viewpoint image from among the three or more virtual viewpoint images.

6. The image processing device according to claim 1 , wherein the first display area and the second display area are included in different display devices.

7. The image processing device according to claim 1 , wherein the first display area and the second display area are included in the same display device.

8. The image processing device described in claim 1, characterized in that when a first user operation is acquired, the display control means controls to highlight the specific virtual viewpoint image displayed in the first display area or a subject corresponding to the specific virtual viewpoint image included in the virtual viewpoint image displayed in the second display area, and when a second user operation is acquired, the display control means controls to display the specific virtual viewpoint image in the second display area.

9. The image processing device according to claim 8, characterized in that the first user operation is an operation of selecting a specific virtual viewpoint image from the three or more virtual viewpoint images displayed in the first display area, or an operation of selecting a subject corresponding to the specific virtual viewpoint image included in the virtual viewpoint images displayed in the second display area.

10. The image processing device according to claim 8, characterized in that the second user operation is an operation of selecting the specific virtual viewpoint image highlighted and displayed in the first display area, or an operation of selecting a highlighted subject corresponding to the specific virtual viewpoint image included in the virtual viewpoint image displayed in the second display area.

11. 2. The image processing apparatus according to claim 1, wherein the same subject is a stationary object.

12. The image processing device according to claim 11 , wherein the same subject is a goal.

13. An acquisition step of acquiring three or more virtual viewpoint images including the same subject; a display control step of controlling the display of a specific virtual viewpoint image, among the three or more virtual viewpoint images displayed in the first display area, in a second display area larger than the first display area based on a user operation; 13. An image processing method comprising:

14. A program for causing a computer to function as each of the means of the image processing device according to any one of claims 1 to 12.

Images