Image processing device, image processing method and program

Abstract

To enable generation of a virtual viewpoint image further suitable for a viewing purpose.SOLUTION: An image processing device (100) includes: obtaining means (202) for obtaining visual line information relating to a moving range of the visual line of an object imaged by a plurality of imaging devices; determining means (203) for determining virtual viewpoint information representing a virtual viewpoint relating to a virtual viewpoint image which is on the basis of a plurality of photographed images obtained by photographing carried out by the plurality of imaging devices, and includes the moving range of the visual line of the object identified on the basis of the visual line information obtained by the obtaining means (202); and output means (203) for outputting the virtual viewpoint information determined by the determining means (203).SELECTED DRAWING: Figure 2

Description

[Technical field]

[0001] The present invention relates to an image processing technique for generating a virtual viewpoint image. [Background technology]

[0002] In recent years, a technology that generates an image seen from an arbitrary viewpoint, that is, a virtual viewpoint image equivalent to an image captured by a virtual camera, based on multiple viewpoint images obtained from multiple cameras installed at different positions and capturing images synchronously has been attracting attention. A virtual viewpoint image is generated by collecting multiple viewpoint images captured by multiple cameras in an image processing device such as a server, and performing processes such as foreground / background separation, 3D shape estimation, 3D model generation, and rendering in the image processing device.

[0003] The virtual viewpoint image is then transferred to the user terminal and displayed, allowing the user (viewer) to view the virtual viewpoint image (virtual viewpoint video). This technology allows the user to view highlight scenes of a soccer or basketball game from various positions and angles that were previously impossible to achieve, such as on the field during the game, and provides a higher sense of realism than conventional video. Furthermore, for example, by acquiring information on the gaze of players and referees and using the gaze information to generate and display a virtual viewpoint image, gaze analysis for improving the skills of players and referees becomes possible. Furthermore, with a virtual viewpoint image based on the gaze information of players and referees, the user (viewer) can virtually experience how and where the players and referees were looking during the game, and through this realistic experience, the user can enjoy the game even more. Patent Document 1 discloses a technology in which, in an equipment operation device capable of receiving operation inputs by gaze, a gaze position correction instruction is given by manual input, and the average value of the gaze position error of each correction instruction is reflected in gaze detection. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2019-152957 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, the gazes of players and referees may move quickly and widely up and down and left and right depending on the situation of the game. A virtual viewpoint image generated based on information on gazes that move violently up and down and left and right results in an image with violent movement and blur that cannot be imagined with normal camera work, making the video very difficult for users to watch. Such virtual viewpoint images may be unsuitable for viewing purposes, such as making users more susceptible to motion sickness. Even if the technology disclosed in Patent Document 1 is used, it is difficult to suppress the movement and blur of an image caused by gazes that move violently up and down and left and right.

[0006] Therefore, an object of the present invention is to make it possible to generate a virtual viewpoint image that is more suitable for viewing purposes. [Means for solving the problem]

[0007] The image processing device of the present invention is characterized in having an acquisition means for acquiring gaze information relating to the range of movement of the gaze of a subject photographed by a plurality of photographing devices, a determination means for determining virtual viewpoint information representing a virtual viewpoint related to the virtual viewpoint image based on a plurality of photographed images acquired by photographing performed by the plurality of photographing devices, the virtual viewpoint image including the range of movement of the gaze of the subject identified based on the gaze information acquired by the acquisition means, and an output means for outputting the virtual viewpoint information determined by the determination means. Effect of the Invention

[0008] According to the present invention, it is possible to generate a virtual viewpoint image that is more suitable for viewing purposes. [Brief description of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a hardware configuration of an image processing device. [Diagram 2] 1 is a functional block diagram of an image processing device according to a first embodiment. [Diagram 3] 4 is a flowchart of a virtual viewpoint image generating process in the first embodiment. [Figure 4] FIG. 1 is a diagram illustrating a scene from a soccer game. [Diagram 5] FIG. 13 is an explanatory diagram for estimating the gaze range of the referee. [Figure 6] 11 is an explanatory diagram of virtual camera parameter complementation when generating a virtual viewpoint image. FIG. [Figure 7] FIG. 11 is a functional block diagram of an image processing device according to a second embodiment. [Figure 8] 10 is a flowchart of a virtual viewpoint image generating process in the second embodiment. [Figure 9] 11A and 11B are diagrams illustrating examples of display of objects related to line of sight. [Figure 10] FIG. 13 is a diagram showing an example of a graphics display of a line-of-sight range. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, an embodiment of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the configurations shown in the drawings. In addition, the same configurations or processes in the embodiments will be described with the same reference numerals.

[0011] First Embodiment FIG. 1 is a diagram showing a schematic diagram of a hardware configuration of an image processing device 100 according to the present embodiment. The image processing device 100 includes a CPU 101, a main memory device 102, an auxiliary memory device 103, an external I / F 104, a GPU 105, and a bus 106. Each part of this hardware configuration is connected to each other so that they can communicate with each other via the bus 106. The bus 106 is configured by a general-purpose bus such as PCI Express (registered trademark).

[0012] The CPU 101 is a central processing unit, and performs overall control of the image processing device 100, such as various calculations and decisions, control of image processing, and control of data input / output. The main memory device 102 is a storage device that functions as a work area and a temporary data storage area for the CPU 101. The main memory device 102 is implemented using a storage medium such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0013] The auxiliary storage device 103 is a storage device that stores information such as various programs, various setting information, various image data, camera parameters, 3D shape data, and 2D maps, and is implemented using various storage media. Examples of storage media include non-volatile memory such as a read only memory (ROM) or a flash memory, a hard disk drive (HDD), a solid state drive (SSD), and tape media. Note that a plurality of these storage media may be combined to increase capacity and speed, or may be logically grouped together as a single medium, such as a redundant array of inexpensive disks (RAID).

[0014] The external I / F (Interface) 104 includes a communication interface and an input / output interface. The communication interface is used for communication with external devices such as a camera, a PC (personal computer), and a server. The input / output interface is used for connection to external devices that accept various operations by a user, such as a joystick, a keyboard, a mouse, and a touch panel, and for connecting to external storage and performing file input / output. The external I / F 104 is implemented as an interface having a physical cable connection terminal, or a wireless interface. Examples of interfaces having a physical cable connection terminal include InfiniBand, Ethernet, and USB (Universal Serial Bus). Examples of wireless interfaces include wireless LAN and Bluetooth (registered trademark).

[0015] The GPU 105 is a graphics processing unit for performing image processing calculations at high speed, and has a function of outputting a video signal to an external video display device such as a television. In this embodiment, the image processing that the GPU 105 can perform includes a process of generating a virtual viewpoint image. The virtual viewpoint image is generated by performing processes such as foreground / background separation, estimation of a three-dimensional shape, generation of a three-dimensional model, and rendering using multiple viewpoint images obtained by synchronously capturing images from multiple image capturing devices (cameras not shown in FIG. 1) installed at different positions. The virtual viewpoint image is an image generated based on the position of a virtual viewpoint, which is a specified arbitrary viewpoint, and the line of sight direction from the virtual viewpoint. In addition, for convenience of explanation, the following description will be given using a virtual camera (virtual camera) corresponding to the virtual viewpoint. That is, the position of the virtual camera and the attitude (imaging direction) of the virtual camera correspond to the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint, respectively. In addition, information indicating the position and attitude of the virtual camera is expressed as virtual camera information. The virtual camera information includes information indicating the position of the virtual camera and information indicating the attitude of the virtual camera. The virtual camera information may further include information corresponding to the width of the field of view (angle of view) of the virtual viewpoint. In other words, the virtual viewpoint image is an image generated based on the virtual camera information. The virtual camera information may include at least one of information on the position, the attitude, and the angle of view. The image processing executed by the GPU 105 includes not only the processing of rendering an image seen from a virtual viewpoint, but also the processing of generating shape data and color information required for generating a virtual viewpoint image. Moreover, data of multiple viewpoint images taken by multiple cameras installed at different positions is acquired, for example, via the external I / F 104, and is stored, for example, in the auxiliary storage device 103. In this embodiment, the virtual camera is a virtual camera that is virtually arranged corresponding to an arbitrary viewpoint in the virtual space, and is distinguished from a real camera that is arranged at a different position in the real space and captures a multi-viewpoint image. The virtual camera information is information generated by the CPU 101 of this embodiment, and the details thereof will be described later.

[0016] The video signal output of the GPU 105 is implemented by a video terminal such as DVI (Digital Visual Interface), HDMI (registered trademark), Display Port, or SDI (Serial Digital Interface). The GPU 105 may be configured not to have a function of outputting a video signal, and to perform only calculations such as shape data and color information required for image processing and virtual viewpoint images. In this case, for example, the external I / F 104 may be in charge of a video output function or a data output function required for video output. The GPU 105 may also have a function of executing part or all of the various calculations performed by the CPU 101 other than those described above. The GPU 105 may also include a separate storage device different from the main storage device 102.

[0017] The image processing device 100 of this embodiment is not limited to the configuration shown in FIG. 1, and may be realized by another hardware configuration within the scope of the present invention. For example, the auxiliary storage device 103 may be configured as an external configuration of the image processing device 100, and in that case, the auxiliary storage device may be connected by an external I / F 104. The main storage device 102 and the auxiliary storage device 103 may be realized as one piece of hardware. Instead of the GPU 105, an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), an image processing LIS (Large Scale Integrated Circuit), or the like may be used. The image processing device 100 is not limited to a single device, and may be configured by, for example, multiple devices having similar configurations, and each of the devices may have a different configuration within the scope of the present invention.

[0018] Hereinafter, a process of generating a virtual viewpoint image performed by the image processing device 100 according to the present embodiment will be described. The image processing device 100 according to the present embodiment has a function of performing an acquisition process of acquiring line-of-sight information, a determination process of determining virtual viewpoint information, and a generation process of generating a virtual viewpoint image. In the acquisition process, line-of-sight information is acquired, which is information for identifying the line of sight of a subject photographed by a plurality of photographing devices. The line-of-sight information includes, for example, information representing the line of sight of a subject such as a player or a referee. The image processing device 100 can identify the moving range of the line of sight of the subject based on the line-of-sight information. In the determination process, virtual viewpoint information representing a virtual viewpoint is determined for a virtual viewpoint image based on a plurality of photographed images acquired by photographing performed by a plurality of photographing devices, and including the moving range of the line of sight of the subject identified based on the line-of-sight information acquired in the acquisition process. In the generation process, a virtual viewpoint image is generated based on the virtual viewpoint information determined by the determination process. The image processing device 100 generates a virtual viewpoint image, which is an image virtually photographed by a virtual camera represented by virtual camera information including the position, attitude, angle of view, etc. of the virtual camera installed in the virtual space, based on line-of-sight information relating to the moving range of the line of sight of a person in the virtual space. In the following description, for the sake of simplicity, the range within which the subject's (person's) line of sight moves will be referred to as the "line of sight range."

[0019] In this embodiment, for example, in the case of generating a virtual viewpoint image based on multiple viewpoint images obtained by synchronously shooting a soccer game with multiple real cameras at different positions, the gaze information of the players and the referee is acquired, and the virtual viewpoint image is generated using the gaze information. In this case, as described above, the gaze of the players and the referee moves quickly and widely up, down, left, right, etc., and the virtual viewpoint image generated using the gaze information may become an image with intense movement or blur. In the case of the technology described in Patent Document 1 described above, the average value of the error of each gaze position is reflected in the gaze detection by manual input such as button operation, but in the case of a soccer game, there is no area to be gazed at like when operating a button. In addition, the gaze of the players, the referee, etc., that moves violently is not blurred with respect to the gaze position, but is gazed at many places in a short time. For this reason, if the technology of Patent Document 1 is applied and the intense movement of the gaze is corrected as blur, the direction of the gaze of the players and the referee becomes unclear, and the original purpose of the user virtually watching the field of view of the players and the referee cannot be realized. Note that the user in this embodiment may not only be a viewer who virtually experiences the field of view of a player or a referee during a match, but may also be a player or a referee who watches a virtual viewpoint image with the aim of improving their skills.

[0020] Therefore, the image processing device 100 of this embodiment has the functions shown in Fig. 2, and by performing the process shown in the flowchart of Fig. 3 described later, it is possible to generate a virtual viewpoint image with reduced sudden vertical and horizontal movement and blurring. The image processing device 100 of this embodiment acquires gaze information related to the movement range of the gaze of a person in a virtual space, and sets virtual camera information of a virtual camera corresponding to the virtual viewpoint information related to a virtual viewpoint image including the movement range of the gaze of the person specified based on the gaze information. Then, the image processing device 100 generates a virtual viewpoint image corresponding to an image captured by the virtual camera for which the virtual camera information has been set.

[0021] Fig. 2 is a functional block diagram showing a functional configuration of the image processing device 100 according to the first embodiment. As shown in Fig. 2, the image processing device 100 has functional units, namely, an input unit 201, a line of sight acquisition unit 202, an estimation unit 203, a data storage unit 204, an image generation unit 205, and a display unit 206. Note that the functional units shown in Fig. 2 and the processing shown in the flowchart of Fig. 3 described later are realized, for example, by the CPU 101 shown in Fig. 1 executing processing according to a program according to this embodiment stored in the auxiliary storage device 103.

[0022] The input unit 201 includes an external I / F 104, acquires operation information by a user, and acquires inputs related to various instructions from the user based on the operation information. Examples of user inputs based on operation information include selection of virtual viewpoint image content, video operations such as play / stop, fast forward, and cueing by specifying a time code, and camera steering operations. Furthermore, in this embodiment, user inputs based on operation information include selection of a desired person such as a player or a referee for whom a virtual viewpoint image using gaze information is to be generated (person designation), and an instruction for an update rate for changing the time range in which the gaze range is estimated, which will be described later.

[0023] The line of sight acquisition unit 202 acquires line of sight information of a person who may be a target for generating a virtual viewpoint image in a virtual space, that is, in the example of this embodiment, acquires line of sight information of a person (a player or a referee) on a soccer field. For example, the line of sight acquisition unit 202 acquires line of sight information of a player or a referee from output information of a sensor worn by the player or the referee during a match, or by image analysis of a multi-viewpoint image. In addition, when line of sight information is acquired in advance and stored in the auxiliary storage device 103 or an external medium, the line of sight acquisition unit 202 may acquire line of sight information from a medium or a network connected to the auxiliary storage device 103 or the external I / F 104. Note that in this embodiment, the line of sight acquisition unit 202 acquires line of sight information for each unit time, such as a frame of a video.

[0024] The estimation unit 203 sets virtual camera information for determining the range of shooting by the virtual camera installed in the virtual space. In the present embodiment, the estimation unit 203 acquires user input from the input unit 201, and also acquires gaze information from the gaze acquisition unit 202. Based on the user input from the input unit 201, the estimation unit 203 identifies a person such as a player or a referee selected (designated) by the user as a target for generating a virtual viewpoint image using the gaze information. Then, the estimation unit 203 estimates the movement range of the gaze (hereinafter also referred to as the gaze range) of the identified person within a predetermined time range (within a predetermined period) set in advance, and further sets virtual camera information that includes the gaze range estimated from the gaze position of the person. The virtual camera information is a camera parameter (referred to as a virtual camera parameter) of a virtual camera that virtually shoots a virtual space, and includes at least parameters for setting the position, attitude, shooting direction, and angle of view of the virtual camera.

[0025] Although a detailed description will be given later, the estimation unit 203 at this time acquires the trajectory of the gaze of the person specified in response to the user input, based on gaze information acquired by the gaze acquisition unit 202 for each unit time such as a frame period. Furthermore, the estimation unit 203 estimates a gaze range, which is the range of movement of the gaze of the person within a predetermined time range, based on the gaze trajectory within the predetermined time range among the gaze trajectories of the person. Then, the estimation unit 203 sets virtual camera information (virtual camera parameters) that includes the multiple gaze ranges estimated from the gaze position of the person. Note that the predetermined time range may be determined in advance, or may be specified as an arbitrary time range by the user. These processes in the estimation unit 203 are performed by, for example, the CPU 101 in the configuration of FIG. 1.

[0026] The data storage unit 204 stores data consisting of three-dimensional model data, color information, etc. in the auxiliary storage device 103, etc. In this embodiment, the image generation unit 205 generates a virtual viewpoint image corresponding to an image captured by the virtual camera of the virtual space based on the virtual camera parameters set by the estimation unit 203 based on the line of sight range. At this time, the image generation unit 205 acquires data consisting of three-dimensional model data, color information, etc. from the data storage unit 204, and generates a virtual viewpoint image that is an image virtually captured by the virtual camera based on the data. Note that the generation process of the virtual viewpoint image in the image generation unit 205 is performed by, for example, the GPU 105 in the configuration of FIG. 1.

[0027] The display unit 206 displays the virtual viewpoint image generated by the image generation unit 205 on a television receiver, a monitor of a personal computer, a tablet terminal, or the like via the external I / F 104. This allows the user to view the virtual viewpoint image.

[0028] 3 is a flowchart showing a process flow for calculating a line-of-sight range and generating a virtual viewpoint image in the image processing device 100 of the first embodiment. Hereinafter, with reference to this flowchart, the process flow for the image processing device 100 to generate a virtual viewpoint image according to, for example, the line of sight of a referee will be described using a soccer game as an example.

[0029] In step S301, the input unit 201 acquires an update rate of the time range when estimating the gaze range from input information by a user's operation. Then, information on the update rate of the time range is sent to the estimation unit 203. In this embodiment, the time range when estimating the gaze range is set to, for example, one second.

[0030] In the next step S302, the input unit 201 acquires the playback start time of the virtual viewpoint image content designated by the user from the input information by the user's operation. This information on the playback start time of the virtual viewpoint image content is sent to the image generation unit 205 via the estimation unit 203. For example, when the user instructs to cue the virtual viewpoint image content, the image generation unit 205 plays back the virtual viewpoint image content from its beginning time.

[0031] In the next step S303, the input unit 201 acquires information indicating a person designated (selected) as a target for gaze extraction when generating a virtual viewpoint image using gaze information, from input information input by a user operation. This designated information of the target person for gaze extraction is sent to the estimation unit 203.

[0032] FIG. 4 is a diagram that diagrammatically illustrates a scene from a soccer game. When the target of gaze extraction is designated in step S303, a screen showing a scene such as that shown in FIG. 4 is displayed by the display unit 206, and the user can select a person on the screen using a pointing device such as a mouse or a touch panel. Here, it is assumed that the user designates the referee 401 as the person for whom gaze extraction is to be performed. It is assumed that the referee 401 at this time is facing in the direction of the ball 405 (the direction indicated by the arrow 402). Note that in this example, the referee 401 on the field is designated, but the person for whom gaze extraction is to be performed is not limited to the referee. For example, a player 403 or a goalkeeper 404 may be designated as the person for whom gaze extraction is to be performed.

[0033] In the next step S304, the estimation unit 203 extracts the world coordinates of the eyebrows of the person (the referee 401 in the example of FIG. 4) from the gaze extraction target by face detection processing or the like, and sets them as external parameters of the virtual camera. In the present embodiment, the estimation unit 203 sets the world coordinates of the eyebrows of the person as the virtual camera parameters representing the position of the virtual camera. Here, the world coordinates of the eyebrows are set as the virtual camera parameters representing the position of the virtual camera, but this is not limiting. For example, the positions of the left and right eyeballs may be detected, and the positions of either eyeball or both eyeballs may be set as the virtual camera parameters representing the position of the virtual camera. For example, when the positions of both eyeballs are used, it is possible to generate two images having parallax as the virtual viewpoint image. In addition, for example, the center of the head or the position near the top of the head may be used. Note that the height position of the virtual camera may be set to a specific value (height) for each player or referee, or the same specific value (height) may be set for all players and referees.

[0034] Next, in step S305, estimation unit 203 estimates the gaze range by extracting a gaze trajectory within the time range (one second) specified in step S301 from the gaze information of the person specified in step S303 (referee 401 in the example of FIG. 4). Then, estimation unit 203 obtains, from the estimated gaze range, virtual camera information (virtual camera parameters) other than the virtual camera position obtained in step S304 when virtual shooting is performed at the virtual camera position.

[0035] Figures 5(a) to 5(e) are schematic diagrams showing how the gaze range of referee 401 shown in Figure 4 is estimated and virtual camera parameters are calculated based on that gaze range. The dotted lines in Figures 5(a) to 5(e) represent the trajectory of the movement of the centre of referee 401's gaze over time.

[0036] 5(a) shows the line of sight 501 when the referee 401 moves his / her gaze from gaze position 501a as shown by the dotted line over time, and two seconds later moves his / her gaze to gaze position 501b. For convenience of illustration on paper, each person such as a player is depicted as stationary, but in reality, each person makes arbitrary movements over time.

[0037] Fig. 5(b) shows an estimated gaze range within the field of view of referee 401 at time t1 in an example in which referee 401's gaze moves as shown by trajectory 501 from gaze position 501a to gaze position 501b as shown in Fig. 5(a). Trajectory 502 shown in Fig. 5(b) is the gaze trajectory acquired at time t1, and shows the gaze trajectory during one second of the time range set in step S301. Here, the one second period from time t1-1 to time t1 is the time range for estimating the gaze, and gaze position 501a represents the center position of referee 401's gaze at time t1-1. Also, gaze range 503, indicated by a solid line frame in Fig. 5(b), shows the gaze range estimated within referee 401's field of view at gaze position 501a. At this time, the gaze range 503 estimated with respect to the center of gaze within the field of vision of the referee 401 is set to the range of the effective visual field excluding the range known as peripheral vision, and is set to an average angle of view.

[0038] Figure 5(c) shows how the range of the effective visual field is acquired in sequence along trajectory 502 in the same manner as described above. Gaze range 505a is an estimated range of the effective visual field of referee 401 when the central position of referee 401's gaze moves from gaze position 501a to gaze position 504a over time. Similarly, gaze range 505b is an estimated range of the effective visual field of referee 401 when the center of gaze moves from gaze position 504a to gaze position 504b, and gaze range 505c is an estimated range of the effective visual field of referee 401 when the center of gaze moves from gaze position 504b to gaze position 504c.

[0039] In this way, estimation unit 203 estimates the gaze range sequentially along the gaze trajectory for one second from time t1-1 to time t. Then, estimation unit 203 acquires, as included gaze range 506 at time t, a range that includes all the angles of view of the effective visual field for each gaze range estimated along the gaze trajectory for one second from time t1-1 to time t, as shown in Fig. 5(d).

[0040] Furthermore, estimation unit 203 estimates virtual camera parameters other than the position of the virtual camera at time t based on the angle of view corresponding to inclusive gaze range 506 at time t. Inclusive gaze range 506 does not need to have a shape in which each gaze range estimated within the time range from time t1-1 to time t is inscribed, and may have any shape as long as each gaze range is included inside. In addition, the virtual camera parameters may correspond to a shape in which the aspect ratio of a general video signal is 16:9 or 4:3, or an aspect ratio such as 3:2 used in photographs. Alternatively, the inclusive gaze range may be set to be the narrowest as long as it includes all the gaze ranges in those aspect ratios.

[0041] As in the case of inclusive gaze range 506 described in FIG. 5(d), estimation unit 203 determines the inclusive gaze range for one second from time t2-1 (i.e., time t1) to time t2, with time t2 being the time range for estimating the gaze after time t1. Inclusive gaze range 507 shown in FIG. 5(e) indicates the inclusive gaze range determined for one second from time t2-1 to time t2. Then, estimation unit 203 estimates virtual camera parameters other than the position of the virtual camera at time t2 based on the angle of view corresponding to inclusive gaze range 507 at time t2. Similarly, estimation unit 203 determines the inclusive gaze range and sets the virtual camera parameters every time the time range is updated, that is, at each update rate.

[0042] Returning to the explanation of the flowchart in Fig. 3, in the next step S306, the image generation unit 205 renders a virtual viewpoint image using the virtual camera parameters corresponding to the inclusive line-of-sight ranges at time t1 and time t2 received from the estimation unit 203. Then, the display unit 206 displays the virtual viewpoint image.

[0043] At this time, the information received by the image generating unit 205 is a combination of time information such as time t1 and time t2 and virtual camera parameters corresponding to the inclusive line of sight range linked to the time information. Since the virtual viewpoint image is composed of 60 frames per second, if the time interval between time t1 and time t2 is 1 second, the image generating unit 205 needs to generate images for 60 frames as the virtual viewpoint image between time t1 and time t2.

[0044] Therefore, the image generating unit 205 gradually changes the virtual camera information between the virtual camera information set in a predetermined time range at time t1 and the second virtual camera information set in a time range at time t2 that is different from time t1 by the time of the update rate of the time range. For example, the image generating unit 205 acquires virtual camera information that is complemented for each time that is divided by the number of frames when displaying the virtual viewpoint image, using the first virtual camera information and the second virtual camera information as virtual camera information of key frames and divides the predetermined time range by the number of frames when displaying the virtual viewpoint image. In the case of this embodiment, the image generating unit 205 generates intermediate data that complements the time between the virtual camera parameters at time t1 and the virtual camera parameters at time t2 using the virtual camera parameters at time t1 and the virtual camera parameters at time t2 as key frames. The intermediate data is generated as data that gradually changes from the virtual camera parameters at time t1 to the virtual camera parameters at time t2. In addition, in this embodiment, the image generating unit 205 gradually changes the angle of view of the virtual camera parameters set by the estimation unit 203, especially when the angle of view changes to a narrow angle. As a result, the image generation unit 205 generates virtual viewpoint images based on virtual camera parameters that gradually change over the one second from time t1 to time t2, that is, 60 frames of virtual viewpoint images in which the images gradually change between time t1 and time t2.

[0045] 6(a) to 6(d) are diagrams showing how a virtual viewpoint image is generated while complementing virtual camera parameters for each frame between the inclusive line-of-sight ranges at times t1 and t2 obtained by the estimation unit 203 as described above. Note that, in the examples of Fig. 6(a) to 6(d), one second is divided into four periods due to space limitations, but in reality, virtual viewpoint images equivalent to 60 frames are generated.

[0046] Fig. 6(a) shows the angle of view when a virtual viewpoint image is generated at time t1. At time t1, a virtual viewpoint image is generated corresponding to angle of view 601 equivalent to inclusive line-of-sight range 506 shown in Fig. 5(d). Fig. 6(a) also shows inclusive line-of-sight range 507 calculated in Fig. 5(e) as angle of view 602 indicated by a dotted line as the angle of view when a virtual viewpoint image is generated at time t2.

[0047] 6B is a diagram showing the angle of view 603 at time t1+Δ1 after time Δ1, which is a time shorter than one second of the time range of gaze estimation, has elapsed from time t1. This time t1+Δ1 is a time between time t1 and time t2. Since the virtual camera parameters corresponding to time t1+Δ1 between time t1 and time t2 are not estimated by the estimation unit 203, the image generation unit 205 cannot acquire the virtual camera parameters corresponding to that time t1+Δ1. Therefore, the virtual camera parameters corresponding to time t1+Δ1 are complemented as intermediate data using the virtual camera parameters at time t1 and the virtual camera parameters at time t2 as key frames.

[0048] Here, the angle of view 603 at time t1+Δ1 is set as an intermediate angle of view between the angle of view 601 at time t1 and the angle of view 602 at time t2. In this embodiment, the image generating unit 205 divides the value representing the angle of view 601 at time t1 and the value representing the angle of view 602 at time t2 so that the value is linearly and evenly divided by the number of frames for generating a virtual viewpoint image. Furthermore, the image generating unit 205 sets the angle of view represented by the value corresponding to time t1+Δ1 among the values ​​representing each divided angle of view as the angle of view 603 at the time t1+Δ1. Then, the image generating unit 205 generates a virtual viewpoint image based on the virtual camera parameters corresponding to the angle of view 603. Alternatively, the image generating unit 205 may divide the two values ​​representing the angle of view 601 at time t1 and the angle of view 602 at time t2 by using interpolation such as curvilinear acceleration or deceleration to draw a Bezier curve or a sine curve. Furthermore, the present invention is not limited to these, and any complementation method may be used as long as it is possible to acquire complemented virtual parameters that are intermediate values ​​between the virtual camera parameters at time t1 and the virtual camera parameters at time t2.

[0049] Fig. 6(c) is a diagram showing an angle of view 604 at time t1+Δ2, which is a further time that has passed since time t1+Δ1. At time t1+Δ2 in Fig. 6(c), a similar interpolation method as described above is used to obtain an angle of view 604 that is closer to the angle of view 602 at time t2 than the angle of view 603 at time t1+Δ1.

[0050] After that, at time t2, the angle of view becomes 602 as shown in Fig. 6(d). In the image generating unit 205, as described in Fig. 6(a) to Fig. 6(d), the virtual camera parameters are complemented between time t1 and time t2, and a virtual viewpoint image is generated based on these virtual parameters.

[0051] As described above, the image processing device 100 of the first embodiment generates a virtual viewpoint image by obtaining virtual camera parameters from a gaze range estimated based on the gaze trajectory for each predetermined time range. As a result, when a virtual viewpoint image corresponding to the gaze of a person from whom gaze extraction is to be performed is generated, it is possible to generate a virtual viewpoint image that is presumed to have been viewed by that person while suppressing rapid image movement.

[0052] In the first embodiment, for example, when the inclusive gaze range of time t1 is obtained, it is estimated based on gaze information before time t1 according to the time range, but this is not limited thereto, and the inclusive gaze range may be obtained based on gaze information after time t1. Also, when estimating within a time range of 1 second including and before time t1, for example, as described above, any method may be used as long as the gaze estimation range is obtained based on gaze information for consecutive 1 second periods, such as 0.5 seconds before and after time t1.

[0053] In the present embodiment, the angle of view according to the included line of sight range may be the narrowest angle of view according to the aspect ratio of the output image, including the line of sight trajectory detected among the viewing angles, or may be an angle of view that is expanded by a certain range from the narrowest angle of view, for example, by a user specification.

[0054] In the present embodiment, the line of sight range is set to an average angle of view as the effective visual field, excluding a range called peripheral visual field, but is not limited to this. For example, the line of sight range may be calculated using a narrower angle of view, such as using an angle considered to be the visual field angle of the fovea for the line of sight information. In addition, the effective visual field may be an angle of view including any angle of view set by the user, and any range including the center of the line of sight may be set as the effective visual field. In the present embodiment, the image processing device 100 has been described as having the image generation unit 205 and the display unit 206, but is not limited thereto. For example, at least one of the image generation unit 205 and the display unit 206 may be configured to be connected to the outside of the image processing device 100 as another device. In addition, when both the image generation unit 205 and the display unit 206 are devices connected to the outside of the image processing device 100, the image generation unit 205 and the display unit 206 may be the same device or different devices. That is, the image processing device 100 at least performs a process of determining virtual camera parameters in the estimation unit 203 and outputting the virtual camera parameters to a processing unit that generates a virtual viewpoint image or another device.

[0055] <Second embodiment> Fig. 7 is a functional block diagram showing the functional configuration of an image processing device 700 according to the second embodiment. In the image processing device 700, the input unit 201 to the display unit 206 are similar to the corresponding functional units of the image processing device 100 shown in Fig. 2, and therefore their description will be omitted. Note that the hardware configuration of the image processing device 700 according to the second embodiment is similar to the configuration shown in Fig. 1, and therefore illustration and description thereof will be omitted.

[0056] The graphics generation unit 701 acquires the gaze information acquired by the gaze acquisition unit 202, information on the included gaze range estimated by the estimation unit 203, and virtual camera parameters when the image generation unit 205 generates the virtual viewpoint image. Based on each of these pieces of information, the graphics generation unit 701 generates graphics data for displaying, for example, the gaze and its trajectory, the gaze point at which the person who is the subject of gaze extraction is gazing, or the gaze range, and sends the generated graphics data to the display unit 206. As a result, the display unit 206 performs a graphics display of the gaze and its trajectory, the gaze point, the gaze range, etc.

[0057] Fig. 8 is a flowchart showing a process flow for calculating a line-of-sight range, generating a virtual viewpoint image, and displaying graphics in the image processing device 700 of the second embodiment. In Fig. 8, the processes from step S301 to step S306 are similar to the corresponding steps in Fig. 3, and therefore their description will be omitted. In the flowchart of Fig. 8, after step S306, the process of the image processing device 700 proceeds to step 801.

[0058] In step S801, the graphics generation unit 701 acquires information on the time and virtual camera parameters of the virtual viewpoint image generated by the image generation unit 205, the gaze information acquired by the gaze acquisition unit 202, and the included gaze range estimated by the estimation unit 203. Then, the graphics generation unit 701 generates graphics data to be displayed on the display unit 206 based on the information. That is, the graphics generation unit 701 performs a related image generation process to generate an object as a related image related to the gaze of, for example, the referee 401 who is a person to be subjected to gaze extraction. The display unit 206 displays the object generated by the graphics generation unit 701 based on the graphics data in a superimposed manner on the virtual viewpoint image generated by the image generation unit 205. Note that, in this embodiment, the graphics data and the virtual viewpoint image are displayed in a superimposed manner on the display unit 206, but this is not limited thereto. For example, the image generation unit 205 may perform a process of superimposing the graphics data and the virtual viewpoint image, and the display unit 206 may display the image generated by the image generation unit 205.

[0059] 9(a) to 9(c) are schematic diagrams showing how objects are superimposed as related images related to the gaze of referee 401, who is a person whose gaze is to be extracted, on a virtual viewpoint image 900. Note that Figs. 9(a), 9(b) and 9(c) show examples in which objects related to the gaze of referee 401 are displayed in different forms.

[0060] FIG. 9(a) shows an example in which a trajectory object 901 showing the trajectory of the gaze of the referee 401 is superimposed on a virtual viewpoint image 900 generated based on the virtual camera parameters of the inclusive gaze range at time t1. This allows the user to know the gaze range at which the referee 401 was gazing at a certain time within the inclusive gaze range estimated for the referee 401 while viewing the gaze range from above. At this time, the graphics generating unit 701 may display the gaze position (gaze position) at the time when the virtual viewpoint image is generated in an emphasized manner, as shown by a point object 902, for example. Also, for example, the graphics generating unit 701 may express the movement of the gaze by the point object 902 and display the trajectory object 901 as the trajectory. Also, the graphics generating unit 701 may display the trajectory object 901 so that the trajectory object 901 gradually disappears from the oldest part as time passes.

[0061] Fig. 9(b) shows an example in which a frame representing the gaze range of the referee 401 is superimposed on the virtual viewpoint image. In the example of Fig. 9(b), a range object 903 representing the gaze range 505c of the referee 401 at time t1 is superimposed on the virtual viewpoint image. In this way, the gaze range at each time is superimposed on the virtual viewpoint image as the range object 903, so that the user can know the gaze range of the referee 401 while viewing his field of vision from an overhead view.

[0062] FIG. 9(c) shows an example of a case where the field of view of the referee 401 is displayed on the virtual viewpoint image in a PiP (Picture In Picture) manner. FIG. 9(c) shows an example in which, for example, an image area of ​​the gaze range of the referee 401 at time t1 (for example, the gaze range 503 in FIG. 5(b)) is trimmed from the virtual viewpoint image and superimposed as a window object 904. The same effect as described above can be obtained by performing such a superimposed display by PiP. In this example, the image area of ​​the gaze range is trimmed from the virtual viewpoint image, but the present invention is not limited to this example. For example, the estimation unit 203 may separately generate virtual camera parameters that are equivalent to the angle of view of the gaze range of the referee 401, and the image generation unit 205 may generate a virtual viewpoint image superimposed in the same manner as the window object 904. Although the window object 904 is superimposed on the virtual viewpoint image in the above example, it may be displayed separately on a display device different from the display device that displays the virtual viewpoint image.

[0063] Figures 10(a) and 10(b) are schematic diagrams showing an example of a display method that makes it possible to present to a user the changes in the gaze of a person who is the subject of gaze extraction by changing the brightness of an image area representing the gaze range on a virtual viewpoint image in accordance with the gaze trajectory. The virtual viewpoint image 1001 is an image created to correspond to the line of sight from the position of the referee 401, and is an image created to have a wider angle than the viewing angle of an average human. For example, the virtual viewpoint image 1001 may be an image assumed to be taken with a super-wide-angle lens with a wide angle of view, an image assumed to be taken with a fisheye lens, or an image assumed to be taken with a lens with an angle of view that covers the entire surroundings, such as a 360-degree camera.

[0064] 10(a) and 10(b), trajectory 1002a represents the trajectory of the gaze from time t1-1 seconds to time t1, and trajectory 1002b represents the trajectory of the gaze from time t1 to time t2. Gaze range 1003a in Fig. 10(b) represents the gaze range estimated at time t1, and gaze range 1003b represents the gaze range estimated at time t2. Note that gaze range 1003a and gaze range 1003b are determined by the same method as that for inclusive gaze range 506 described above.

[0065] Here, assuming that the current time is time t2, the graphics generation unit 701 sets the luminance of the area of ​​the gaze range 1003b estimated at time t2 to a higher value (brighter) than the luminance of the area of ​​the gaze range 1003a at time t1, which is earlier than time t2. Furthermore, the graphics generation unit 701 sets the luminance value of the gaze range 1003a at time t1 to gradually lower (darker) as time passes. The time over which the luminance decreases may be the elapsed time in the real world, or may be the elapsed time of a virtual time based on the time code of the virtual viewpoint image. Furthermore, the graphics generation unit 701 sets the luminance of the area other than the gaze range 1003a and the gaze range 1003b, that is, the area where the gaze of the referee 401 is not directed, to be further lower (darker).

[0066] In the example of FIG. 10(b), the line of sight range 1003a and the line of sight range 1003b are circular regions including the line of sight locus, but are not limited to this example and may be any shape such as a rectangular region as shown in FIG. 5 or an elliptical region. The graphics generating unit 701 may change not only the luminance of the region indicating the line of sight range but also the shape as time passes. The graphics generating unit 701 may move a circular region or a rectangular region of a certain size indicating the line of sight range along the line of sight locus 1002a or the locus 1002b. In the above example, the luminance of the line of sight range 1003a at time t1 is darkened as a whole in the same manner within the region, but the luminance may be changed to have a gradation according to the line of sight locus, for example. In the explanation of the example of FIG. 10, the graphics generating unit 701 changes the luminance of the line of sight range and the virtual viewpoint image, but the image generating unit 205 may change the luminance.

[0067] As described above, in the second embodiment, information related to a person's line of sight, such as the line of sight, line of sight trajectory, line of sight range, etc., is displayed graphically and presented to the user. As a result, according to the second embodiment, not only is it possible to suppress the virtual viewpoint image corresponding to the line of sight of the person who is the subject of line of sight extraction from moving drastically as in the first embodiment, but it is also possible to present to the user in an easy-to-understand manner where the line of sight is directed at a certain time.

[0068] In addition, the various graphic displays described in the second embodiment may be combined in part or in whole as appropriate. For example, graphic displays such as a trajectory object 901, a range object 903, and a window object 904 may be performed while changing the brightness of the virtual viewpoint image as shown in Fig. 10. Also, which of the objects is to be displayed may be appropriately switched according to a user's instruction. In the present embodiment, the image processing device 700 has been described as having the image generating unit 205, the display unit 206, and the graphics generating unit 701, but is not limited thereto. For example, at least one of the image generating unit 205, the display unit 206, and the graphics generating unit 701 may be configured to be connected to the outside of the image processing device 700 as another device. Also, the image generating unit 205, the display unit 206, and the graphics generating unit 701 may be the same device or different devices. That is, like the image processing device 100 in the first embodiment described above, the image processing device 700 at least performs a process of determining virtual camera parameters in the estimation unit 203 and outputting the virtual camera parameters to a processing unit that generates a virtual viewpoint image or to another device.

[0069] The present invention can also be realized by supplying a program for implementing one or more of the functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more of the functions. The above-mentioned embodiments are merely examples of the implementation of the present invention, and the technical scope of the present invention should not be interpreted as being limited by these. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0070] 100: image processing device, 101: CPU, 102: main storage device, 103: auxiliary storage device, 104: external I / F, 105: GPU, 201: input unit, 202: gaze acquisition unit, 203: estimation unit, 204: data storage unit, 205: image generation unit, 206: display unit

Claims

1. an acquisition means for acquiring line-of-sight information relating to a moving range of the line of sight of a subject photographed by a plurality of photographing devices; a determination means for determining virtual viewpoint information representing a virtual viewpoint related to the virtual viewpoint image based on a plurality of photographed images acquired by photographing performed by the plurality of photographing devices, the virtual viewpoint image including a moving range of the line of sight of the subject specified based on line of sight information acquired by the acquisition means; and an output means for outputting the virtual viewpoint information determined by the determination means; 13. An image processing device comprising:

2. The image processing apparatus according to claim 1 , wherein the acquiring means acquires the line-of-sight information relating to a moving range of the line of sight based on a trajectory of the movement of the line of sight of the subject within a predetermined time range.

3. the acquiring means acquires the line-of-sight information relating to a range of movement of the line of sight based on a plurality of trajectories of movement of the line of sight acquired for each predetermined time range; 3 . The image processing apparatus according to claim 2 , wherein the determining means determines the virtual viewpoint information based on the line-of-sight information acquired based on a trajectory of movement of the plurality of lines of sight.

4. 4. The image processing apparatus according to claim 2, wherein the acquisition means acquires an instruction for the predetermined time range, and updates the predetermined time range in response to the instruction.

5. 5. The image processing device according to claim 2, wherein the determining means gradually changes the virtual viewpoint information between first virtual viewpoint information determined corresponding to a first predetermined time range and second virtual viewpoint information determined corresponding to a second predetermined time range that differs from the first predetermined time range by the predetermined time range.

6. The image processing device according to claim 5, characterized in that the determination means determines the first virtual viewpoint information and the second virtual viewpoint information as virtual viewpoint information of key frames, and determines virtual viewpoint information supplemented for each time period obtained by dividing the specified time range by the number of frames when displaying the virtual viewpoint image as the gradually changing virtual viewpoint information.

7. 7. The image processing apparatus according to claim 5, wherein, when the virtual viewpoint information includes information for changing the width of the field of view of the virtual viewpoint to a narrower angle, the determining means gradually changes the width of the field of view.

8. 8. The image processing device according to claim 1, further comprising a control means for controlling so that an image related to the line of sight of the subject and a virtual viewpoint image corresponding to the virtual viewpoint information determined by the determination means are superimposed on each other.

9. 9. The image processing apparatus according to claim 8, wherein the control means controls so that an image representing a center of the subject's line of sight is superimposed as the image related to the subject's line of sight.

10. 10. The image processing device according to claim 8, wherein the control means controls the image related to the line of sight of the subject superimposed on the virtual viewpoint image so as to gradually disappear over time.

11. 11. The image processing device according to claim 1, wherein the virtual viewpoint information includes at least one of information representing a position of the virtual viewpoint, information representing a line of sight from the virtual viewpoint, and information representing a width of a field of view of the virtual viewpoint.

12. 12. The image processing device according to claim 1, wherein the movement range of the line of sight is included in a range of an effective visual field excluding a range of a peripheral visual field in the line of sight of the subject.

13. 13. The image processing device according to claim 1, wherein the subject is a player or a referee photographed by the plurality of photographing devices.

14. 14. The image processing apparatus according to claim 1, further comprising a generating unit that generates a virtual viewpoint image based on the virtual viewpoint information output by the output unit.

15. An image processing method executed by an image processing device, comprising: an acquisition step of acquiring line-of-sight information relating to a moving range of a line of sight of a subject photographed by a plurality of photographing devices; a determination step of determining virtual viewpoint information representing a virtual viewpoint related to the virtual viewpoint image based on a plurality of photographed images acquired by photographing performed by the plurality of photographing devices, the virtual viewpoint image including a moving range of the line of sight of the subject specified based on line of sight information acquired by the acquisition step; an output step of outputting the virtual viewpoint information determined by the determination step; 13. An image processing method comprising:

16. A program for causing a computer to function as the image processing device according to any one of claims 1 to 14.

Images