Image processing device, image processing method, and program

Abstract

To enable generation of a virtual viewpoint image further suitable for a viewing purpose.SOLUTION: An image processing device (100) includes: obtaining means (202) for obtaining visual line information relating to a moving range of the visual line of an object imaged by a plurality of imaging devices; determining means (203) for determining virtual viewpoint information representing a virtual viewpoint relating to a virtual viewpoint image which is on the basis of a plurality of photographed images obtained by photographing carried out by the plurality of imaging devices, and includes the moving range of the visual line of the object identified on the basis of the visual line information obtained by the obtaining means (202); and output means (203) for outputting the virtual viewpoint information determined by the determining means (203).SELECTED DRAWING: Figure 2

Description

[Technical Field]

[0001] The present invention relates to an image processing technique for generating a virtual viewpoint image. [Background technology]

[0002] In recent years, attention has been focused on a technology that generates an image viewed from an arbitrary viewpoint, i.e., a virtual viewpoint image equivalent to an image captured by a virtual camera, based on the multiple viewpoint images obtained from multiple cameras installed in different positions and capturing images synchronously. A virtual viewpoint image is generated by collecting the multiple viewpoint images captured by multiple cameras in an image processing device such as a server, and performing processes such as foreground / background separation, 3D shape estimation, 3D model generation, and rendering in the image processing device.

[0003] The virtual viewpoint image is then transferred to a user terminal and displayed, allowing the user (viewer) to view the virtual viewpoint image (virtual viewpoint video). This technology allows users to view highlights of soccer, basketball, and other games from various positions and angles that were previously unrealizable, such as on the field during the game, providing a more realistic experience than conventional video. Furthermore, for example, by acquiring information on the gaze of players and referees and using that information to generate and display virtual viewpoint images, gaze analysis can be performed to improve the skills of players and referees. Furthermore, virtual viewpoint images based on the gaze information of players and referees allow users (viewers) to virtually experience where and how the players and referees were looking during the game, allowing them to enjoy the game more through this realistic experience. Patent Document 1 discloses a technology in which, in a device operating device capable of accepting gaze-based operation input, a user manually inputs a gaze position correction instruction, and the average value of the gaze position error for each correction instruction is reflected in gaze detection. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-152957 Summary of the Invention [Problem to be solved by the invention]

[0005] However, depending on the situation of the game, the gazes of players and referees may move quickly and widely up and down and left and right. A virtual viewpoint image generated based on information about gazes moving rapidly up and down and left and right results in an image with intense movement and blurring that would be unthinkable with normal camera work, making the video very difficult for users to watch. Such virtual viewpoint images may be unsuitable for viewing purposes, for example, they may cause motion sickness. Note that even if the technology disclosed in Patent Document 1 is used, it is difficult to suppress the movement and blurring of an image caused by gazes moving rapidly up and down and left and right.

[0006] Therefore, an object of the present invention is to make it possible to generate a virtual viewpoint image that is more suitable for viewing purposes. [Means for solving the problem]

[0007] The image processing device of the present invention is configured to: position and an acquisition means for acquiring the Based on the position of the subject, a position of a virtual viewpoint related to a virtual viewpoint image is calculated. a determining means for determining a generating means for generating the virtual viewpoint image based on a plurality of photographed images acquired by photographing performed by the plurality of photographing devices and the position of the virtual viewpoint; and a control means for controlling the virtual viewpoint image so that a graphic representing the line of sight or field of view of the subject is superimposed on the virtual viewpoint image. The present invention is characterized by having the following. [Effects of the Invention]

[0008] According to the present invention, it is possible to generate a virtual viewpoint image that is more suitable for viewing purposes. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a diagram illustrating a hardware configuration of an image processing apparatus. [Figure 2] 1 is a functional block diagram of an image processing device according to a first embodiment. [Figure 3] 4 is a flowchart of a virtual viewpoint image generation process in the first embodiment. [Figure 4] FIG. 1 is a diagram schematically illustrating a scene from a soccer match. [Figure 5] FIG. 10 is an explanatory diagram of estimating the gaze range of the referee. [Figure 6] FIG. 10 is an explanatory diagram of virtual camera parameter complementation when generating a virtual viewpoint image. [Figure 7] FIG. 10 is a functional block diagram of an image processing device according to a second embodiment. [Figure 8] 10 is a flowchart of a virtual viewpoint image generation process in the second embodiment. [Figure 9] FIG. 10 is a diagram showing a display example of an object related to a line of sight. [Figure 10] FIG. 10 is a diagram showing an example of a graphics display of a line-of-sight range. DETAILED DESCRIPTION OF THE INVENTION

[0010] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the present invention is not limited to the illustrated configurations. In addition, the same configurations or processes in the embodiments will be described with the same reference numerals.

[0011] First Embodiment FIG. 1 is a diagram schematically illustrating the hardware configuration of an image processing apparatus 100 according to this embodiment. The image processing device 100 includes a CPU 101, a main memory device 102, an auxiliary memory device 103, an external I / F 104, a GPU 105, and a bus 106. These hardware components are interconnected via the bus 106 so that they can communicate with each other. The bus 106 is configured using a general-purpose bus such as PCI Express (registered trademark).

[0012] The CPU 101 is a central processing unit that performs overall control of the image processing device 100, such as various calculations and decisions, control of image processing, and control of data input / output. The main memory device 102 is a storage device that functions as a work area and a temporary data storage area for the CPU 101. The main memory device 102 is implemented using a storage medium such as a dynamic random access memory (DRAM) or a static random access memory (SRAM).

[0013] The auxiliary storage device 103 is a storage device that stores information such as various programs, various setting information, various image data, camera parameters, 3D shape data, and 2D maps, and is implemented using various storage media. Examples of storage media include non-volatile memory such as read-only memory (ROM) and flash memory, hard disk drives (HDDs), solid state drives (SSDs), and tape media. Note that a plurality of these storage media may be combined to increase capacity and speed, or may be logically grouped together, such as in a redundant array of inexpensive disks (RAID).

[0014] The external I / F (Interface) 104 includes a communication interface and an input / output interface. The communication interface is used for communication with external devices such as a camera, a PC (personal computer), and a server. The input / output interface is used for connecting to external devices that accept various user operations, such as a joystick, keyboard, mouse, and touch panel, and for connecting to external storage and performing file input and output. The external I / F 104 is implemented as an interface with a physical cable connection terminal, or a wireless interface. Examples of interfaces with a physical cable connection terminal include InfiniBand, Ethernet, and USB (Universal Serial Bus). Examples of wireless interfaces include wireless LAN and Bluetooth (registered trademark).

[0015] The GPU 105 is a graphics processing unit for performing image processing calculations at high speed and has a function of outputting a video signal to an external video display device such as a television. In this embodiment, the image processing that the GPU 105 can perform includes a process for generating a virtual viewpoint image. The virtual viewpoint image is generated by performing processes such as foreground / background separation, 3D shape estimation, 3D model generation, and rendering using multiple viewpoint images obtained by synchronously capturing images from multiple image capturing devices (cameras not shown in FIG. 1) installed at different positions. The virtual viewpoint image is an image generated based on the position of a virtual viewpoint, which is a specified arbitrary viewpoint, and the line of sight direction from the virtual viewpoint. For convenience of explanation, the following description will be given using a virtual camera (virtual camera) corresponding to the virtual viewpoint. That is, the position and orientation (imaging direction) of the virtual camera correspond to the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint, respectively. Information indicating the position and orientation of the virtual camera will be referred to as virtual camera information. The virtual camera information includes information indicating the position and orientation of the virtual camera. The virtual camera information may further include information corresponding to the width of the field of view (angle of view) of the virtual viewpoint. In other words, the virtual viewpoint image can be said to be an image generated based on the virtual camera information. The virtual camera information may include at least one of information on the position, orientation, and angle of view. The image processing executed by the GPU 105 includes not only the processing of rendering an image seen from a virtual viewpoint but also the processing of generating shape data and color information required for generating a virtual viewpoint image. Data of multiple viewpoint images taken by multiple cameras installed at different positions is acquired via, for example, the external I / F 104 and stored in, for example, the auxiliary storage device 103. In this embodiment, a virtual camera is a virtual camera that is virtually placed corresponding to an arbitrary viewpoint in a virtual space, and is distinguished from a real camera that is placed at a different position in a real space and captures images from multiple viewpoints. The virtual camera information is information generated by the CPU 101 in this embodiment, and its details will be described later.

[0016] The video signal output of the GPU 105 is implemented using a video terminal such as a DVI (Digital Visual Interface), HDMI (registered trademark), Display Port, or SDI (Serial Digital Interface). The GPU 105 may be configured to perform only image processing and calculations such as shape data and color information required for virtual viewpoint images without having a video signal output function. In this case, for example, the external I / F 104 may be responsible for the video output function or the data output function required for video output. The GPU 105 may also have a function to execute some or all of the various calculations performed by the CPU 101, other than those described above. The GPU 105 may also include a separate storage device separate from the main storage device 102.

[0017] The image processing device 100 of this embodiment is not limited to the configuration shown in FIG. 1 and may be realized by other hardware configurations without departing from the spirit of the present invention. For example, the auxiliary storage device 103 may be configured external to the image processing device 100, and in that case, the auxiliary storage device may be connected via an external I / F 104. The main storage device 102 and the auxiliary storage device 103 may be realized as a single piece of hardware. Instead of the GPU 105, a field programmable gate array (FPGA), a digital signal processor (DSP), an image processing large scale integrated circuit (LIS), or the like may be used. The image processing device 100 is not limited to a single device, and may be configured by, for example, multiple devices having similar configurations, each of which may have a different configuration without departing from the spirit of the present invention.

[0018] The following describes the virtual viewpoint image generation process performed by the image processing device 100 according to this embodiment. The image processing device 100 according to this embodiment has functions of performing an acquisition process for acquiring line-of-sight information, a determination process for determining virtual viewpoint information, and a generation process for generating a virtual viewpoint image. In the acquisition process, line-of-sight information is acquired, which is information for identifying the line of sight of a subject captured by multiple image capture devices. The line-of-sight information includes, for example, information representing the line of sight of the subject, such as a player or a referee. The image processing device 100 can identify the range of movement of the subject's line of sight based on the line-of-sight information. In the determination process, virtual viewpoint information representing a virtual viewpoint is determined for the virtual viewpoint image, which is based on multiple captured images captured by multiple image capture devices and includes the range of movement of the subject's line of sight identified based on the line-of-sight information acquired in the acquisition process. In the generation process, the virtual viewpoint image is generated based on the virtual viewpoint information determined in the determination process. The image processing device 100 generates a virtual viewpoint image, which is an image virtually captured by a virtual camera represented by virtual camera information including the position, orientation, angle of view, etc. of the virtual camera installed in the virtual space, based on line-of-sight information relating to the range of movement of the person's line of sight in the virtual space. In the following description, for the sake of simplicity, the range of movement of the subject's (person's) line of sight will be referred to as the "line of sight range."

[0019] In this embodiment, for example, a virtual viewpoint image is generated based on multiple viewpoint images obtained by synchronously capturing a soccer game using multiple real cameras at different positions. The virtual viewpoint image is generated using the gaze information of players and referees. In this case, as described above, the gaze of players and referees may move rapidly and widely in various directions, resulting in a virtual viewpoint image generated using the gaze information that exhibits intense movement and blur. While the technology described in Patent Document 1 incorporates the average error value of each gaze position into gaze detection through manual input, such as button operation, in a soccer game, there is no specific area to focus on, as occurs when buttons are operated. Furthermore, the rapidly moving gaze of players and referees is not due to blurring relative to the gaze position, but rather to the fact that the player or referee gazes at multiple locations in a short period of time. Therefore, if the technology described in Patent Document 1 is applied to correct the rapid gaze movement as blur, it becomes difficult to determine where the player or referee was looking, which defeats the purpose of the user virtually viewing the player or referee's field of view. Note that the user in this embodiment may not only be a viewer who virtually experiences the field of view of a player or a referee during a match, but may also be a player or a referee who watches a virtual viewpoint image in order to improve their skills.

[0020] Therefore, the image processing device 100 of this embodiment has the functions shown in Fig. 2 and performs the processing shown in the flowchart of Fig. 3, which will be described later, to generate a virtual viewpoint image with reduced sudden vertical and horizontal movement and blur. The image processing device 100 of this embodiment acquires gaze information related to the range of movement of the gaze of a person in a virtual space, and sets virtual camera information of a virtual camera corresponding to the virtual viewpoint information related to a virtual viewpoint image including the range of movement of the gaze of the person identified based on the gaze information. Then, the image processing device 100 generates a virtual viewpoint image corresponding to an image captured by the virtual camera for which the virtual camera information has been set.

[0021] Fig. 2 is a functional block diagram showing the functional configuration of the image processing device 100 according to the first embodiment. As shown in Fig. 2, the image processing device 100 has functional units, namely, an input unit 201, a gaze acquisition unit 202, an estimation unit 203, a data storage unit 204, an image generation unit 205, and a display unit 206. Note that the functional units shown in Fig. 2 and the processing shown in the flowchart of Fig. 3, which will be described later, are realized, for example, by the CPU 101 shown in Fig. 1 executing processing in accordance with a program according to this embodiment stored in the auxiliary storage device 103.

[0022] The input unit 201 includes an external I / F 104, acquires operation information from a user, and acquires inputs related to various instructions from the user based on the operation information. Examples of user inputs based on operation information include selection of virtual viewpoint image content, video operations such as play / stop, fast forward, and cueing by specifying a time code, and camera operation. Furthermore, in this embodiment, user inputs based on operation information include selection of a desired person such as a player or referee for whom a virtual viewpoint image using gaze information is to be generated (person designation), and an instruction for an update rate for changing the time range for estimating the gaze range (described later).

[0023] The gaze acquisition unit 202 acquires gaze information of people who can be targets for generating a virtual viewpoint image in the virtual space, that is, in the example of this embodiment, acquires gaze information of people (players and referees) on the soccer field. For example, the gaze acquisition unit 202 acquires the gaze information of players and referees from output information of sensors worn by players, referees, etc. during a match, or by image analysis of multi-viewpoint images. Furthermore, if the gaze information has been acquired in advance and stored in the auxiliary storage device 103 or external media, the gaze acquisition unit 202 may acquire the gaze information from media, a network, etc. connected to the auxiliary storage device 103 or the external I / F 104. Note that in this embodiment, the gaze acquisition unit 202 acquires gaze information for each unit time, such as a frame of a video.

[0024] The estimation unit 203 sets virtual camera information for determining the range of photography by a virtual camera installed in a virtual space. In this embodiment, the estimation unit 203 acquires user input from the input unit 201 and also acquires gaze information from the gaze acquisition unit 202. Based on the user input from the input unit 201, the estimation unit 203 identifies a person, such as a player or a referee, selected (designated) by the user as a target for generating a virtual viewpoint image using the gaze information. Then, the estimation unit 203 estimates the gaze movement range (hereinafter also referred to as gaze range) of the identified person within a predetermined time range (within a predetermined period) set in advance, and further sets virtual camera information that includes the gaze range estimated from the gaze position of the person. The virtual camera information is camera parameters (referred to as virtual camera parameters) of a virtual camera that virtually photographs a virtual space, and includes at least parameters for setting the position, attitude, photographing direction, and angle of view of the virtual camera.

[0025] Although a detailed description will be given later, the estimation unit 203 at this time acquires a trajectory of the gaze of a person identified in response to user input, based on gaze information acquired by the gaze acquisition unit 202 at each unit time, such as a frame period. Furthermore, the estimation unit 203 estimates a gaze range, which is the range of gaze movement of the person within a predetermined time range, based on the gaze trajectory of the person within that time range. The estimation unit 203 then sets virtual camera information (virtual camera parameters) that encompasses the multiple gaze ranges estimated from the person's gaze position. The predetermined time range may be predetermined, or may be any time range designated by the user. These processes in the estimation unit 203 are performed by, for example, the CPU 101 in the configuration of FIG. 1 .

[0026] The data storage unit 204 stores data consisting of 3D model data, color information, etc. in the auxiliary storage device 103, etc. In this embodiment, the image generation unit 205 generates a virtual viewpoint image corresponding to an image captured by a virtual camera of a virtual space, based on the virtual camera parameters set by the estimation unit 203 based on the line-of-sight range. At this time, the image generation unit 205 acquires data consisting of 3D model data, color information, etc. from the data storage unit 204, and generates a virtual viewpoint image, which is an image virtually captured by the virtual camera, based on the data. Note that the process of generating the virtual viewpoint image in the image generation unit 205 is performed by, for example, the GPU 105 in the configuration of FIG. 1.

[0027] The display unit 206 displays the virtual viewpoint image generated by the image generation unit 205 on a television receiver, a monitor of a personal computer, a tablet terminal, or the like via the external I / F 104. This allows the user to view the virtual viewpoint image.

[0028] 3 is a flowchart showing the flow of processing for calculating a line-of-sight range and generating a virtual viewpoint image in the image processing device 100 of the first embodiment. Hereinafter, using this flowchart, the flow of processing for generating a virtual viewpoint image in accordance with the line of sight of, for example, the referee by the image processing device 100 will be described using a soccer match as an example.

[0029] In step S301, the input unit 201 acquires the update rate of the time range when estimating the gaze range from input information input by a user operation. Then, information on the update rate of the time range is sent to the estimation unit 203. In this embodiment, the time range when estimating the gaze range is set to, for example, one second.

[0030] In the next step S302, the input unit 201 acquires the playback start time of the virtual viewpoint image content specified by the user from the input information entered by the user's operation. This information on the playback start time of the virtual viewpoint image content is sent to the image generation unit 205 via the estimation unit 203. For example, when the user instructs to cue the virtual viewpoint image content, the image generation unit 205 plays back the virtual viewpoint image content from its beginning.

[0031] In the next step S303, the input unit 201 acquires information indicating a person designated (selected) as a target for gaze extraction when generating a virtual viewpoint image using gaze information, from input information input by a user operation. This designated information of the target person for gaze extraction is sent to the estimation unit 203.

[0032] FIG. 4 is a diagram that schematically illustrates a scene from a soccer match. When designating a target for gaze extraction in step S303, a screen showing a scene such as that shown in FIG. 4 is displayed on the display unit 206, and the user can select a person on the screen using a pointing device such as a mouse or touch panel. Here, it is assumed that the user designates a referee 401 as a target for gaze extraction. At this time, it is assumed that the referee 401 is facing in the direction of the ball 405 (the direction indicated by the arrow 402). Note that in this example, the referee 401 on the field is designated, but the person for gaze extraction is not limited to the referee. For example, a player 403 or a goalkeeper 404 may also be designated as a target for gaze extraction.

[0033] In the next step S304, the estimation unit 203 extracts the world coordinates of the inter-eyebrow position of the person (the referee 401 in the example of FIG. 4) from the gaze extraction target by face detection or the like, and sets the extracted coordinates as the external parameters of the virtual camera. In this embodiment, the estimation unit 203 sets the world coordinates of the inter-eyebrow position of the person as the virtual camera parameters representing the position of the virtual camera. Here, the world coordinates of the inter-eyebrow position are set as the virtual camera parameters representing the position of the virtual camera, but this is not limiting. For example, the positions of the left and right eyeballs may be detected, and the positions of either or both eyeballs may be set as the virtual camera parameters representing the position of the virtual camera. For example, if the positions of both eyeballs are used, it is possible to generate two images with parallax as virtual viewpoint images. Alternatively, for example, a position near the center of the head or the top of the head may be used. Note that the height direction position of the virtual camera may be assigned a specific value (height) for each player and referee, or the same specific value (height) for all players and referees.

[0034] Next, in step S305, the estimation unit 203 estimates the gaze range by extracting the gaze trajectory within the time range (one second) specified in step S301 from the gaze information of the person specified in step S303 (referee 401 in the example of FIG. 4). Then, the estimation unit 203 obtains, from the estimated gaze range, virtual camera information (virtual camera parameters) other than the virtual camera position obtained in step S304 when virtual photography is performed at the virtual camera position.

[0035] Figures 5(a) to 5(e) are schematic diagrams showing how the gaze range of referee 401 shown in Figure 4 is estimated and how virtual camera parameters are calculated based on that gaze range. In Figures 5(a) to 5(e), the dotted lines represent the trajectory of the center of referee 401's gaze as it moves over time.

[0036] The gaze path 501 shown in Figure 5(a) is the path that the referee 401 takes when he moves his gaze from gaze position 501a as shown by the dotted line in the figure over time, and then after two seconds, moves his gaze to gaze position 501b. For the sake of convenience in the illustration, each person, such as a player, is depicted as standing still, but in reality, each person moves freely over time.

[0037] Figure 5(b) shows the estimated gaze range within the field of view of referee 401 at time t1 in an example where referee 401's gaze moves as shown by trajectory 501 from gaze position 501a to gaze position 501b as shown in Figure 5(a). Trajectory 502 shown in Figure 5(b) is the gaze trajectory acquired at time t1, and shows the gaze trajectory over one second of the time range set in step S301. Here, the one second from time t1-1 to time t1 is the time range for estimating the gaze, and gaze position 501a represents the center position of referee 401's gaze at time t1-1. Also in Figure 5(b), gaze range 503, enclosed by a solid line, shows the estimated gaze range within referee 401's field of view at gaze position 501a. At this time, the gaze range 503 estimated from the center of gaze within the field of vision of the referee 401 is the range of the effective field of vision excluding the range known as peripheral vision, and is set to an average angle of view.

[0038] Figure 5(c) shows how the range of the effective visual field is sequentially acquired along trajectory 502 in the same manner as described above. Gaze range 505a is the gaze range estimated as the range of referee 401's effective visual field when the center of gaze of referee 401 moves from gaze position 501a to gaze position 504a over time. Similarly, gaze range 505b is the gaze range estimated when the gaze center moves from gaze position 504a to gaze position 504b, and gaze range 505c is the gaze range estimated when the gaze center moves from gaze position 504b to gaze position 504c.

[0039] In this way, estimation unit 203 estimates the gaze range sequentially along the gaze trajectory for one second from time t1-1 to time t. Then, estimation unit 203 acquires, as included gaze range 506 at time t, a range that includes all angles of view of the effective field of view for each gaze range estimated along the gaze trajectory for one second from time t1-1 to time t, as shown in FIG. 5(d).

[0040] Furthermore, estimation unit 203 estimates virtual camera parameters other than the position of the virtual camera at time t based on the angle of view corresponding to inclusive line-of-sight range 506 at time t. Note that inclusive line-of-sight range 506 does not need to have a shape inscribed with each line-of-sight range estimated within the time range from time t1-1 to time t, and may have any shape as long as all line-of-sight ranges are included within it. Furthermore, the virtual camera parameters may correspond to a shape with an aspect ratio of 16:9 or 4:3, which is a typical aspect ratio of video signals, or an aspect ratio such as 3:2 used in photography. Alternatively, the inclusive line-of-sight range may be set to be the narrowest possible, as long as it includes all line-of-sight ranges within those aspect ratios.

[0041] As with inclusive gaze range 506 described in FIG. 5(d), estimation unit 203 determines the inclusive gaze range for one second from time t2-1 (i.e., time t1) to time t2, where time t2 is the time range for estimating the gaze after time t1. Inclusive gaze range 507 shown in FIG. 5(e) indicates the inclusive gaze range determined for one second from time t2-1 to time t2. Then, estimation unit 203 estimates virtual camera parameters other than the position of the virtual camera at time t2 based on the angle of view corresponding to inclusive gaze range 507 at time t2. Similarly, estimation unit 203 determines the inclusive gaze range and sets the virtual camera parameters each time the time range is updated, that is, at each update rate.

[0042] Returning to the explanation of the flowchart in Fig. 3, in the next step S306, the image generation unit 205 renders a virtual viewpoint image using the virtual camera parameters corresponding to the inclusive line-of-sight ranges at time t1 and time t2 received from the estimation unit 203. Then, the display unit 206 displays the virtual viewpoint image.

[0043] At this time, the information received by the image generation unit 205 is a combination of time information such as time t1 and time t2 and virtual camera parameters corresponding to the inclusive line-of-sight range linked to the time information. Here, since the virtual viewpoint image is composed of 60 frames per second, if there is a time interval of 1 second between time t1 and time t2, the image generation unit 205 needs to generate images for 60 frames as the virtual viewpoint image between time t1 and time t2.

[0044] Therefore, the image generation unit 205 gradually changes the virtual camera information between the virtual camera information set for a predetermined time range at time t1 and the second virtual camera information set for a time range at time t2, which differs from time t1 by the update rate of the time range. For example, the image generation unit 205 uses the first virtual camera information and the second virtual camera information as keyframe virtual camera information and acquires virtual camera information interpolated for each time period obtained by dividing the predetermined time range by the number of frames when displaying the virtual viewpoint image. In this embodiment, the image generation unit 205 uses the virtual camera parameters at times t1 and t2 obtained from the estimation unit 203 as keyframes and generates intermediate data that interpolates between the virtual camera parameters at time t1 and the virtual camera parameters at time t2. The intermediate data is generated as data that gradually changes from the virtual camera parameters at time t1 to the virtual camera parameters at time t2. Furthermore, in this embodiment, when the angle of view of the virtual camera parameters set by the estimation unit 203 changes to a narrow angle, the image generation unit 205 gradually changes the angle of view. As a result, the image generation unit 205 generates virtual viewpoint images based on virtual camera parameters that gradually change over the one second from time t1 to time t2, that is, 60 frames of virtual viewpoint images in which the images gradually change between time t1 and time t2.

[0045] 6(a) to 6(d) are diagrams showing how virtual viewpoint images are generated while complementing virtual camera parameters for each frame between the inclusive line-of-sight ranges at times t1 and t2 calculated as described above by estimation unit 203. Note that, due to space limitations, the examples in Fig. 6(a) to 6(d) show one second divided into four periods, but in reality, virtual viewpoint images equivalent to 60 frames are generated.

[0046] Fig. 6(a) shows the angle of view when a virtual viewpoint image is generated at time t1. At time t1, a virtual viewpoint image is generated corresponding to angle of view 601, which corresponds to inclusive line-of-sight range 506 shown in Fig. 5(d). Fig. 6(a) also shows inclusive line-of-sight range 507 calculated in Fig. 5(e) as angle of view 602, indicated by a dotted line, as the angle of view when a virtual viewpoint image is generated at time t2.

[0047] 6(b) is a diagram showing the angle of view 603 at time t1+Δ1, which is a time Δ1 that is shorter than one second within the time range of gaze estimation from time t1. This time t1+Δ1 is a time between time t1 and time t2. Because the virtual camera parameters corresponding to time t1+Δ1 between time t1 and time t2 are not estimated by the estimation unit 203, the image generation unit 205 cannot acquire the virtual camera parameters corresponding to time t1+Δ1. Therefore, the virtual camera parameters corresponding to time t1+Δ1 are complemented as intermediate data using the virtual camera parameters at time t1 and the virtual camera parameters at time t2 as key frames.

[0048] Here, the angle of view 603 at time t1+Δ1 is set as an intermediate angle of view between the angle of view 601 at time t1 and the angle of view 602 at time t2. In this embodiment, the image generation unit 205 linearly divides the value representing the angle of view 601 at time t1 and the value representing the angle of view 602 at time t2 evenly based on the number of frames for generating a virtual viewpoint image. Furthermore, the image generation unit 205 sets the angle of view represented by the value corresponding to time t1+Δ1 among the values representing each divided angle of view as the angle of view 603 at time t1+Δ1. The image generation unit 205 then generates a virtual viewpoint image based on virtual camera parameters corresponding to the angle of view 603. Alternatively, the image generation unit 205 may divide the two values representing the angle of view 601 at time t1 and the angle of view 602 at time t2 using interpolation that accelerates or decelerates in a curved manner, such as drawing a Bezier curve or a sine curve. Furthermore, the present invention is not limited to these, and any interpolation method may be used as long as it is possible to acquire interpolated virtual parameters that are intermediate values between the virtual camera parameters at time t1 and the virtual camera parameters at time t2.

[0049] 6(c) is a diagram showing an angle of view 604 at time t1+Δ2, which is a further time that has passed since time t1+Δ1. At time t1+Δ2 in FIG. 6(c), an angle of view 604 that is closer to the angle of view 602 at time t2 than the angle of view 603 at time t1+Δ1 is obtained using the same interpolation method as described above.

[0050] Thereafter, at time t2, the angle of view becomes 602 as shown in Fig. 6(d). In the image generation unit 205, the virtual camera parameters are interpolated between time t1 and time t2 as described with reference to Figs. 6(a) to 6(d), and a virtual viewpoint image is generated based on these virtual parameters.

[0051] As described above, the image processing device 100 of the first embodiment generates a virtual viewpoint image by obtaining virtual camera parameters from a gaze range estimated based on the gaze trajectory for each predetermined time range. As a result, when generating a virtual viewpoint image corresponding to the gaze of a person from whom gaze extraction is to be performed, it is possible to generate a virtual viewpoint image that is estimated to be viewed by that person while suppressing rapid image movement.

[0052] In the first embodiment, for example, when the inclusive gaze range at time t1 is calculated, it is estimated based on gaze information before time t1 according to the time range, but this is not limiting, and the inclusive gaze range may be calculated based on gaze information after time t1. Furthermore, when estimating over a time range including time t1, for example, a time range of one second as described above, any method may be used as long as the gaze estimation range is calculated based on gaze information for consecutive one seconds, such as 0.5 seconds before and after time t1.

[0053] In this embodiment, the angle of view corresponding to the included line-of-sight range may be the narrowest angle of view that includes the detected line-of-sight trajectory and corresponds to the aspect ratio of the output video. Alternatively, the angle of view corresponding to the included line-of-sight range may be an angle of view that is expanded by a certain range relative to the narrowest angle of view, for example, as specified by the user.

[0054] In this embodiment, the gaze range is set to an average angle of view as the effective field of view, excluding a range known as peripheral vision, but is not limited to this. For example, the gaze range may be calculated using a narrower angle of view, such as using an angle considered to be the field of view of the fovea for the gaze information. Alternatively, the effective field of view may be set to any field of view angle in response to a user instruction, and may be an angle of view that includes that field of view angle. Any range including the center of the gaze may be set as the effective field of view. Furthermore, in the present embodiment, the image processing device 100 has been described as having the image generation unit 205 and the display unit 206, but is not limited to this. For example, at least one of the image generation unit 205 and the display unit 206 may be configured to be connected to the outside of the image processing device 100 as another device. Furthermore, when both the image generation unit 205 and the display unit 206 are devices connected to the outside of the image processing device 100, the image generation unit 205 and the display unit 206 may be the same device or different devices. That is, the image processing device 100 at least determines virtual camera parameters in the estimation unit 203 and performs processing to output the virtual camera parameters to a processing unit that generates a virtual viewpoint image or another device.

[0055] Second Embodiment Fig. 7 is a functional block diagram showing the functional configuration of an image processing device 700 according to the second embodiment. In the image processing device 700, the input unit 201 to the display unit 206 are the same as the corresponding functional units of the image processing device 100 shown in Fig. 2, and therefore a description thereof will be omitted. Note that the hardware configuration of the image processing device 700 according to the second embodiment is the same as the configuration shown in Fig. 1, and therefore illustration and description thereof will be omitted.

[0056] The graphics generation unit 701 acquires the gaze information acquired by the gaze acquisition unit 202, information on the included gaze range estimated by the estimation unit 203, and virtual camera parameters used when the image generation unit 205 generates the virtual viewpoint image. Based on this information, the graphics generation unit 701 generates graphics data for displaying, for example, the gaze and its trajectory, the gaze point where the person who is the subject of gaze extraction is gazing, or the gaze range, and sends the generated graphics data to the display unit 206. As a result, the display unit 206 performs a graphics display of the gaze and its trajectory, the gaze point, the gaze range, etc.

[0057] Fig. 8 is a flowchart showing the flow of processing in the image processing device 700 of the second embodiment, in which the line-of-sight range is calculated, a virtual viewpoint image is generated, and graphics display is performed. In Fig. 8, the processing from step S301 to step S306 is the same as the corresponding steps in Fig. 3, and therefore a description thereof will be omitted. In the flowchart of Fig. 8, after step S306, the processing of the image processing device 700 proceeds to step 801.

[0058] In step S801, the graphics generation unit 701 acquires information on the time and virtual camera parameters of the virtual viewpoint image generated by the image generation unit 205, the gaze information acquired by the gaze acquisition unit 202, and the included gaze range estimated by the estimation unit 203. Based on this information, the graphics generation unit 701 then generates graphics data to be displayed on the display unit 206. That is, the graphics generation unit 701 performs a related image generation process to generate an object as a related image related to the gaze of a person who is the subject of gaze extraction, such as the referee 401. The display unit 206 superimposes and displays the object generated by the graphics generation unit 701 based on the graphics data on the virtual viewpoint image generated by the image generation unit 205. Note that in this embodiment, the graphics data and the virtual viewpoint image are superimposed and displayed on the display unit 206, but this is not limiting. For example, the image generation unit 205 may perform a process to superimpose the graphics data and the virtual viewpoint image, and the display unit 206 may display the image generated by the image generation unit 205.

[0059] 9(a) to 9(c) are schematic diagrams showing how objects are superimposed on a virtual viewpoint image 900 as related images related to the gaze of referee 401, who is the person whose gaze is to be extracted. Note that Fig. 9(a), Fig. 9(b), and Fig. 9(c) show examples in which objects related to the gaze of referee 401 are displayed in different forms.

[0060] FIG. 9(a) shows an example in which a trajectory object 901 representing the trajectory of the gaze of the referee 401 is superimposed on a virtual viewpoint image 900 generated based on virtual camera parameters of the inclusive gaze range at time t1, for example. This allows the user to see from a bird's-eye view where the referee 401 was gazing at a certain time within the inclusive gaze range estimated for the referee 401, and to know the gaze range. At this time, the graphics generation unit 701 may highlight the gaze position (gaze position) at the time the virtual viewpoint image is generated, as indicated by, for example, a point object 902. Alternatively, for example, the graphics generation unit 701 may represent the movement of the gaze using the point object 902 and display the trajectory object 901 as the trajectory. Alternatively, the graphics generation unit 701 may display the trajectory object 901 so that the oldest part gradually disappears as time passes.

[0061] Fig. 9(b) shows an example in which a frame representing the gaze range of the referee 401 is superimposed on the virtual viewpoint image. In the example of Fig. 9(b), a range object 903 representing the gaze range 505c of the referee 401 at time t1 is superimposed on the virtual viewpoint image. In this way, the gaze range at each time is superimposed on the virtual viewpoint image as the range object 903, allowing the user to know the gaze range of the referee 401 while viewing his field of vision from a bird's eye view.

[0062] FIG. 9(c) shows an example in which the field of view of the referee 401 is displayed on the virtual viewpoint image in a PiP (Picture In Picture) format. FIG. 9(c) shows an example in which the image area of the referee 401's line of sight (for example, line of sight 503 in FIG. 5(b)) at time t1 is cropped from the virtual viewpoint image and superimposed as a window object 904. Such a PiP superimposed display can also achieve the same effect as described above. In this example, the image area of the line of sight is cropped from the virtual viewpoint image, but the present invention is not limited to this example. For example, the estimation unit 203 may separately generate virtual camera parameters equivalent to the angle of view of the referee 401's line of sight, and the image generation unit 205 may generate a virtual viewpoint image superimposed in the same manner as the window object 904. Although the window object 904 is superimposed on the virtual viewpoint image in the above example, it may also be displayed separately on a display device different from the display device displaying the virtual viewpoint image.

[0063] Figures 10(a) and 10(b) are schematic diagrams showing an example of a display method that makes it possible to present to the user the changes in the gaze of a person whose gaze is to be extracted by changing the brightness of an image area representing the gaze range on a virtual viewpoint image in accordance with the trajectory of the gaze. The virtual viewpoint image 1001 is an image created to correspond to the line of sight from the position of the referee 401, and is an image created to have a wider angle than the viewing angle of an average human. For example, the virtual viewpoint image 1001 may be an image assumed to be taken with an ultra-wide-angle lens with a wide angle of view, an image assumed to be taken with a fisheye lens, or an image assumed to be taken with a lens with an angle of view that covers the entire surroundings, such as a 360-degree camera.

[0064] 10(a) and 10(b), trajectory 1002a represents the trajectory of the gaze from time t1-1 seconds to time t1, and trajectory 1002b represents the trajectory of the gaze from time t1 to time t2. Gaze range 1003a in Fig. 10(b) represents the gaze range estimated at time t1, and gaze range 1003b represents the gaze range estimated at time t2. Note that gaze range 1003a and gaze range 1003b are determined in the same manner as inclusive gaze range 506 described above.

[0065] Here, assuming that the current time is time t2, the graphics generation unit 701 sets the brightness of the area of the gaze range 1003b estimated at time t2 to a higher value (brighter) than the brightness of the area of the gaze range 1003a at time t1, which is earlier than time t2. Furthermore, the graphics generation unit 701 sets the brightness value of the gaze range 1003a at time t1 to gradually lower (darker) over time. The time over which the brightness decreases may be determined by the elapsed time in the real world or by the elapsed time in virtual time based on the time code of the virtual viewpoint image. Furthermore, the graphics generation unit 701 further lowers (darkers) the brightness of areas other than the gaze ranges 1003a and 1003b, i.e., areas where the referee 401's gaze is not directed.

[0066] In the example of FIG. 10(b), the line-of-sight range 1003a and the line-of-sight range 1003b are circular regions encompassing the line-of-sight trajectory. However, the present invention is not limited to this example and may be any shape, such as a rectangular region as shown in FIG. 5 or an elliptical region. The graphics generation unit 701 may change not only the brightness of the regions representing the line-of-sight ranges but also their shapes over time. The graphics generation unit 701 may also move a circular or rectangular region of a certain size representing the line-of-sight range along the line-of-sight trajectory 1002a or trajectory 1002b. In the above example, the brightness of the line-of-sight range 1003a at time t1 is uniformly dark throughout the region. However, the brightness may also change, for example, in a gradation according to the line-of-sight trajectory. In the example of FIG. 10, the graphics generation unit 701 changes the brightness of the line-of-sight range and the virtual viewpoint image. However, these brightness changes may be performed by the image generation unit 205.

[0067] As described above, in the second embodiment, information related to a person's gaze, such as the gaze, gaze trajectory, and gaze range, is displayed graphically and presented to the user. As a result, according to the second embodiment, not only is it possible to suppress drastic image movements in the virtual viewpoint image corresponding to the gaze of the person who is the subject of gaze extraction, as in the first embodiment, but it is also possible to present to the user in an easy-to-understand manner where the gaze is directed at a given time.

[0068] Furthermore, the various graphics displays described in the second embodiment may be combined in part or in whole as appropriate. For example, graphics displays such as a trajectory object 901, a range object 903, and a window object 904 may be performed while changing the brightness of the virtual viewpoint image as shown in Fig. 10. Furthermore, which object is displayed may be switched as appropriate by a user instruction. Furthermore, in the present embodiment, the image processing device 700 has been described as including the image generation unit 205, the display unit 206, and the graphics generation unit 701, but is not limited to this. For example, at least one of the image generation unit 205, the display unit 206, and the graphics generation unit 701 may be configured to be connected to the outside of the image processing device 700 as another device. Furthermore, the image generation unit 205, the display unit 206, and the graphics generation unit 701 may be the same device or different devices. That is, similar to the image processing device 100 in the first embodiment described above, the image processing device 700 at least determines virtual camera parameters in the estimation unit 203 and performs processing to output the virtual camera parameters to a processing unit that generates a virtual viewpoint image or to another device.

[0069] The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. The above-described embodiments are merely examples of specific implementations of the present invention, and the technical scope of the present invention should not be construed as being limited by these embodiments. In other words, the present invention can be implemented in various forms without departing from its technical concept or main features. [Explanation of symbols]

[0070] 100: Image processing device, 101: CPU, 102: Main storage device, 103: Auxiliary storage device, 104: External I / F, 105: GPU, 201: Input unit, 202: Line-of-sight acquisition unit, 203: Estimation unit, 204: Data storage unit, 205: Image generation unit, 206: Display unit

Claims

1. an acquisition means for acquiring positions of subjects photographed by a plurality of photographing devices; a determining means for determining a position of a virtual viewpoint related to a virtual viewpoint image based on the position of the subject; a generating means for generating the virtual viewpoint image based on a plurality of captured images acquired by the plurality of image capturing devices and the position of the virtual viewpoint; a control means for controlling the virtual viewpoint image so that a graphic representing the line of sight or field of view of the subject is superimposed on the virtual viewpoint image; 1. An image processing device comprising:

2. 2. The image processing apparatus according to claim 1, wherein the acquiring means acquires information about the subject's line of sight or field of view based on the orientation of the subject's face.

3. 3. The image processing device according to claim 1, wherein the acquiring means acquires information about the line of sight or field of view of the subject based on a trajectory of movement of the line of sight of the subject within a predetermined time range.

4. 4. The image processing apparatus according to claim 3, wherein the acquisition means acquires an instruction for the predetermined time range, and updates the predetermined time range in accordance with the instruction.

5. An image processing device described in any one of claims 1 to 4, characterized in that the control means controls so that an object representing the center of the subject's gaze is superimposed as a graphic representing the subject's gaze.

6. The image processing device described in Claim 5, characterized in that the control means controls the graphic representing the subject's line of sight so that it gradually disappears over time.

7. An image processing device described in any one of claims 1 to 4, characterized in that the control means controls so that a frame is superimposed within the virtual viewpoint image as a graphic representing the field of view of the subject.

8. An image processing device described in any one of claims 1 to 4, characterized in that the control means controls the area of the virtual viewpoint image corresponding to the field of view of the subject to be cropped as a graphic representing the field of view of the subject, and superimposed on the virtual viewpoint image.

9. An image processing device described in any one of claims 1 to 8, characterized in that the subject is a person photographed by the multiple photographing devices.

10. An acquisition step of acquiring positions of subjects photographed by a plurality of photographing devices; a determining step of determining a position of a virtual viewpoint associated with a virtual viewpoint image based on the position of the subject; a generating step of generating the virtual viewpoint image based on a plurality of captured images acquired by the plurality of image capturing devices and the position of the virtual viewpoint; a control step of controlling the virtual viewpoint image so that a graphic representing the line of sight or field of view of the subject is superimposed on the virtual viewpoint image; An image processing method comprising:

11. A program for causing a computer to function as the image processing device according to any one of claims 1 to 9.

Images