Information processing device, information processing method, and program

Abstract

To enable setting a virtual viewpoint appropriate for the scene.SOLUTION: An information processing device includes: acquisition means for acquiring subject information indicating a position of a subject captured by a plurality of imaging devices, in a virtual space that corresponds to a position in a real space, region information indicating a region in a three-dimensional space, and interest position information indicating an interest position associated with the region; and determination means for, in a case where the subject exists within the region, determining a position of a virtual viewpoint corresponding to a virtual viewpoint image and a line-of-sight direction from the virtual viewpoint based on the subject information and the interest position information.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an information processing device that sets a virtual viewpoint corresponding to a virtual viewpoint image. [Background technology]

[0002] A technology that generates an image (virtual viewpoint image) from any virtual camera (virtual viewpoint) based on a user's operation by using the multiple images obtained by installing multiple cameras in different positions and capturing images simultaneously is attracting attention. With such a technology, for example, it becomes possible to watch highlight scenes of soccer or basketball from various angles, which can give the user a high sense of realism compared to normal images.

[0003] In order to facilitate the operation of the virtual viewpoint, Patent Document 1 describes a method for determining the position and orientation of the virtual viewpoint based on position information of a first subject of interest and position information of a second subject that is to be included in the virtual viewpoint image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-171436 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when generating a virtual viewpoint image of a subject performing complex movements, it was not possible to determine a virtual viewpoint appropriate for the scene. For example, in a basketball game, when trying to generate a virtual viewpoint image focusing on a certain player, the offense and defense frequently switch, so the appropriate virtual viewpoint position changes with the change in the scene, and it was not possible to set the position of the virtual viewpoint and the line of sight from the virtual viewpoint appropriate for the scene.

[0006] An object of the present disclosure is to enable a virtual viewpoint to be set that is appropriate for a scene. [Means for solving the problem]

[0007] The information processing device according to the present disclosure has the following configuration: an acquisition means for acquiring position information indicating a position of a subject included in an imaging area captured by a plurality of imaging devices, an identification means for identifying one of a plurality of positions of interest based on a moving direction of the subject, and a determination means for determining a position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and a line of sight direction from the virtual viewpoint based on the position of the subject and the position of the subject identified by the identification means. Effect of the Invention

[0008] According to the present disclosure, a virtual viewpoint suitable for a scene can be set. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a configuration of an image processing system according to a first embodiment. [Diagram 2] 5A to 5C are diagrams for explaining the process of tracking a subject position in the first embodiment. [Diagram 3] 6A to 6C are diagrams for explaining the process of assigning an identifier to a subject position in the first embodiment. [Figure 4] 4 is a diagram showing an example of a representative position of a subject in the first embodiment. FIG. [Diagram 5] 11 is a flowchart showing a process in which the tracking unit 106 tracks the position of a subject in the first embodiment. [Figure 6] 10 is a flowchart showing a process in which the interest subject determining unit 110 in the first embodiment determines an interest subject. [Figure 7] 5A to 5C are diagrams showing examples of the velocity vectors and average velocity vectors of each subject in the first embodiment. [Figure 8] 4 is a diagram showing an example of the position of a virtual viewpoint generated by a viewpoint generating unit 109 in the first embodiment. FIG. [Figure 9]13 is a flowchart showing a process in which the interest subject determining unit 110 in the second embodiment determines an interest subject. [Figure 10] 13 is a diagram showing an example of a target of interest and a determination region in the second embodiment. FIG. [Figure 11] 13A to 13C are diagrams illustrating an example of the movements of a subject, a target of interest, and a virtual viewpoint in the second embodiment. [Figure 12] 2 is a block diagram showing an example of a hardware configuration of an information processing device 3. FIG. [Figure 13] 13 is a flowchart showing an example of processing by a viewpoint generating unit 109 in the third embodiment. [Figure 14] FIG. 13 is a diagram showing an example of a viewpoint position determination region in the third embodiment. [Figure 15] FIG. 13 is a diagram showing an example of the angle of view of a virtual viewpoint generated by a viewpoint generation unit 109 in the third embodiment. [Figure 16] FIG. 13 is a diagram showing an example of a table showing combinations of viewpoint position determination regions and objects of interest in the third embodiment. [Figure 17] FIG. 13 is a diagram showing an example of a virtual viewpoint generated by a viewpoint generating unit 109 in the fourth embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment 1> (System configuration and operation of image processing device) The image processing system is a system that generates a virtual viewpoint image that represents a scene from a specified virtual viewpoint based on a plurality of images based on images captured by a plurality of imaging devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by a user, and includes, for example, an image corresponding to a viewpoint selected by a user from a plurality of candidates. In addition, although the present embodiment will mainly describe a case where the virtual viewpoint image is a moving image, the virtual viewpoint image may also be a still image.

[0011] The viewpoint information used to generate the virtual viewpoint image is information indicating the position and orientation (line of sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter indicating the three-dimensional position of the virtual viewpoint and a parameter indicating the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter indicating the size of the visual field (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting a video of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points.

[0012] The image processing system has a plurality of imaging devices that capture images of an imaging area from a plurality of directions. The imaging area is, for example, a stadium where sports such as soccer and karate are held, or a stage where a concert or a play is held. The plurality of imaging devices are installed at different positions so as to surround such an imaging area, and capture images synchronously. Note that the plurality of imaging devices do not need to be installed all around the imaging area, and may be installed only in a part of the periphery of the imaging area depending on restrictions on the installation location, etc. Also, the number of imaging devices is not limited to the example shown in the figure, and for example, when the imaging area is a soccer stadium, about 30 imaging devices may be installed around the stadium. Also, imaging devices with different functions, such as a telephoto camera and a wide-angle camera, may be installed.

[0013] In this embodiment, the multiple imaging devices are cameras each having an independent housing and capable of capturing images from a single viewpoint. However, this is not limited to this, and two or more imaging devices may be configured in the same housing. For example, a single camera having multiple lens groups and multiple sensors and capable of capturing images from multiple viewpoints may be installed as the multiple imaging devices.

[0014] FIG. 1 is a block diagram showing the configuration of an image processing system in the first embodiment. The image processing system includes, for example, an imaging unit 101, a synchronization unit 102, a three-dimensional shape estimation unit 103, a shape extraction unit 104, an identification setting unit 105, a tracking unit 106, a subject position calculation unit 107, a storage unit 108, a viewpoint generation unit 109, a subject of interest determination unit 110, and a time indication unit 111. The image processing system further includes an image generation unit 112 and a display unit 113. In this embodiment, the shape extraction unit 104, the identification setting unit 105, the tracking unit 106, and the subject position calculation unit 107 are collectively referred to as a subject position detection unit 114. The image processing system may be configured by one image processing device, or may be a system configured by multiple image processing devices. For example, the viewpoint generation unit 109, the subject of interest determination unit 110, the time indication unit 111, and the subject position detection unit 114 may be included in a single information processing device different from the image processing device, or the storage unit 108 may be configured by a dedicated server device. In the following description, the image processing system will be described as one image processing device.

[0015] The imaging units 101 capture images in synchronization with each other based on a synchronization signal from the synchronization unit 102. Then, the imaging units 101 output the captured images to the three-dimensional shape estimation unit 103. Note that the imaging units 101 are installed so as to surround an imaging area including a subject in order to capture the subject from multiple directions.

[0016] The synchronization unit 102 outputs a synchronization signal to the multiple image capture units 101 .

[0017] The three-dimensional shape estimation unit 103 generates a silhouette image of the subject using the input captured images from multiple viewpoints. Furthermore, it generates a 3D model of the subject using a volume intersection method or the like. The three-dimensional shape estimation unit 103 also outputs the generated 3D model of the subject and the captured images to the storage unit 108 and the shape extraction unit 104. Here, the subject is an object for which a 3D model is to be generated, and includes people and items handled by people.

[0018] The shape extraction unit 104 extracts a part of the 3D model of the subject acquired from the three-dimensional shape estimation unit 103. By extracting a part of the 3D model of the subject, it is possible to reduce the possibility that, when there are 3D models of multiple subjects, the subjects cannot be identified due to contact or adjacency between them. Furthermore, the shape extraction unit 104 generates a two-dimensional image by projecting the extracted 3D model of the subject from one direction. In this embodiment, the two-dimensional image is generated by projecting from a direction perpendicular to the floor surface. Then, extraction information of the 3D model of the subject is extracted from the generated two-dimensional image and transmitted to the tracking unit 106 and the identification setting unit 105.

[0019] The identification setting unit 105 assigns an identifier to the extracted information obtained from the shape extraction unit 104. The identifier assigned to each extracted shape is transmitted to the tracking unit .

[0020] In response to an identifier input from the identification setting unit 105, the tracking unit 106 assigns the identifier to each extracted shape as an initial state. After that, the tracking unit 106 tracks the extracted shape to which the identifier has been assigned. Each tracked extracted shape and its identifier are transmitted to the subject position calculation unit 107.

[0021] The subject position calculation unit 107 calculates the representative position of each subject (the position of the subject) using each extracted shape and identifier acquired from the tracking unit 106. The calculated representative position of the subject is transmitted to the storage unit 108.

[0022] The storage unit 108 saves and stores the following data group as data (virtual viewpoint material data) used to generate a virtual viewpoint image. In this embodiment, the data used to generate a virtual viewpoint image is specifically a 3D model of a subject and a captured image input from the three-dimensional shape estimation unit 103. In addition, the data used to generate a virtual viewpoint image includes camera parameters such as the position and orientation and optical characteristics of each imaging unit, and subject information acquired by the subject position detection unit 114. It is assumed that a 3D model of the background and a background texture image are saved (stored) in the storage unit 108 in advance as data used to generate the background of the virtual viewpoint image. Furthermore, the 3D models stored in the storage unit 108 are stored in association with a type. Here, the type is set for each imaging target, and for example, in the case of basketball, it is a player, a ball, a referee, etc. It is to be noted that the type of each 3D model may be specified by the user, or may be automatically set according to a preset condition. In addition, the storage unit 108 stores a candidate target of interest, which will be described later. The candidate of interest may be a background model or a specific three-dimensional coordinate (position of interest) in a three-dimensional space (virtual space). The candidate of interest is described below as being set in advance by a user, but may be automatically set by the device. The candidate of interest automatically set by the device may be, for example, the center position of an image capturing area by a plurality of image capturing locations or a subject at that center position, or the position of a boundary area between an image capturing area and other areas or a subject at that position. In this embodiment, the candidate of interest is described as a background model. The background model is, for example, a stationary object such as a basketball goal, a soccer goal, or a goal line in track and field. The background model also includes venues such as a basketball court or a soccer court.

[0023] The viewpoint generating unit 109 generates a virtual viewpoint based on the subject information detected by the subject position detecting unit 114 and the position information of the subject of interest output by the subject of interest determining unit 110. Specifically, as shown in FIG. 8, the viewpoint generating unit 109 sets the position of the virtual viewpoint at a position at a predetermined distance from the subject in a direction rotated at a predetermined rotation angle θ from a straight line connecting the subject and the center of gravity of a background model that is the subject of interest, with the position of the subject specified by the user as the center. Then, the viewpoint generating unit 109 sets the line of sight and angle of view from the virtual viewpoint so that the subject and the background model are included in the field of view of this virtual viewpoint (virtual camera). Then, the viewpoint generating unit 109 generates virtual viewpoint information indicating parameters such as the position of the virtual viewpoint, the line of sight from the virtual viewpoint, and the angle of view of the virtual viewpoint, and outputs the virtual viewpoint information to the image generating unit 112. Note that the position of the virtual camera may be set on a straight line connecting the position of the subject specified by the user and the center of gravity of the background model that is the subject of interest. Also, the rotation angle θ may be set by the user. The viewpoint generating unit 109 also includes a viewpoint operating unit, which is a physical user interface such as a joystick or an operation button (not shown), and a display unit for displaying a virtual viewpoint image. The viewpoint operating unit can set parameters for generating virtual viewpoint information, such as the distance between a subject included in the virtual viewpoint image and the virtual viewpoint, the direction from the virtual viewpoint, and the height of the virtual viewpoint. In response to changes in these parameters, the virtual viewpoint image is updated as needed by the image generating unit 112 (described later) and displayed on the display unit. This display unit may share the display unit 113 (described later), or may include a separate display device. The virtual viewpoint information output by the viewpoint generating unit 109 includes information equivalent to external parameters of the camera, such as the position and direction of the virtual viewpoint, information equivalent to internal parameters of the camera, such as the focal length and angle of view, and time information specifying the shooting time to be played back.

[0024] The subject of interest determining unit 110 determines one of a plurality of pre-set subject of interest candidates as the subject of interest based on a subject of interest determining process described below, and outputs the determined subject of interest to the viewpoint generating unit 109 .

[0025] A time indication unit 111 generates time information for indicating the time of a virtual viewpoint image to be generated by an image generation unit 112 described below, and outputs the information to the viewpoint generation unit 109 and the interest subject determination unit 110. The time indication unit 111 is provided with at least one user interface, such as a physical user interface including a plurality of buttons and a jog dial (not shown) or a GUI, and is capable of changing the time at which an image is generated by a user operation.

[0026] The image generating unit 112 acquires material data at the time of shooting from the storage unit 108 based on the time information included in the input virtual viewpoint information. The image generating unit 112 generates a virtual viewpoint image at the set virtual viewpoint using the 3D model of the subject and the captured image from the acquired material data, and outputs it to the display unit 113. The 3D model of the subject generated by the three-dimensional shape estimating unit 103 is colored using the captured image. Specifically, an imaging device whose position and line of sight from the imaging device are close to the position of the set virtual viewpoint and the line of sight from the virtual viewpoint are identified among the multiple imaging devices. The number of imaging devices identified may be multiple or may be one imaging device. Next, the captured images captured by the identified imaging device are blended to generate a texture image. At this time, the weight of the blend may be increased in the order of the imaging device closest to the position of the virtual viewpoint and the line of sight from the virtual viewpoint. By the above process, the color of the captured image captured in the real space can be set as the color of the 3D model of the subject, and a 3D model closer to the real subject can be generated.

[0027] The display unit 113 is a display means for displaying the image input from the image generating unit 112. The display unit 113 is configured with a display, a head mounted display (HMD), or the like.

[0028] (How to track the subject position) Next, a method for tracking the three-dimensional position of a subject in this embodiment will be described.

[0029] First, three-dimensional shape estimation section 103 generates a 3D model of the subject, and outputs the generated 3D model to storage section 108 and also outputs the generated 3D model to shape extraction section 104.

[0030] FIG. 2 is a diagram for explaining the tracking process of the subject position in the first embodiment. The shape extraction unit 104 cuts out the lower part of the 3D model of the subject as shown in FIG. 2(b) from the 3D model of the subject as shown in FIG. 2(a). In this embodiment, the cutout is made from the bottom surface of the circumscribed rectangular parallelepiped of the 3D model of the subject to a predetermined height (e.g., a height equivalent to 50 cm). For example, as shown in FIG. 2(c), when one of the subjects is standing and one of the subjects is away from the floor surface of the shooting area by jumping or the like, the 3D model of the subject is cut out as shown in FIG. 2(d). That is, for both of the 3D models of the subjects, the 3D model is cut out from the part equivalent to the feet to a predetermined height. Note that the area cut out from the 3D model is not limited to the lower part of the 3D model. For example, the central part of the 3D model of the subject (e.g., an area from 50 cm to 100 cm from the bottom surface of the circumscribed rectangular parallelepiped of the 3D model) may be cut out.

[0031] Next, as shown in FIG. 2(e), the shape extraction unit 104 projects the cut-out 3D model onto the floor to generate a two-dimensional image. In this embodiment, the projected image is a binary image in which the cut-out 3D model part is white and the other part is black. The shape extraction unit 104 divides this two-dimensional image into independent regions and obtains circumscribing rectangles 201 to 204 shown in FIG. 2(e). The shape extraction unit 104 outputs vertex information of this circumscribing rectangle as the extracted shape of the extracted 3D model. Specifically, the shape extraction unit 104 converts the vertex information of the circumscribing rectangle into the same coordinate system and unit as the three-dimensional space of the shooting area and outputs it. In addition, to determine the independent shape in the shape extraction unit 104, a method such as continuous component analysis is used for the projected two-dimensional image. By using such a method, the shape extraction unit 104 can divide the 3D model into individual regions.

[0032] FIG. 3 is a diagram for explaining the process of assigning an identifier to the subject position in the first embodiment. The identification setting unit 105 assigns an identifier to the extracted shape output by the shape extraction unit 104. Specifically, the identification setting unit 105 calculates the distance between each extracted shape, and assigns an identifier according to the distance between the extracted shapes. For example, as shown in FIG. 3(a), the identification setting unit 105 assigns the same identifier to extracted shapes whose distance between the extracted shapes is less than a predetermined distance (solid line arrow), and assigns a different identifier to extracted shapes whose distance between the extracted shapes is equal to or greater than the predetermined distance (dashed line arrow). The threshold value of the predetermined distance used as the criterion for judgment is preferably a distance equivalent to the width of the subject's legs when standing. In this embodiment, the explanation will be given assuming that the threshold value of the predetermined distance is set to 50 cm.

[0033] The identification setting unit 105 displays the assigned identifiers on a display unit provided in the identification setting unit 105 by a graphical user interface (GUI) as shown in FIG. 3(b). The user operates the image processing system while viewing this GUI. Specifically, the identification setting unit 105 displays the current identifier assignment (identifier assignment in the initial state) on the GUI by distinguishing it with at least one of characters and color coding. In FIG. 3(b), the identification setting unit 105 displays each identifier by using both characters and color coding. The user checks the GUI and checks whether the desired identifier assignment is set as the initial state. If the desired identifier assignment is not set, the user instructs the subject to change its standing position or to close its legs, and repeats this until the desired assignment is set. Alternatively, the user operates the image processing system via the GUI and issues a change instruction so that the desired identifier assignment is set. If the desired identifier assignment is set, the user presses, for example, a decision button (initial identifier decision button) on a graphical user interface as shown in FIG. 3(b). In response to this operation, the identification setting unit 105 decides the initial state identifier. Furthermore, the type of each subject may be set for the identifier in the initial state. It is assumed that this type is set for each imaging target (for example, baseball, live performance, etc.), and in this embodiment, a basketball is assumed, and one of the types of a player, a referee, or a ball is set for each subject. It is assumed that this type is set by user input, but is not limited to this. For example, in the initial state, a condition may be set in advance such that a subject corresponding to an extracted shape that does not have other extracted shapes within a predetermined distance is set as a ball, and the type may be set at the same time as the identifier is determined. Then, the identification setting unit 105 outputs information indicating the identifier and type determined for each extracted shape to the tracking unit 106.

[0034] In response to receiving an identifier from the identification setting unit 105, the tracking unit 106 assigns the identifier to each extracted shape as an initial state. After that, the tracking unit 106 tracks the extracted shape to which the identifier has been assigned. The identifier assigned to the extracted shape during tracking is not the identifier determined by the identification setting unit 105, but an identifier determined based on the tracking result of the position of each extracted shape by the tracking unit 106. In tracking the extracted shape, the tracking unit 106 tracks the extracted shape based on the position of each extracted shape at the time immediately before the shooting time of the extracted shape, the identifier of each extracted shape, and information on the subject position input from the subject position calculation unit described later. The specific processing of tracking by the tracking unit 106 will be described later. The tracking unit 106 assigns an identifier to each extracted shape at the time based on the tracking result, and outputs each extracted shape to the subject position calculation unit 107.

[0035] Fig. 4 is a diagram showing the representative position of a subject in embodiment 1. The subject position calculation unit 107 obtains a representative position for each extracted shape to which an identifier obtained from the tracking unit 106 has been assigned. For example, as shown in Fig. 4, the subject position calculation unit 107 obtains a position indicating each extracted shape group for each extracted shape group to which the same identifier has been assigned, such as representative position 401 and representative position 402. In this embodiment, the representative position is the center position of the extracted shape group.

[0036] However, since this representative position is affected by shape estimation errors and fluctuations in boundary parts when the shape is cut out by the shape extraction unit 104, the position may fluctuate at each time even if the subject is stationary. Therefore, in this embodiment, the subject position calculation unit 107 performs processing such as low-pass filtering or moving average in the time direction on the center position information at each time to generate position information in which high-frequency components are suppressed. Then, the subject position calculation unit 107 outputs the position information of the representative position together with the identifier as the position of the subject to the tracking unit 106. In addition, the subject position calculation unit 107 records (stores) information in which information on the time when the subject was photographed is added to the position information of the representative position in the storage unit 108 as position information of the subject (subject information).

[0037] As shown in the above process, a 3D model of the subject is generated from an image captured in real space, and a representative position of the 3D model is obtained, thereby making it possible to identify the position of the subject in virtual space that corresponds to the position of the subject in real space.

[0038] (Tracking process by tracking unit 106) FIG. 5 is a flowchart showing a process in which the tracking unit 106 tracks the position of a subject in the first embodiment.

[0039] In step S501, the tracking unit 106 receives an input from the identification setting unit 105 and acquires the identifier and type of each extracted shape.

[0040] In step S 502 , the tracking unit 106 acquires the extracted shape that is next input from the shape extraction unit 104 .

[0041] In step S503, the tracking unit 106 assigns the identifiers and types acquired from the identification setting unit 105 to each of the acquired extracted shapes, and outputs each extracted shape to which the identifiers and types have been assigned, to the subject position calculation unit 107.

[0042] In step S504, the subject position calculation unit 107 calculates the subject position from the group of extracted shapes to which the same identifier is assigned, and outputs the subject position to the tracking unit .

[0043] The above processing in steps S501 to S504 corresponds to the initialization processing.

[0044] The subsequent steps S505 to S509 are executed every time while the imaging unit 101 is imaging the subject. When the imaging process of the subject by the imaging unit 101 is completed, the process of step S509 is completed and the process of this flowchart is terminated.

[0045] In step S505, tracking unit 106 acquires the extracted shape input from shape extraction unit 104 and the subject position at the previous time (previous time) calculated by subject position calculation unit 107. The previous time is, for example, the shooting time of the extracted shape generated one frame before the extracted shape currently being processed. Here, for comparison, the current time is also referred to as the current time. Here, the current time refers to the shooting time of the image used to generate the extracted shape currently being processed.

[0046] In step S506, when the subject position at the previous time overlaps with the representative position of each extracted shape at the current time, the tracking unit 106 assigns to the extracted shape an identifier that is assigned to the subject position that overlaps with the representative position. Here, in step S506, when the representative position of one extracted shape overlaps with multiple subject positions, the tracking unit 106 assigns to the extracted shape an identifier indicating "undeterminable" at the current time. This is because, for example, multiple extracted shapes with different identifiers assigned may overlap at the current time, such as when two subjects are close to each other, so an identifier indicating "undeterminable" is assigned in the processing of this step. The extracted shape to which an identifier including an identifier indicating "undeterminable" is assigned is subjected to the processing of S509 described later.

[0047] In step S507, if the representative position of an extracted shape to which an identifier has not yet been assigned overlaps with the extracted shape of the previous time, the tracking unit 106 assigns the identifier assigned to the extracted shape of the previous time to the extracted shape of the current time.

[0048] In step S508, if there is another extracted shape to which an identifier has already been assigned at the current time within a predetermined range from an extracted shape to which an identifier has not yet been assigned, the tracking unit 106 assigns the identifier assigned to the other extracted shape. The predetermined range is preferably a range equivalent to the width of the subject's legs when standing. For example, the predetermined range is a range of a radius of 50 cm from the center of the extracted shape. Here, if there are multiple other extracted shapes to which an identifier has been assigned within a predetermined range from a certain extracted shape, the tracking unit 106 assigns the identifier of the extracted shape that is closest among the other extracted shapes to the extracted shape. For an extracted shape to which an identifier has not been assigned at the stage of completing the processing up to step S508, the tracking unit 106 determines that the extracted shape is not a tracking target. In this case, the tracking unit 106 does not output the extracted shape determined to be not a tracking target to the subject position calculation unit 107.

[0049] In step S509, the tracking unit 106 outputs to the subject position calculation unit 107 the extracted shape to which an identifier has been assigned in the processes of steps S506 to S508, and the identifier assigned thereto.

[0050] In step S510, a control unit (not shown) determines whether or not the imaging process of the subject by the imaging unit 101 has ended. If it is determined that the imaging process of the subject by the imaging unit 101 has not ended, the process of step S508 is executed. If it is determined that the imaging process of the subject by the imaging unit 101 has ended, the process of this flowchart ends.

[0051] Note that the processes from step S506 to step S508 are performed for each extracted shape. By repeating the processes from step S506 to step S509, the identifier set by the identification setting unit 105 is linked to the extracted shape at each time. Using this identifier, the subject position calculation unit 107 can distinguish between each subject and calculate the subject position.

[0052] In addition, when the tracking unit 106 assigns an "undeterminable" identifier to the extracted shape, there is a possibility that some of the identifiers set in the initial setting will not be assigned at a certain time. In such a case, the subject position calculation unit 107 does not update the subject information having the same identifier as the identifier not assigned to the extracted shape. As a result, even if the extracted shapes overlap due to multiple subjects approaching each other, the multiple subject information will not be in the same position. In this case, the multiple subject positions are maintained at their respective positions up to the previous time. After that, when the subjects move away from each other and the overlapped multiple extracted shapes are separated again, an identifier is assigned to each extracted shape based on the most recent subject position. In other words, in response to the elimination of the overlap of the multiple extracted shapes, the updating of each subject information is resumed.

[0053] By the above processing, the image processing system can track each subject and obtain the position information of each subject even when multiple subjects are in the shooting area. Furthermore, by the above processing, the image processing system can track each subject even when the subjects approach or move away from each other, causing overlapping or separation in the generated 3D model.

[0054] (Interest Determination Process) 6 is a flowchart showing a process in which the interest target determining unit 110 in the first embodiment determines an interest target. In this embodiment, an example of determining an interest target when generating a virtual viewpoint image of basketball will be described. In this case, the interest target candidates are the basketball goals of both teams, and position information of the interest target candidates is acquired in advance from the storage unit 108. This process is performed for each frame. It is assumed that a specific background model is determined as the interest target candidate from among multiple background models based on a separate user operation.

[0055] 7A and 7B are diagrams showing the velocity vectors and average velocity vectors of each subject in this embodiment. As shown in Fig. 7A, coordinate axes x and y are defined, the center of the court is set as the origin (0,0), and the basketball goal is located on the x-axis.

[0056] In step S601, the subject of interest determining unit 110 acquires subject information based on the time specified by the time specifying unit 111 from the subject information detected by the subject position detecting unit 114 and recorded in the storage unit 108. At this time, not only the subject information at the specified time but also the subject information for several to several tens of frames before and after that time is acquired. The following steps S602 to S605 are processed for each subject.

[0057] In step S602, it is determined whether a certain subject is a subject to be calculated. Specifically, it is determined whether to perform subsequent processing based on whether the type corresponding to the identifier for identifying the subject included in the position information matches the calculation target information. The calculation target information is information that specifies the calculation target of the interest target determination process, and is information that specifies the type corresponding to the subject's identifier. For example, calculation target information is generated such that subjects whose type corresponding to the identifier is a player are calculated, and is recorded in advance in the storage unit 108 or the interest target determination unit 110. In this case, subjects whose types are umpire and ball can be excluded from the interest target determination process by excluding them from the calculation target.

[0058] In step S603, a filtering process is performed on the position information of the object determined to be the object of calculation. Specifically, position information for multiple frames acquired in S601 is averaged to calculate position information. This makes it possible to reduce blurring due to detection errors of the object position detection unit 114.

[0059] In step S604, subject information from a time earlier than the time of the subject to be calculated, for example, one second ago and several to several tens of frames (predetermined time) before and after that time is acquired and an average value is calculated.

[0060] In step S605, the position information after averaging at a past time is subtracted from the position information after averaging at the current time, and then the difference in position information between the current time and the past time is divided by the time difference between the current time and the past time, here by one second, to obtain the position deviation (position change) at the current time unit time. The magnitude of this position deviation at the unit time is the moving speed, and the direction of the vector on the two-dimensional plane is the moving direction of the subject. The above steps S602 to S605 are performed for each subject. When the process has been completed for all subjects, proceed to S606.

[0061] In step S606, an average velocity vector of the obtained velocity vectors of all the objects to be calculated is calculated. Fig. 7(b) shows the calculated average velocity vector of all the objects to be calculated. In this embodiment, the component of the goal direction contained in this average velocity vector is obtained. Specifically, since the goal direction is the positive and negative directions of the x-axis, the direction and magnitude of the vector obtained by orthogonally projecting the average velocity vector onto the vector (1,0) are obtained.

[0062] In step S607, it is determined whether the average velocity vector calculated in S606 satisfies the target of interest determination condition. It is assumed that the target of interest determination condition differs for each imaging target. In this embodiment, it is determined whether the vector obtained by projecting the average velocity vector in the goal direction satisfies the target of interest determination condition. In the initial state where the target of interest has not been determined, the process proceeds to S608 regardless of whether the target of interest determination condition is satisfied. In this embodiment, the target of interest determination condition is whether the direction of the vector obtained by projecting the average velocity vector in the goal direction is toward the current target of interest, and whether the magnitude of the vector is greater than a predetermined magnitude. When the current target of interest has been determined, a change in the target of interest is determined based on the direction and magnitude of the orthogonally projected vector. If the orthogonally projected vector is toward the positive direction, is greater than or equal to a predetermined magnitude, and the current target of interest is not the x-axis positive side goal 71, the process proceeds to S608. If the vector is toward the negative direction, is greater than or equal to a predetermined magnitude, and the currently focused goal is not the x-axis negative side goal 72, the process proceeds to S608. If neither of the two conditions is met, the current subject of interest is left unchanged, and the process of determining the subject of interest in the current frame ends.

[0063] In step S608, one of the multiple candidates for the target of interest is determined based on the average velocity vector. In the initial state where the target of interest has not been determined, a straight line is extended in the direction of the average velocity vector, and the target of interest closest to the line is identified. In addition, if there are multiple targets of interest closest to the line, the target of interest closest to the position of the subject of interest is identified. Alternatively, the initial value of the target of interest may be set in advance by a user operation. In this embodiment, a determination is made as to whether or not to change the target of interest based on the direction and magnitude of the orthogonally projected vector. If the orthogonally projected vector is oriented in the positive direction, is equal to or greater than a predetermined magnitude, and the current target of interest is not the x-axis positive side goal 71, the goal 71 is determined (specified) as the target of interest. In addition, if the vector is oriented in the negative direction, is equal to or greater than a predetermined magnitude, and the currently focused goal is not the x-axis negative side goal 72, the goal 72 is determined as the target of interest.

[0064] By performing the processes of S601 to S608 for each frame, the movements of all subjects are detected and a subject of interest is determined. The subject of interest determining unit 110 outputs information on the determined subject of interest to the viewpoint generating unit 109.

[0065] As described above, the viewpoint generator 109 places a virtual viewpoint in a direction rotated by a predetermined angle from the straight line connecting the subject and the object of interest. When the object of interest switches from goal 71 to goal 72, the viewpoint does not switch immediately, but rather calculates the current angle and the angle after the switch, and rotates by this angle difference over a predetermined time to complement the virtual viewpoint. This allows the viewer to correctly recognize that the object of interest he or she is currently focusing on has switched when the object of interest switches.

[0066] In this way, the present disclosure makes it possible to detect a change in offense and defense from the movements of all subjects in sports such as basketball, and to automatically generate a virtual viewpoint that captures the goal of the attacking team by the viewpoint generating unit 109 accordingly. This makes it possible to create a virtual viewpoint image suited to the scene, while tracking a subject specified by the user and including it in the virtual viewpoint image.

[0067] In this embodiment, the goal direction component of the average velocity vector is calculated in S606, but which direction component is calculated varies depending on the imaging subject and the candidate interest subject. For example, when the imaging subject is a 100m sprint, the straight line from the start to the goal is set as the x-axis direction. In this way, which direction component of the average velocity vector is calculated varies depending on the imaging subject and the candidate interest subject. Therefore, for example, when a combination of multiple candidate interest subjects is determined, a predetermined component direction may be set and stored in advance for the combination.

[0068] In this embodiment, the component of the average velocity vector in the goal direction is calculated in S606, but this is not limiting. The target of interest may be identified from the direction of the average velocity vector. Specifically, a straight line is set in the direction of the average velocity vector, and the target of interest closest to the straight line is identified. There is also a possibility that there are multiple target of interest candidates closest to the straight line. In this case, the target of interest closest to the position of the subject of interest is identified. This makes it possible to identify one target of interest from multiple target of interest candidates.

[0069] (Other aspects of embodiment 1) In this embodiment, subject position detection means based on shape estimation results has been described as subject position detection unit 114, but the present disclosure is not limited to this method of detecting the subject position. For example, a configuration may be used in which a position sensor such as a GPS is attached to the player and the sensor value is acquired, or a configuration may be used in which the subject position is detected using image recognition technology from images obtained by multiple imaging means.

[0070] In addition, in this embodiment, the subject of interest determination unit 110 is configured to calculate the speed and movement direction for each frame when generating a virtual viewpoint image, but this is not necessarily limited to this. For example, the subject position detection unit 114 may detect the position, calculate the speed vector of each subject, and record it in the storage unit 108. In this case, the subject of interest determination unit 110 may obtain the speed vector of each subject from the storage unit 108 and determine the subject of interest.

[0071] In this embodiment, the filtering process of the position information is performed by averaging, but the present invention is not limited to this. For example, a low-pass filter such as an IIR filter or an FIR filter may be used. However, in the case of a configuration in which the speed or the like is calculated each time, if a low-pass filter is used, the value becomes incorrect when the time at which the virtual viewpoint image is reproduced is changed discontinuously. Therefore, in such a configuration, it is desirable to obtain information on nearby times and then average the information as described above.

[0072] In this embodiment, basketball is used as an example, but it may be applied to sports such as soccer and rugby. For example, in the case of soccer, the game progresses more slowly than in basketball, and the players may be stationary for a relatively long period of time. In such a case, it is possible to determine which side of the field there are more subjects (players), and determine the subject of interest by combining the result of this determination with the velocity vectors of the subjects.

[0073] <Embodiment 2> In the first embodiment, the object of interest is determined from the average velocity vector of all subjects that are players. However, the object of interest may be determined based on the movement of one player. For example, when generating a virtual viewpoint image that follows a baseball runner, the object of interest may be determined by focusing on the movement of the runner. In this embodiment, an example in which the imaging object is a baseball will be described. In this embodiment, the block diagram is the same as in FIG. 1, and components that are not particularly described are the same as those in the first embodiment, so their description will be omitted.

[0074] 9 is a flowchart showing the process of determining an interest subject by the interest subject determining unit 110 in embodiment 2. This process is performed for each frame.

[0075] 10 is a diagram showing an example of an interest subject candidate and an interest subject determination region (predetermined region) in embodiment 2. In this embodiment, a home base 1005 and each of a first base 1006, a second base 1007, and a third base 1008 in baseball are regarded as interest subject candidates.

[0076] In step S901, the subject of interest determination unit 110 sets a first subject of interest determination region 1001, a second subject of interest determination region 1002, a third subject of interest determination region 1003, and a fourth subject of interest determination region 1004 by a user operation. At this time, a subject of interest candidate to be linked to each subject of interest determination region is set. Specifically, a home base 1005 and a first base 1006 are linked to the first subject of interest determination region 1001. Similarly, a first base 1006 and a second base 1007 are linked to the second subject of interest determination region 1002, a second base 1007 and a third base 1008 are linked to the third subject of interest determination region 1003, and a third base 1008 and a home base 1005 are linked to the fourth subject of interest determination region 1004. In addition, the subject of interest determination unit 110 acquires position information of the subject of interest candidates linked to the subject of interest determination regions, and obtains a directional vector linking the two subject of interest candidates. For example, in the case of the second subject of interest determination region 1002, a unit direction vector connecting first base 1006 and second base 1007 is obtained. At this time, the unit direction vector is a vector indicating the direction of second base. Note that position information of these subject of interest determination regions and subject of interest candidates is managed as condition information, and may be stored in subject of interest unit 110 or in storage unit 108. Note that S901 is not performed for every frame, but is performed only for the first frame in which a subject of interest is determined. Note that the process is not limited to this, and S901 may be performed at any timing of the user's choice.

[0077] In step S902, the subject of interest determining unit 110 acquires subject information based on the time specified by the time specifying unit 111 from the subject information detected by the subject position detecting unit 114 and recorded in the storage unit .

[0078] In step S903, the velocity vector of the subject is obtained, as in embodiment 1. At this time, as in embodiment 1, filter processing is performed on the obtained position information of the subject.

[0079] In step S904, the subject of interest determination unit 110 determines in which of the four pre-set subject of interest determination regions the subject is present. In this embodiment, a case will be described in which the subject is present in the second subject of interest determination region 1002. Next, a plurality of subject of interest candidates corresponding to the second subject of interest determination region 1002 are identified. In this embodiment, first base 1006 and second base 1007 are the subject of interest candidates.

[0080] In step S905, it is determined whether the velocity vector calculated in S903 satisfies the target of interest determination condition. In the initial state where the target of interest has not been determined as in the first embodiment, the process proceeds to S906 regardless of whether the target of interest determination condition is satisfied. The target of interest determination unit 110 obtains the direction and magnitude of the vector obtained by orthogonally projecting the subject's velocity vector onto the unit direction vector associated with the target of interest determination region. In this embodiment, the target of interest determination condition is whether the magnitude of the vector obtained by orthogonal projection is greater than a predetermined magnitude. If the magnitude of the vector obtained by orthogonal projection is greater than the predetermined magnitude, the process proceeds to S906, and if the magnitude of the vector obtained by orthogonal projection is equal to or less than the predetermined magnitude, the process ends.

[0081] In step S906, one of the multiple candidate objects of interest identified in S904 is determined based on the average velocity vector. The processing in the initial state where the object of interest has not been determined is the same as that in the first embodiment. If the magnitude of the vector obtained by orthogonal projection is larger than a predetermined value and the direction of the vector obtained by orthogonal projection is the same as the unit direction vector, toward second base, second base 1007 is determined as the object of interest. If the direction of the vector obtained by orthogonal projection is opposite to the unit direction vector, first base 1006 is determined as the object of interest. By performing the processing of S902 to S906 for each frame, a virtual viewpoint that focuses on an appropriate object of interest while following the movement of the player can be generated.

[0082] FIG. 11 is a diagram showing the movements of a subject, an object of interest, and a virtual viewpoint in the second embodiment. Specifically, the movement of a player (runner) and switching of an object of interest will be described with reference to FIG. 11. Here, a state will be described in which a runner 1101 is in the vicinity of first base 1006 as shown in FIG. 11(a). When the runner 1101 is in the vicinity of first base 1006, the runner has passed through the first object of interest determination region 1001 before reaching the first base 1006, and therefore the object of interest currently being observed is first base 1006. At this time, the runner 1101 takes a lead and enters the second object of interest determination region 1002, but the object of interest remains first base 1006 due to the low speed of movement. The virtual viewpoint generated by the viewpoint generating unit 109 at this time is a virtual viewpoint 1102 that shows the runner 1101 and the first base 1006 from diagonally behind the runner 1101 as shown in FIG. 11(a). Next, when the runner 1101 starts running to advance and his speed exceeds a predetermined speed, the target of interest is switched to second base 1007. As a result, the virtual viewpoint 1102 following the runner 1101 rotates around the runner 1101 as indicated by the dashed arrow 1103 in FIG. 11(b) and automatically transitions to a virtual viewpoint 1102 that shows second base 1007 over the shoulder of the runner 1101 while following the runner 1101. Also, for example, if the runner 1101 tries to return to first base 1006 while advancing, the target of interest is switched to first base 1006 and transitions to a virtual viewpoint 1102 that includes the runner 1101 and the first base 1006. Next, if the runner 1101 successfully advances and then attempts to advance to third base 1008, the runner 1101 enters the third target of interest determination region 1003. As a result, the moving direction and speed are determined based on the unit direction vector defined by second base 1007 and third base 1008, which is linked to the third object of interest determination region 1003. As a result, the object of interest to be observed becomes third base 1008 upon entering the third object of interest determination region 1003, and the virtual viewpoint 1102 moves to a position that includes the runner 1101 and third base 1008, as shown in FIG.

[0083] By applying the present disclosure in this manner, it becomes possible to identify an appropriate target of interest from among multiple target of interest candidates based on the position and speed information of the player, and the viewpoint generation unit 109 can generate an appropriate virtual viewpoint based on the identified target of interest. This makes it possible to create appropriate camera work suited to the situation even without an operator to operate the virtual viewpoint.

[0084] (Other aspects of embodiment 2) As in the first embodiment, the subject position detection unit 114 may use another position detection method such as GPS or image recognition.

[0085] In this embodiment, the virtual viewpoint generated by the viewpoint generating unit 109 has been described as a virtual viewpoint that follows the subject from behind, but is not limited to this. Any virtual viewpoint may be generated using information on the subject being watched and the target of interest to be noted. For example, the viewpoint generating unit 109 may be configured to place a virtual viewpoint near the base that is the target of interest, wait for the subject to be tracked, and create camerawork so that the subject fits within the angle of view. In this case, it is desirable that, as the target of interest is switched, the placement of the virtual viewpoint transitions to near the base that is the target of interest while capturing the subject to be tracked from the front.

[0086] In this embodiment, the virtual viewpoint is automatically generated by the viewpoint generation unit 109, and an operator is not required, but this is not necessarily limited to the above. For example, the distance of the virtual viewpoint generated by the viewpoint generation unit 109 from the player, the height at which it is placed, the angle of view, etc. may be controlled by the operator through a user interface (not shown). In this way, the viewpoint generation unit 109 is responsible for fitting the subject and the object of interest within the angle of view, but the operator can control the composition, such as the angle of view, which is more desirable.

[0087] In this embodiment, the present invention is applied to baseball, but it may also be applied to softball or other sports in which the virtual viewpoint to be included in the virtual viewpoint is switched depending on the position of the player.

[0088] In this embodiment, the candidate object of interest is used as a background model, and the position and orientation of the virtual viewpoint are determined based on the position of the subject specified by the user and the center of gravity of the object of interest, but this is not limited thereto. For example, a three-dimensional position different from the center of gravity may be associated with the background model of the candidate object of interest. In this case, the position and orientation of the virtual viewpoint are determined based on the position of the subject specified by the user and the three-dimensional position corresponding to the object of interest. This process makes it possible to easily set a virtual viewpoint suitable for a scene when a goal net is located at a position different from the center of gravity of the background model, such as a basketball goal. The three-dimensional position corresponding to the background model may be set at the same height as the center of gravity of the subject. For example, when the object of interest is a baseball base, it is present on the floor surface, and there is a risk of generating a virtual viewpoint image corresponding to a virtual viewpoint facing the floor surface. Therefore, by adjusting the height of the three-dimensional position corresponding to the background model, it is possible to easily set a virtual viewpoint suitable for a scene.

[0089] <Embodiment 3> In the first and second embodiments, the position of the virtual viewpoint is set at a position a predetermined distance away from the subject, in a direction rotated at a predetermined rotation angle θ from a straight line connecting the position of the subject and the center of gravity of a background model that is an object of interest, with the position of the subject specified by the user as the center. In this embodiment, the direction of rotation is determined according to the position of the subject, thereby setting a position of the virtual viewpoint more suitable for the scene. In this embodiment, the process of generating a virtual viewpoint video of a basketball is described as an example. The object of interest is a basketball goal.

[0090] In this embodiment, the direction of rotation is determined according to the viewpoint position determination area in which the subject is located. The viewpoint position determination area is an area in which a direction of rotation is set around the subject with respect to a straight line connecting the subject and the object of interest. The viewpoint position determination area is assumed to be generated by a user operation, and in this embodiment, a three-dimensional area in a three-dimensional space and an object of interest area corresponding to the three-dimensional area and a direction of rotation are specified by a user device (not shown) and stored in the storage unit 108. In this embodiment, the viewpoint position determination area is associated with an object of interest, and a direction of rotation is determined and stored for each object of interest. Note that a plurality of viewpoint position determination areas may be stored as one combination, or a plurality of objects of interest may be associated with one viewpoint position determination area. In this embodiment, a plurality of objects of interest corresponding to each of a plurality of viewpoint position inversion areas are determined, and a direction of rotation corresponding to the combination is stored as a table.

[0091] FIG. 16 is a diagram showing an example of a table showing a combination of a viewpoint position determination area and an object of interest in this embodiment. This table is created based on an input from a user device (not shown) and stored in the storage unit 108. In this embodiment, it is assumed that a virtual viewpoint video of a basketball is generated, and a table showing a combination of two viewpoint position determination areas and two objects of interest is stored. In this embodiment, the combination of the viewpoint position determination area 1403 and the object of interest 1401 has a clockwise rotation direction. The combination of the viewpoint position determination area 1403 and the object of interest 1402 has a counterclockwise rotation direction. The combination of the viewpoint position determination area 1404 and the object of interest 1401 has a counterclockwise rotation direction. The combination of the viewpoint position determination area 1404 and the object of interest 1402 has a clockwise rotation direction.

[0092] (Viewpoint generation processing) Fig. 13 is a flowchart showing an example of the processing of the viewpoint generating unit 109 in this embodiment. Note that the processing in this flowchart is executed after the processing of determining the subject of interest shown in Fig. 6 is completed. Therefore, the description of the processing of determining the subject of interest from among a plurality of subject of interest candidates is omitted.

[0093] In step S1301, the viewpoint generation unit 109 acquires area information indicating a viewpoint position determination area from the storage unit 108. In this embodiment, a table indicating a combination of the viewpoint position determination area and the object of interest stored in the storage unit 108 is acquired. Next, area information indicating an area in a three-dimensional space of the viewpoint position determination area recorded in the table, and object of interest information (interest position information) indicating the position of the object of interest recorded in the table are acquired. Specifically, the viewpoint position determination area 1403 and the viewpoint position determination area 1404 shown in FIG. 14 are acquired from the storage unit 108.

[0094] In step S1302, the viewpoint generating unit 109 acquires position information of a subject designated by a user operation. Hereinafter, the designated subject will be referred to as a tracking target. Furthermore, not only subject information at the designated time but also subject information for several to several tens of frames before and after the designated time is acquired.

[0095] In step S1303, the viewpoint generation unit 109 performs a filter process on the position information of the tracking target. Specifically, the viewpoint generation unit 109 calculates position information by averaging the position information for multiple frames acquired in S1302. This makes it possible to reduce blurring due to detection errors of the subject position detection unit 114.

[0096] In step S1304, the viewpoint generation unit 109 specifies a viewpoint position determination area in which the target to be tracked is located. An explanation will be given taking FIG. 14 as an example. In FIG. 14(a), a target of interest 1401, a target of interest 1402, a viewpoint position determination area 1403, a viewpoint position determination area 1404, a target to be tracked 1405, and a virtual viewpoint 1406 are depicted. At this time, the viewpoint position determination area in which the target to be tracked 1405 is located is specified as the viewpoint position determination area 1403. In other words, the target to be tracked 1405 is located within the viewpoint position determination area 1403.

[0097] In step S1305, the viewpoint generating unit 109 generates virtual viewpoint information by locating a camera at a location a predetermined distance away from the target to be followed and in a direction rotated a predetermined angle θ from a straight line connecting the target of interest. The direction of rotation at this time is determined based on the target of interest and the viewpoint position determination area. In the example shown in FIG. 14(a), when the target of interest is a basketball goal 1401 and the target to be followed 1405 is in the viewpoint position determination area 1403, the virtual viewpoint 1406 is located at a position rotated 20 degrees clockwise. In this embodiment, the description will be given assuming that the rotation is 20 degrees, but this is not limited to this. The rotation angle θ is determined based on a user operation. Note that the clockwise direction may be the negative direction and the counterclockwise direction may be the positive direction, or the clockwise direction may be the positive direction and the counterclockwise direction may be the negative direction. In FIG. 14, the clockwise direction is expressed as the negative direction and the counterclockwise direction is expressed as the positive direction. 14B, the object of interest remains the basketball goal 1401, and when the object to be followed 1405 is located in the viewpoint position determination area 1404 on the opposite side, the virtual viewpoint 1407 is positioned with the specified angle θ being +20 degrees, which is a counterclockwise rotation angle. When the object of interest becomes the basketball goal 1402 and is in the viewpoint position determination area 1403, the rotation angle θ of the virtual viewpoint 1408 is +20 degrees, and when the object of interest is in the viewpoint position determination area 1404, the rotation direction is changed to -20 degrees for the virtual viewpoint 1409.

[0098] When the target of interest and the viewpoint position determination area are switched, the predetermined angle θ is not switched discontinuously, but is smoothly switched by interpolating the virtual viewpoint before and after the switching over a period of about 0.3 to 1 second as shown in Fig. 14(c). In the example of Fig. 14(c), when the tracked target 1405 moves from the viewpoint position determination area 1403 to the viewpoint position determination area 1404, the virtual viewpoint transitions from the virtual viewpoint 1406 to the virtual viewpoint 1407 while the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint are interpolated. If the target of interest and the viewpoint position determination area are not switched, the virtual viewpoint information changed in S1305 is used as it is.

[0099] By performing the processes in S1302 to S1305 described above for each frame, virtual viewpoint information based on the positions of the target of interest and the target to be tracked is generated and output to the image generating unit 112.

[0100] By performing the above process, it is possible to automatically set the position of the virtual viewpoint and the line of sight from the virtual viewpoint so that the court can be viewed while the target to be followed and the target of interest are included in the angle of view, based on the three factors of the target to be followed, the target of interest, and the viewpoint position determination area. Specifically, when the position of the virtual viewpoint and the line of sight from the virtual viewpoint are determined based on the two factors of the target to be followed and the position of interest, a virtual viewpoint image shown in FIG. 15(b) is generated, and the plays of other players are not displayed, which may make it difficult to grasp the scene. On the other hand, when the position of the virtual viewpoint and the line of sight from the virtual viewpoint are determined based on the three factors of the target to be followed, the position of interest, and the viewpoint position determination area, a virtual viewpoint image shown in FIG. 15(a) is generated, and the plays of other players are displayed more frequently, making it easy to grasp the scene.

[0101] In the present embodiment, the rotation direction is defined for the combination of the viewpoint position determination region and the target of interest, but the present invention is not limited to this. For example, the rotation direction and the rotation angle may be defined. Alternatively, the positional relationship between the target of interest and the subject in the virtual viewpoint image may be defined. Specifically, the position of the subject may be defined to be on the left side of the position of interest in the virtual viewpoint image, or the position of the subject may be defined to be on the right side of the position of interest. In other words, the positional relationship between the position of the target of interest and the position of the subject based on the optical axis of the virtual viewpoint may be defined. The position of the virtual viewpoint and the line of sight from the virtual viewpoint are determined so as to correspond to the defined positional relationship.

[0102] In this embodiment, the viewpoint generating unit 109 automatically generates a virtual viewpoint, but it is preferable that the parameters for generating the virtual viewpoint can be set by the user. For example, the configuration may include a viewpoint operating unit, which is a physical user interface such as a joystick or an operation button (not shown), and a display unit for displaying a virtual viewpoint image. In addition, the target to be tracked and the target of interest may be dynamically changed by the viewpoint operating unit. In this case, it is preferable that the parameters include the distance between the target to be tracked and the virtual viewpoint, the rotation angle, and the height of the virtual viewpoint. In response to changes in these parameters, the image generating unit 112 updates the virtual viewpoint image at any time and displays it on the display unit. This display unit may share the display unit 7, or may be provided with a separate display device. In addition, the height of the virtual viewpoint may be increased based on the height of the target to be tracked, in the case of a tall target to be tracked. In this case, the height at which the virtual viewpoint is gazed upon may also be changed.

[0103] In this embodiment, no particular reference is made to switching of the target to be followed, but it is preferable to select from a list of players in a user interface, or to click or tap on a player on the video to switch the selection. In addition, each player may be assigned to an operation button, and the player may be switched by pressing the button. When switching players, the virtual viewpoint may be moved instantly to switch the camera, but it is preferable to create a viewpoint that is interpolated between the viewpoint that follows the player before the switch and the viewpoint that follows the player after the switch. This creates a viewpoint in which the virtual camera flies between the players, making it easier for the viewer to recognize it as a continuous image. It is also easier to recognize that the target to be followed has been switched, and it is also easier to grasp the positional relationship before and after the switch. It is also preferable that the interpolation at this time is an interpolation with easing with acceleration and deceleration rather than linear interpolation.

[0104] In this embodiment, the viewpoint position determination area is set as two areas, but it is not limited to this. For example, it may be a sector-shaped area based on the angle from the goal, which is the object of interest. In this case, the rotation angle θ may be increased based on the area (angle).

[0105] In this embodiment, the viewpoint position determination regions 1403 and 1404 are spaced apart. For example, if the viewpoint position determination regions are adjacent to each other, when the tracking target moves back and forth across the boundary between the regions, the virtual viewpoint will repeatedly switch viewpoints accordingly, resulting in an undesirable image in which the image vibrates left and right. In contrast, by providing a region where no determination is performed between the viewpoint position determination regions, the angle of the virtual viewpoint is not changed during this period, and the rotation angle θ set up until that point is maintained. This provides a substantial hysteresis, suppressing continuous switching of the virtual viewpoint. On the other hand, even if the viewpoint position determination regions are adjacent to each other, a configuration may be used in which a delay is provided to suppress continuous switching.

[0106] <Embodiment 4> In the first to third embodiments, the description is given assuming that there is one target of interest, but this is not necessarily limited to this. Two or more targets of interest may be used, and the viewpoint generating unit 109 may generate a viewpoint. In this embodiment, an example will be described with reference to FIG. 17 in which two targets of interest, a basketball goal 1702 and a basketball 1703, are treated as targets of interest. First, different weights are set in advance for the basketball goal 1702 and the basketball 1703. Specifically, the basketball goal 1702 is set as the main target of interest, and the basketball 1703 is set as the secondary target of interest. As a difference between the main and secondary targets, the main target (basketball goal 1702) is used as one of the criteria for determining the position of the virtual viewpoint 1704 together with the target to be tracked 1701, as shown in the first embodiment. In this case, if the secondary target of interest (basketball 1703) is within the angle of view determined by the target to be tracked 1701 and the main target of interest (basketball goal 1702), a predetermined rotation angle (here, θ1) is maintained. Next, when the secondary target of interest (basketball 1703) reaches a position about 10 to 20 percent from the edge of the angle of view (dashed line in FIG. 17), the viewpoint generation unit 109 changes the virtual viewpoint so that the secondary target of interest fits within the angle of view of the virtual viewpoint. Specifically, the angle of the virtual viewpoint is changed to the direction of the secondary target of interest, the virtual viewpoint is moved away from the target to be tracked, the focal length of the virtual viewpoint is shortened to make it wider, and so on, to obtain an angle of view in which both the main and secondary targets of interest fit. In FIG. 17(a), the ball within the angle of view is moving and is about to go out of the angle of view. At this time, as shown in FIG. 17(b), the viewpoint generation unit 109 rotates the basketball 1703 so that the rotation angle when going around the target to be tracked 1701 is θ2 so that the basketball 1703 fits within the angle of view of the virtual viewpoint, thereby fitting both the main and secondary targets of interest within the angle of view.

[0107] However, in the positional relationship between the target and the main and secondary targets of interest, there is a possibility that the target may be abnormally wide-angled or may be too far away from the target. In this case, it is desirable to set a limit on the distance away from the target and the angle of view, and if the secondary target of interest moves outside the angle of view within the limit, not to track it any further.

[0108] In this embodiment, basketball is used as an example, but it may be applied to sports such as soccer or rugby, or may be applied to baseball or softball, with a configuration in which a viewpoint position determination region is set for each space between bases.It may also be applied to other sports in which the virtual viewpoint to be included in the virtual viewpoint is switched depending on the position of the player.

[0109] (Other configurations) In the above embodiment, each processing unit shown in Fig. 1 is configured as hardware. However, the processing performed by each processing unit shown in these figures may be configured as a computer program.

[0110] FIG. 12 is a block diagram showing an example of the hardware configuration of a computer applicable to the indirect position estimation device according to each of the above embodiments.

[0111] The CPU 1201 controls the entire computer using computer programs and data stored in the RAM 1202 and the ROM 1203, and executes the processes described above as being performed by the indirect position estimation device according to each of the above embodiments. That is, the CPU 1201 functions as each of the processing units shown in FIG.

[0112] The RAM 1202 has an area for temporarily storing computer programs and data loaded from an external storage device 1206, and data acquired from the outside via an I / F (interface) 1207. Furthermore, the RAM 1202 has a work area used when the CPU 1201 executes various processes. That is, the RAM 1202 can be allocated as a frame memory, for example, or can provide various other areas as appropriate.

[0113] The ROM 1203 stores setting data for the computer, a boot program, and the like. The operation unit 1204 is composed of a keyboard, a mouse, and the like, and can be operated by a user of the computer to input various instructions to the CPU 1201. The output unit 1205 displays the results of processing by the CPU 1201. The output unit 1205 is composed of, for example, a liquid crystal display. For example, the viewpoint generation unit 109 is composed of the operation unit 1204, and the display unit 113 is composed of the output unit 1205.

[0114] The external storage device 1206 is a large-capacity information storage device, such as a hard disk drive. The external storage device 1206 stores an operating system (OS) and computer programs for causing the CPU 1201 to realize the functions of each unit shown in Fig. 1. Furthermore, the external storage device 1206 may store image data to be processed.

[0115] Computer programs and data stored in the external storage device 1206 are loaded into the RAM 1202 as appropriate under the control of the CPU 1201, and become the subject of processing by the CPU 1201. A network such as a LAN or the Internet, and other devices such as a projector or display device can be connected to the I / F 1207, and the computer can obtain and transmit various information via the I / F 1207. In the first embodiment, the imaging unit 101 is connected to this, and inputs captured images and controls each of them. 1208 is a bus connecting the above-mentioned units.

[0116] The operation of the above-mentioned configuration is controlled mainly by the CPU 1201 as described in the above embodiment.

[0117] A computer program for implementing all or part of the control in the present embodiment may be provided to an image processing system or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the image processing system or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0118] The disclosure of this embodiment includes the following configurations, methods, and programs.

[0119] (Configuration 1) An acquisition means for acquiring subject information indicating a position in a virtual space corresponding to a position in a real space of a subject imaged by a plurality of imaging devices, area information indicating an area in the virtual space, and position of interest information indicating a position of interest associated with the area; a determination means for determining, when the subject is present within the region, a position of a virtual viewpoint corresponding to a virtual viewpoint image and a line of sight direction from the virtual viewpoint based on the subject information and the interest position information; 13. An information processing device comprising:

[0120] (Configuration 2) The information processing device described in Configuration 1, characterized in that the determination means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint based on a straight line passing through the virtual space position of the subject and the position of interest.

[0121] (Configuration 3) The region and the position of interest correspond to a predetermined rotation direction; The information processing device described in configuration 2, characterized in that, when the subject is present within the area, the determination means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint to a position rotated in the predetermined rotation direction with the virtual space position of the subject as the center with respect to the straight line.

[0122] (Configuration 4) The information processing device described in Configuration 3, wherein the determination means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint to a position rotated a predetermined angle in the predetermined rotation direction with respect to the straight line as the center of the virtual space position of the subject.

[0123] (Configuration 5) The information processing device according to configuration 4, wherein the predetermined angle is determined based on a user operation.

[0124] (Configuration 6) The information processing device according to configuration 1, wherein the position of interest is set based on a user operation.

[0125] (Configuration 7) The information processing device according to configuration 1, wherein the position of interest corresponds to a background model.

[0126] (Configuration 8) The information processing device according to configuration 7, wherein the background model is a stationary object.

[0127] (Configuration 9) The information processing device according to configuration 8, which identifies the background model as a basketball goal.

[0128] (Configuration 10) The information processing device according to configuration 1, wherein the determination means determines the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint so that the subject and the position of interest are included in the virtual viewpoint image.

[0129] (Method) An acquisition step of acquiring subject information indicating a position in a virtual space corresponding to a position in a real space of a subject imaged by a plurality of imaging devices, area information indicating an area in a three-dimensional space, and position of interest information indicating a position of interest associated with the area; a determining step of determining, when the subject is present within the region, a position of a virtual viewpoint corresponding to a virtual viewpoint image and a line of sight direction from the virtual viewpoint based on the subject information and the interest position information; 13. An information processing method comprising:

[0130] (Program) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 10. [Explanation of symbols]

[0131] 104 Shape extraction part 105 Identification setting section 106 Tracking Department 107 Subject position calculation section 109 Viewpoint Generation Unit 110 Interest Determination Division 111 Time indicator

Claims

1. an acquisition means for acquiring subject information indicating a position in a virtual space corresponding to a position in a real space of a subject imaged by a plurality of imaging devices, area information indicating an area in the virtual space, and position of interest information indicating a position of interest associated with the area; a determination means for determining, when the subject is present within the region, a position of a virtual viewpoint corresponding to a virtual viewpoint image and a line of sight direction from the virtual viewpoint based on the subject information and the interest position information; 13. An information processing device comprising:

2. 2 . The information processing apparatus according to claim 1 , wherein the determining means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint based on a straight line passing through the position of the subject in the virtual space and the position of interest.

3. the region and the location of interest are associated with a predetermined rotational orientation; The information processing device according to claim 2, characterized in that, when the subject is present within the area, the determination means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint to a position rotated in the predetermined rotation direction with the virtual space position of the subject as the center with respect to the straight line.

4. 4. The information processing device according to claim 3, wherein the determining means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint to a position rotated by a predetermined angle in the predetermined direction with the virtual space position of the subject as a center with respect to the straight line.

5. The information processing apparatus according to claim 4 , wherein the predetermined angle is determined based on a user operation.

6. The information processing apparatus according to claim 1 , wherein the position of interest is set based on a user operation.

7. The information processing apparatus according to claim 1 , wherein the position of interest corresponds to a background model.

8. 8. The information processing apparatus according to claim 7, wherein the background model is a stationary object.

9. The information processing apparatus according to claim 8 , wherein the background model is identified as a basketball goal.

10. 2 . The information processing apparatus according to claim 1 , wherein the determining means determines the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint so that the subject and the position of interest are included in the virtual viewpoint image.

11. an acquisition step of acquiring subject information indicating a position in a virtual space corresponding to a position in a real space of a subject imaged by a plurality of imaging devices, area information indicating an area in a three-dimensional space, and position of interest information indicating a position of interest associated with the area; a determining step of determining, when the subject is present within the region, a position of a virtual viewpoint corresponding to a virtual viewpoint image and a line of sight direction from the virtual viewpoint based on the subject information and the interest position information; 13. An information processing method comprising:

12. A program for causing a computer to function as the information processing device according to any one of claims 1 to 10.

Images