Image processing apparatus, image processing method, and program

Abstract

To set a virtual viewpoint suitable for a scene.SOLUTION: An information processing apparatus has: acquisition means that acquires position information indicating the position of a subject included in an imaging area, images of which are picked up by a plurality of imaging apparatuses; specification means that specifies one position of interest of a plurality of positions of interest on the basis of the direction of movement of the subject; and determination means that, on the basis of the position of interest specified by the specification means and the position of the subject, determines the position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and the direction of line of sight from the virtual viewpoint.SELECTED DRAWING: Figure 1

Description

Technical Field

[0001] The present disclosure relates to an image processing apparatus that generates a virtual viewpoint image.

Background Art

[0002] Techniques for installing multiple cameras at different positions and synchronously shooting, and generating an image (virtual viewpoint image) from an arbitrary virtual camera (virtual viewpoint) based on user operations using the multiple images obtained by shooting have attracted attention. According to such a technique, for example, it becomes possible to view highlight scenes of soccer or basketball from various angles, and a higher sense of presence can be given to the user compared to normal images.

[0003] In order to facilitate the operation of the virtual viewpoint, Patent Document 1 describes a method for determining the position and orientation of the virtual viewpoint based on the position information of the first subject to be focused on and the position information of the second subject to be included in the virtual viewpoint image.

Prior Art Documents

Patent Documents

[0004]

Patent Document 1

Summary of the Invention

Problems to be Solved by the Invention

[0005] However, when generating a virtual viewpoint image of a subject performing complex movements, a virtual viewpoint suitable for the scene could not be determined. For example, in basketball, when generating a virtual viewpoint image focusing on a certain player, since offense and defense frequently switch, the position of the appropriate virtual viewpoint changes as the scene changes, and the position of the virtual viewpoint suitable for the scene and the line-of-sight direction from the virtual viewpoint could not be set.

[0006] An object of the present disclosure is to enable setting a virtual viewpoint suitable for a scene.

Means for Solving the Problem

[0007] The information processing apparatus according to the present disclosure has the following configuration. That is, acquisition means for acquiring position information indicating the position of a subject included in an imaging area imaged by a plurality of imaging devices, specifying means for specifying one of a plurality of positions of interest based on the moving direction of the subject, and based on the position of interest specified by the specifying means and the position of the subject, determining means for determining the position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and the line-of-sight direction from the virtual viewpoint.

Effect of the Invention

[0008] According to the present disclosure, a virtual viewpoint suitable for a scene can be set.

Brief Description of the Drawings

[0009]

Figure 1

Figure 2

Figure 3

Figure 4

Figure 5

Figure 6

Figure 7

Figure 8

Figure 9

Figure 10

Figure 11

Figure 12

Embodiments for Carrying Out the Invention

[0010] <Embodiment 1> (System Configuration and Operation of the Image Processing Apparatus) The image processing system is a system that generates a virtual viewpoint image representing a scene from a specified virtual viewpoint based on a plurality of images obtained by a plurality of imaging devices and a specified virtual viewpoint. The virtual viewpoint image in the present embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by the user. For example, an image corresponding to a viewpoint selected by the user from a plurality of candidates is also included in the virtual viewpoint image. In addition, in the present embodiment, the case where the virtual viewpoint image is a moving image will be mainly described, but the virtual viewpoint image may be a still image.

[0011] The viewpoint information used for generating the virtual viewpoint image is information indicating the position and orientation (line-of-sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including parameters representing the three-dimensional position of the virtual viewpoint and parameters representing the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter representing the size (angle of view) of the field of view of the virtual viewpoint. In addition, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting the moving image of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points.

[0012] The image processing system has a plurality of imaging devices that image an imaging area from a plurality of directions. The imaging area is, for example, an arena where sports such as soccer or karate are held, or a stage where concerts or plays are held. The plurality of imaging devices are installed at different positions so as to surround such an imaging area and perform imaging synchronously. Note that the plurality of imaging devices do not necessarily have to be installed over the entire circumference of the imaging area, and depending on limitations on the installation location, etc., they may be installed only in a part of the periphery of the imaging area. Also, the number of imaging devices is not limited to the example shown in the figure. For example, when the imaging area is a soccer stadium, about 30 imaging devices may be installed around the stadium. Also, imaging devices with different functions such as a telephoto camera and a wide-angle camera may be installed.

[0013] Note that the plurality of imaging devices in the present embodiment are assumed to be cameras each having an independent housing and capable of imaging from a single viewpoint. However, it is not limited to this, and two or more imaging devices may be configured in the same housing. For example, a single camera having a plurality of lens groups and a plurality of sensors and capable of imaging from a plurality of viewpoints may be installed as the plurality of imaging devices.

[0014] FIG. 1 is a block diagram showing the configuration of the image processing system according to Embodiment 1. The image processing system includes, for example, an imaging unit 101, a synchronization unit 102, a three-dimensional shape estimation unit 103, a shape extraction unit 104, an identification setting unit 105, a tracking unit 106, a subject position calculation unit 107, a storage unit 108, a viewpoint generation unit 109, an object of interest determination unit 110, and a time indication unit 111. Further, the image processing system includes an image generation unit 112 and a display unit 113. In this embodiment, the shape extraction unit 104, the identification setting unit 105, the tracking unit 106, and the subject position calculation unit 107 are collectively referred to as a subject position detection unit 114. The image processing system may be configured by one image processing device or a system configured by a plurality of image processing devices. For example, the viewpoint generation unit 109, the object of interest determination unit 110, the time indication unit 111, and the subject position detection unit 114 may be included in one information processing device different from the image processing device, or the storage unit 108 may be configured by a dedicated server device. In the following description, the image processing system will be described as one image processing device.

[0015] The imaging unit 101 performs imaging in synchronization with each other based on the synchronization signal from the synchronization unit 102. Then, the imaging unit 101 outputs the captured imaging image to the three-dimensional shape estimation unit 103. The imaging unit 101 is installed so as to surround the imaging area including the subject in order to photograph the subject from a plurality of directions.

[0016] The synchronization unit 102 outputs a synchronization signal to the plurality of imaging units 101.

[0017] The three-dimensional shape estimation unit 103 generates a silhouette image of the subject using the captured images from a plurality of input viewpoints. Further, a 3D model (3D model) of the subject is generated using a volume intersection method or the like. The three-dimensional shape estimation unit 103 also outputs the generated 3D model of the subject and the captured image to the storage unit 108 and the shape extraction unit 104. Here, the subject is an object to be the generation target of the 3D model, and includes a person and an article handled by the person.

[0018] The shape extraction unit 104 extracts a part of the 3D model of the subject obtained from the three-dimensional shape estimation unit 103. By extracting a part of the 3D model of the subject, when there are 3D models of a plurality of subjects, it is possible to reduce the possibility that the subjects cannot be identified due to contact or adjacency between the subjects. Further, the shape extraction unit 104 generates a two-dimensional image obtained by projecting the extracted 3D model of the subject from one direction. In the present embodiment, a two-dimensional image projected from a direction perpendicular to the floor surface is generated. Then, extraction information of the 3D model of the subject is extracted from the generated two-dimensional image and transmitted to the tracking unit 106 and the identification setting unit 105.

[0019] The identification setting unit 105 assigns an identifier to the extraction information obtained from the shape extraction unit 104. The identifier assigned to each extracted shape is transmitted to the tracking unit 106.

[0020] The tracking unit 106 assigns the identifier to each extracted shape as an initial state in response to receiving the identifier from the identification setting unit 105. Thereafter, the tracking unit 106 tracks the extracted shape to which this identifier is assigned. Each tracked extracted shape and the identifier are transmitted to the subject position calculation unit 107.

[0021] The subject position calculation unit 107 calculates the representative position (position of the subject) of each subject using each extracted shape and the identifier obtained from the tracking unit 106. The calculated representative position of the subject is transmitted to the storage unit 108.

[0022] The storage unit 108 stores and remembers the following data groups as data (virtual viewpoint material data) used for generating virtual viewpoint images. In this embodiment, the data used for generating virtual viewpoint images are specifically the 3D model of the subject and the captured images input from the three-dimensional shape estimation unit 103. Further, the data used for generating virtual viewpoint images includes camera parameters such as the position and orientation and optical characteristics of each imaging unit, and the subject position information acquired by the subject position detection unit 114. It is assumed that a 3D model of the background and a background texture image are stored (remembered) in the storage unit 108 in advance as data used for generating the background of the virtual viewpoint image. Furthermore, the 3D models stored in the storage unit 108 are stored with their types associated respectively. Here, the type is set for each imaging target. For example, in the case of basketball, it is players, balls, referees, etc. Note that the type of each 3D model may be specified by the user or automatically set according to preset conditions. Also, the storage unit 108 stores the candidate of the object of interest to be described later. The candidate of the object of interest may be a background model or a specific three-dimensional coordinate (position of interest) in the three-dimensional space. Although the candidate of the object of interest is described hereinafter as being set in advance by the user, the device may set it automatically. The candidate of the object of interest automatically set by the device may be, for example, the center position of the imaging area by a plurality of imaging locations or the subject at the center position, or the position of the boundary area between the imaging area and other areas or the subject at that position. In this embodiment, the candidate of the object of interest is described as being a background model. The background model is, for example, the goal of basketball, the goal of soccer, or the goal line of track and field.

[0023] The viewpoint generation unit 109 generates a virtual viewpoint based on the subject position information detected by the subject position detection unit 114 and the position information of the object of interest output by the object of interest determination unit 110. Specifically, as shown in FIG. 8, the viewpoint generation unit 109 rotates a predetermined rotation angle θ from the straight line connecting the position of the subject specified by the user and the center-of-gravity position of the background model that is the object of interest, and sets the position of the virtual viewpoint at a position separated from the subject by a predetermined distance in the rotated direction. Then, the viewpoint generation unit 109 sets the line-of-sight direction and the angle of view from the virtual viewpoint so that the subject and the background model are included in the field of view of this virtual viewpoint (virtual camera). Then, the viewpoint generation unit 109 generates virtual viewpoint information indicating parameters such as the position of these virtual viewpoints, the line-of-sight direction from the virtual viewpoint, and the angle of view of the virtual viewpoint, and outputs the virtual viewpoint information to the image generation unit 112. Note that the position of the virtual camera may be set on the straight line connecting the position of the subject specified by the user and the center-of-gravity position of the background model that is the object of interest. Also, the rotation angle θ may be set by the user. The viewpoint generation unit 109 also includes a viewpoint operation unit, which is a physical user interface such as a joystick or operation buttons (not shown), and a display unit for displaying the virtual viewpoint image. The viewpoint operation unit can set parameters for generating virtual viewpoint information, such as the distance between the subject included in the virtual viewpoint image and the virtual viewpoint, the direction from the virtual viewpoint, and the height of the virtual viewpoint. In response to the change of this parameter, the virtual viewpoint image is updated by the image generation unit 112 described later and displayed on the display unit. This display unit may share the display unit 113 described later, or may be configured to include a separate display device. The virtual viewpoint information output by the viewpoint generation unit 109 includes information corresponding to external parameters of the camera, such as the position and orientation of the virtual viewpoint, information corresponding to internal parameters of the camera, such as the focal length and the angle of view, and time information specifying the shooting time to be played back.

[0024] Based on the object-of-interest determination process described later, the object-of-interest determination unit 110 determines one from among a plurality of preset object-of-interest candidates as the object of interest, and outputs it to the viewpoint generation unit 109.

[0025] The time indication unit 111 generates time information for indicating the time of the virtual viewpoint image to be generated by the image generation unit 112 described later, and outputs it to the viewpoint generation unit 109 and the object of interest determination unit 110. The time indication unit 111 includes at least one of a physical user interface such as a plurality of buttons or a jog dial (not shown) or a GUI, and can change the time for generating a video by a user operation.

[0026] Based on the time information included in the input virtual viewpoint information, the image generation unit 112 acquires the material data at the shooting time from the storage unit 108. Using the 3D model of the subject and the captured image among the acquired material data, the image generation unit 112 generates a virtual viewpoint image at the set virtual viewpoint and outputs it to the display unit 113.

[0027] The display unit 113 is a display means for displaying the video input from the image generation unit 112. The display unit 113 is composed of a display or a head-mounted display (HMD) or the like.

[0028] (Method for Tracking Subject Position) Next, a method for tracking the three-dimensional position of the subject in the present embodiment will be described.

[0029] First, the three-dimensional shape estimation unit 103 generates a 3D model of the subject, outputs the generated 3D model to the storage unit 108, and outputs the generated 3D model to the shape extraction unit 104.

[0030] FIG. 2 is a diagram for explaining the subject position tracking process in Embodiment 1. The shape extraction unit 104 cuts out the lower part of the 3D model of the subject as shown in FIG. 2(b) from the 3D model of the subject as shown in FIG. 2(a). In the present embodiment, it is assumed that the area from the bottom surface of the circumscribed rectangular parallelepiped of the 3D model of the subject to a predetermined height (for example, a height corresponding to 50 cm) is cut out. For example, as shown in FIG. 2(c), when one of the subjects is standing and one of the subjects jumps and is away from the floor surface of the shooting area, the 3D model of the subject is cut out as shown in the range of FIG. 2(d). That is, for the 3D model of either subject, the 3D model from the portion corresponding to the feet to a predetermined height is cut out. Note that the area cut out from the 3D model is not limited to the lower part of the 3D model. For example, the middle part of the 3D model of the subject (for example, the area from 50 cm to 100 cm from the bottom surface of the circumscribed rectangular parallelepiped of the 3D model) may be cut out.

[0031] Next, as shown in FIG. 2(e), the shape extraction unit 104 projects the cut-out 3D model onto the floor surface to generate a two-dimensional image. In the present embodiment, the projected image is a binary image in which the portion of the cut-out 3D model is white and the other portions are black. The shape extraction unit 104 divides this two-dimensional image into independent regions and obtains the circumscribed rectangles 201 to 204 shown in FIG. 2(e). The shape extraction unit 104 outputs the vertex information of this circumscribed rectangle as the extracted shape of the extracted 3D model. Specifically, the shape extraction unit 104 outputs the vertex information of the circumscribed rectangle after converting it into the same coordinate system and unit as the three-dimensional space of the shooting area. In addition, for the determination of independent shapes by the shape extraction unit 104, a method such as connected component analysis is used for the projected two-dimensional image. By using such a method, the shape extraction unit 104 can divide the 3D model into individual regions.

[0032] FIG. 3 is a diagram for explaining the process of assigning identifiers to subject positions in Embodiment 1. The identification setting unit 105 assigns identifiers to the extracted shapes output by the shape extraction unit 104. Specifically, the identification setting unit 105 calculates the distances between the respective extracted shapes and assigns identifiers according to the distances between the extracted shapes. For example, as shown in FIG. 3(a), the identification setting unit 105 assigns the same identifier to the extracted shapes whose distances between the extracted shapes are less than a predetermined distance (solid line arrow), and assigns different identifiers to those whose distances between the extracted shapes are greater than or equal to the predetermined distance (dashed line arrow). The threshold value of the predetermined distance used as the criterion for determination is desirably a distance corresponding to the foot opening width in the state where the subject stands. In the present embodiment, it is described that the threshold value of the predetermined distance is set to 50 cm.

[0033] The identification setting unit 105 displays the assigned identifier on a display unit provided in the identification setting unit 105 by means of a graphical user interface (GUI) as shown in Fig. 3(b). The user operates the image processing system while viewing this GUI. Specifically, the identification setting unit 105 displays the current identifier assignment (the identifier assignment in the initial state) on the GUI, distinguishing it by at least one of characters and color coding. In Fig. 3(b), the identification setting unit 105 displays each identifier with both character and color coding. The user checks the GUI to confirm whether the desired identifier assignment is in place as the initial state. If the desired identifier assignment is not in place, the user gives instructions such as changing the standing position relative to the subject or asking the subject to close their legs, and repeats this until the desired assignment is achieved. Alternatively, the user operates the image processing system via the GUI and gives a change instruction so that the desired identifier assignment is made. If it is the desired identifier assignment, the user presses a determination button (initial identifier determination button) on the graphical user interface as shown in Fig. 3(b), for example. In response to this operation, the identification setting unit 105 determines the identifier in the initial state. Furthermore, for the identifier in the initial state, the type of each subject may be set. It is assumed that this type is set for each imaging target (for example, baseball, live, etc.). In this embodiment, assuming basketball, either the type of player, referee, or ball is set for each subject. Although it is assumed that this type is set by user input, it is not limited to this. For example, in the initial state, a condition may be preset such that a subject corresponding to an extraction shape in which no other extraction shape exists within a predetermined distance is set as a ball, and the type may be set simultaneously with the determination of the identifier. Then, the identification setting unit 105 outputs information indicating the determined identifier and type for each extraction shape to the tracking unit 106.

[0034] In response to receiving an identifier from the identification setting unit 105, the tracking unit 106 assigns the identifier to each extracted shape as an initial state. Thereafter, the tracking unit 106 tracks the extracted shapes to which this identifier is assigned. Note that the identifier assigned to the extracted shape during tracking is not the identifier determined by the identification setting unit 105, but an identifier determined based on the tracking result of the position of each extracted shape by the tracking unit 106. In tracking the extracted shape, the tracking unit 106 performs tracking of the extracted shape based on the position of each extracted shape at the time immediately before the shooting time of the extracted shape, the identifier of each extracted shape, and the subject position information input from the subject position calculation unit described later. The specific processing of the tracking by the tracking unit 106 will be described later. Based on the tracking result, the tracking unit 106 assigns an identifier to each extracted shape at that time and outputs each extracted shape to the subject position calculation unit 107.

[0035] FIG. 4 is a diagram showing representative positions of the subject in Embodiment 1. The subject position calculation unit 107 obtains a representative position for each extracted shape to which the identifier obtained from the tracking unit 106 is assigned. For example, as shown in FIG. 4, the subject position calculation unit 107 obtains the positions indicating each group of extracted shapes for each group of extracted shapes to which the same identifier is assigned, such as the representative position 401 and the representative position 402. In the present embodiment, the representative position is the center position of the group of extracted shapes.

[0036] However, since this representative position is affected by the shape estimation error and the fluctuation of the boundary portion when the shape is cut out by the shape extraction unit 104, the position may fluctuate at each time even when the subject is stationary. Therefore, in the present embodiment, the subject position calculation unit 107 performs processing such as a low-pass filter or a moving average in the time direction on the center position information at each time to generate position information with high-frequency components suppressed. Then, the subject position calculation unit 107 outputs the position information of the representative position together with the identifier as the position of the subject to the tracking unit 106. Further, the subject position calculation unit 107 records (stores) in the storage unit 108 the information obtained by adding the information on the shooting time of the subject to the position information of the representative position as the subject position information (subject position information).

[0037] (Tracking Process by Tracking Unit 106) FIG. 5 is a flowchart showing the process by which the tracking unit 106 in Embodiment 1 tracks the position of a subject.

[0038] In step S501, the tracking unit 106 receives an input from the identification setting unit 105 and acquires the identifier and type of each extracted shape.

[0039] In step S502, the tracking unit 106 next acquires the extracted shape input from the shape extraction unit 104.

[0040] In step S503, the tracking unit 106 assigns the identifiers and types acquired from the identification setting unit 105 to each of the acquired extracted shapes, and outputs each of the extracted shapes with the identifiers and types assigned thereto to the subject position calculation unit 107.

[0041] In step S504, the subject position calculation unit 107 obtains the subject position from the group of extracted shapes with the same identifier, and outputs the subject position to the tracking unit 106.

[0042] The processing up to steps S501 to S504 described above corresponds to the initialization processing.

[0043] The processing in steps S505 to S509 subsequent thereto is processing for each time, and is repeatedly executed while the imaging unit 101 is imaging the subject. When the imaging process of the subject by the imaging unit 101 is completed, the processing of this flowchart is terminated in response to the completion of the processing in step S509.

[0044] In step S505, the tracking unit 106 acquires the extracted shape input from the shape extraction unit 104 and the subject position at the previous time (the previous time) calculated by the subject position calculation unit 107. The previous time is, for example, the imaging time of the extracted shape generated one frame before the currently processed extracted shape. Here, for the sake of comparison, the current time is also referred to as the current time. Here, the current time refers to the imaging time of the image used to generate the currently processed extracted shape.

[0045] In step S506, when the subject position at the previous time overlaps with the representative position of each extracted shape at the current time, the tracking unit 106 assigns to the extracted shape the identifier assigned to the subject position that overlaps with the representative position thereof. Here, in step S506, when the representative position of one extracted shape overlaps with a plurality of subject positions, the tracking unit 106 assigns to the extracted shape an identifier indicating "unable to determine" at the current time. This is because, for example, in a state where two subjects are close to each other, there may be a possibility that a plurality of extracted shapes with different identifiers overlap at the current time, and thus, in the process of this step, an identifier indicating "unable to determine" is assigned. The extracted shape to which an identifier including the identifier indicating "unable to determine" is assigned is subjected to the process of S509 described later.

[0046] In step S507, when the representative position of an extracted shape that has not yet been assigned an identifier overlaps with the extracted shape at the previous time, the tracking unit 106 assigns the identifier assigned to the extracted shape at the previous time to the current-time extracted shape.

[0047] In step S508, when there is another extracted shape within a predetermined range from an extracted shape that has not yet been assigned an identifier and that has already been assigned an identifier at the current time, the tracking unit 106 assigns the identifier assigned to the other extracted shape. The predetermined range is preferably a range corresponding to the foot separation width in a state where the subject stands. For example, the predetermined range is a range with a radius of 50 cm from the center of the extracted shape. Here, when there are a plurality of other extracted shapes within the predetermined range from a certain extracted shape and to which identifiers are assigned, the tracking unit 106 assigns the identifier of the extracted shape closest to the other extracted shapes to the extracted shape. For an extracted shape that has not been assigned an identifier at the stage when the processing up to step S508 is completed, the tracking unit 106 determines that the extracted shape is out of the tracking target. In this case, the tracking unit 106 does not output the extracted shape determined to be out of the tracking target to the subject position calculation unit 107.

[0048] In step S509, the tracking unit 106 outputs the extracted shape to which the identifier is assigned in the processes from step S506 to step S508, and the assigned identifier to the subject position calculation unit 107.

[0049] In step S510, it is determined by a control unit (not shown) whether the imaging process of the subject by the imaging unit 101 has been completed. If it is determined that the imaging process of the subject by the imaging unit 101 has not been completed, the process of step S508 is executed. If it is determined that the imaging process of the subject by the imaging unit 101 has been completed, the process of this flowchart is terminated.

[0050] Regarding the processes from step S506 to step S508, in each process, the process is performed for each extracted shape. By repeating the processes from step S506 to step S509, the identifier set by the identification setting unit 105 is associated with the extracted shape at each time. Using this identifier, the subject position calculation unit 107 can obtain the subject position by distinguishing each subject.

[0051] Also, in the tracking unit 106, when the identifier "undetermined" is assigned to the extracted shape, there is a possibility that some of the identifiers defined in the initial setting are not assigned at a certain time. In such a case, in the subject position calculation unit 107, the subject position information having the same identifier as the identifier not assigned to the extracted shape is not updated. Thereby, even when the extracted shapes overlap due to multiple subjects approaching each other, the multiple subject position information does not become the same position. In this case, the respective positions of the multiple subjects are maintained until the previous time. After that, when the multiple overlapping extracted shapes are separated again due to the subjects moving away from each other, an identifier is assigned to each extracted shape based on the most recent subject position. That is, in response to the elimination of the overlap of the multiple extracted shapes, the update of each subject position information is resumed.

[0052] Through the above processing, even when multiple subjects are in the shooting area, the image processing system can track individual subjects and obtain the position information of individual subjects. Furthermore, through the above processing, the image processing system can track individual subjects even when the generated 3D model model undergoes superposition or separation due to the subject approaching or moving away.

[0053] (Processing for determining the object of interest) FIG. 6 is a flowchart showing the process by which the object-of-interest determination unit 110 in Embodiment 1 determines the object of interest. In this embodiment, an example of determining the object of interest during the generation of a virtual viewpoint image of a basketball will be described. At this time, the object-of-interest candidates are the basketball goals of both teams, and it is assumed that the position information of the object-of-interest candidates has been acquired from the storage unit 108 in advance. Note that this process is performed for each frame.

[0054] FIG. 7 is a diagram showing the velocity vectors and average velocity vectors of each subject in this embodiment. As shown in FIG. 7(a), coordinate axes x and y are defined, with the center of the court as the origin (0, 0), and it is assumed that there is a basketball goal on the x-axis.

[0055] In step S601, the object-of-interest determination unit 110 acquires the subject position information based on the time indicated by the time indication unit 111 from the subject position information detected by the subject position detection unit 114 and recorded in the storage unit 108. At this time, not only the subject position information at the specified time but also the subject position information of several frames to several tens of frames before and after that time is acquired. From S602 to S605 below, processing is performed for each subject.

[0056] In step S602, it is determined whether a certain subject is the subject to be calculated. Specifically, based on whether the type corresponding to the identifier for identifying the subject included in the position information matches the calculation target information, it is determined whether to perform subsequent processing. The calculation target information is information for specifying the calculation target of the target of interest determination process, and is information for specifying the type corresponding to the identifier of the subject. For example, calculation target information such as setting a subject whose type corresponding to the identifier is an athlete as the calculation target is generated and recorded in advance in the storage unit 108 or the target of interest determination unit 110. In that case, by excluding subjects whose types are referees and balls from the calculation target, they can be excluded from the target of interest determination process.

[0057] In step S603, filter processing is performed on the position information of the subject determined to be the calculation target. Specifically, position information obtained by averaging the position information for a plurality of frames acquired in S601 is calculated. Thereby, blurring due to the detection error of the subject position detection unit 114 can be reduced.

[0058] In step S604, subject position information at a time before the time of the subject to be calculated, for example, one second before and several to several tens of frames (predetermined time) before and after that time, is acquired and its average value is calculated.

[0059] In step S605, after subtracting the averaged position information at the past time from the averaged position information at the current time, the difference in position information is divided by the time difference between the current time and the past time, which is one second here, to obtain the position deviation (position change) per unit time at the current time. The magnitude of this position deviation per unit time is the moving speed, and the direction of the vector on the two-dimensional plane is the moving direction of the subject. The processing of S602 to S605 is performed for each subject. When the processing is completed for all subjects, the process proceeds to S606.

[0060] In step S606, the average velocity vector of the velocity vectors of all the subjects to be calculated is calculated. FIG. 7(b) shows the average velocity vector of all the subjects to be calculated. In the present embodiment, the component in the goal direction included in this average velocity vector is obtained. Specifically, since the direction of the goal is the positive and negative directions of the x-axis, the direction and magnitude of the vector obtained by orthogonally projecting the average velocity vector onto the vector (1,0) are obtained.

[0061] In step S607, it is determined whether or not the average velocity vector calculated in S606 satisfies the target-of-interest determination condition. It is assumed that the target-of-interest determination condition differs for each imaging target. In the present embodiment, it is determined whether or not the vector obtained by projecting the average velocity vector in the goal direction satisfies the target-of-interest determination condition. In the case of the initial state where the target of interest has not been determined, regardless of whether or not the target-of-interest determination condition is satisfied, the process proceeds to S608. In the present embodiment, the target-of-interest determination condition is whether or not the direction of the vector obtained by projecting the average velocity vector in the goal direction is facing the current target of interest and whether or not the magnitude of the vector is greater than a predetermined magnitude. When the current target of interest has been determined, a determination of whether or not to change the target of interest is made based on the direction and magnitude of the orthogonally projected vector. If the orthogonally projected vector is facing the positive direction, is greater than or equal to a predetermined magnitude, and the current target of interest is not the goal 71 on the positive side of the x-axis, the process proceeds to S608. Also, if the vector is facing the negative direction, is greater than or equal to a predetermined magnitude, and the currently focused goal is not the goal 72 on the negative side of the x-axis, the process proceeds to S608. If neither of the two conditions is met, the process ends without changing the current target of interest and determines the target of interest in the current frame.

[0062] In step S608, one object of interest is determined from among a plurality of candidates for the object of interest based on the average velocity vector. In the initial state where the object of interest has not been determined, a straight line is extended in the direction of the average velocity vector, and the object of interest closest to the straight line is specified. Further, when there are a plurality of objects of interest closest to the straight line, the object of interest closest to the position of the subject of interest is specified. Alternatively, an initial value of the object of interest may be set in advance by a user operation. In the present embodiment, determination of a change in the object of interest is made based on the direction and magnitude of the orthogonally projected vector. When the orthogonally projected vector points in the positive direction, has a magnitude equal to or greater than a predetermined value, and the current object of interest is not the goal 71 on the positive x-axis side, the goal 71 is determined (specified) as the object of interest. Further, when the vector points in the negative direction, has a magnitude equal to or greater than a predetermined value, and the goal currently being focused on is not the goal 72 on the negative x-axis side, the goal 72 is determined as the object of interest.

[0063] By performing the processes of S601 to S608 for each frame in this way, the movements of all subjects are detected and the object of interest is determined. The object-of-interest determination unit 110 outputs the information on the determined object of interest to the viewpoint generation unit 109.

[0064] As described above, the viewpoint generation unit 109 arranges a virtual viewpoint in a direction rotated by a predetermined angle from the straight line connecting the subject and the object of interest. When the object of interest switches from the goal 71 to the goal 72, the viewpoint does not immediately switch. Instead, after obtaining the current angle and the angle after the switch, the virtual viewpoint is complemented by rotating by this angle difference over a predetermined time. This enables the viewer to correctly recognize that the object of interest currently being focused on has switched when the object of interest switches.

[0065] As described above, according to the present disclosure, in sports such as basketball, it is possible to detect the switch between offense and defense from the movements of all subjects, and accordingly, the viewpoint generation unit 109 can automatically generate a virtual viewpoint that captures the goal on the offensive side. As a result, it is possible to produce a virtual viewpoint image suitable for the scene while following the subject designated by the user and including it in the virtual viewpoint image.

[0066] In this embodiment, the goal direction component of the average velocity vector is calculated in S606. However, which direction component to calculate shall vary depending on the imaging target and the candidate of the object of interest. For example, when imaging a 100-meter dash on land, the straight line from the start to the goal is set as the x-axis direction. Thus, which direction component of the average velocity vector to calculate varies depending on the imaging target and the object of interest. Therefore, for example, when a combination of a plurality of candidates of the object of interest is determined, a predetermined component direction may be provided and stored in advance for that combination.

[0067] In this embodiment, the component in the goal direction of the average velocity vector is calculated in S606, but it is not limited thereto. The object of interest may be specified from the direction of the average velocity vector. Specifically, a straight line is provided in the direction of the average velocity vector, and the object of interest closest to the straight line is specified. Note that there may be a plurality of candidates of the object of interest closest to the straight line. In that case, the object of interest closest to the position of the subject to be further noted is specified. Thereby, one object of interest can be specified from a plurality of candidates of the object of interest.

[0068] (Other forms of Embodiment 1) In this embodiment, the description of the object position detection means based on the shape estimation result has been given for the object position detection unit 114. However, the present disclosure is not limited to the method of detecting the position of this object. For example, a configuration in which a position sensor such as GPS is attached to the athlete and the sensor value is acquired may be used. Alternatively, a configuration in which the object position is detected from an image obtained by a plurality of imaging means using image recognition technology may also be used.

[0069] Also, in this embodiment, the configuration is such that the object of interest determination unit 110 calculates the speed and the moving direction every frame when generating the virtual viewpoint image, but it is not necessarily limited thereto. For example, a configuration may be adopted in which the object position detection unit 114 detects the position and calculates the velocity vector of each object and records it in the storage unit 108. In this case, the object of interest determination unit 110 may acquire the velocity vector of each object from the storage unit 108 and determine the object of interest.

[0070] In this embodiment, although averaging processing is performed as the filter processing of the position information, the present invention is not limited to this. For example, a low-pass filter such as an IIR filter or an FIR filter may be used. However, in the case of a configuration that calculates the speed etc. each time, when using a low-pass filter, if the time for reproducing the virtual viewpoint image is discontinuously changed, the value becomes incorrect. Therefore, in such a configuration, it is desirable to average after acquiring the information of the neighboring times as described above.

[0071] In this embodiment, basketball is taken as an example, but it may be applied to sports such as soccer and rugby. For example, in the case of soccer, the development of the game is slower than that of basketball, and there may be a case where the whole players are relatively stopped for a long time. In such a case, it may be determined on which side of the field there are more subjects (players), and the object of interest may be determined by comprehensively judging the determination result and the velocity vector of the subject.

[0072] <Embodiment 2> In Embodiment 1, the object of interest was determined from the average velocity vectors of all the subjects who are players. However, the object of interest may be determined based on the movement of one player. For example, when generating a virtual viewpoint image that follows a baseball runner, the object of interest may be determined by paying attention to the movement of the runner. In this embodiment, an example in which the imaging target is baseball will be described. In this embodiment, the block diagram is the same as that of FIG. 1, and the components that are not particularly described are the same as those in Embodiment 1, so the description will be omitted.

[0073] FIG. 9 is a flowchart showing the process in which the object-of-interest determination unit 110 in Embodiment 2 determines the object of interest. Note that this process is performed for each frame.

[0074] FIG. 10 is a diagram showing an example of the object-of-interest candidates and the determination region (predetermined region) in Embodiment 2. In this embodiment, the home base 1005 of baseball and each base of the first base 1006, the second base 1007, and the third base 1008 are set as the object-of-interest candidates.

[0075] In step S901, the object of interest determination unit 110 sets a first determination region 1001, a second determination region 1002, a third determination region 1003, and a fourth determination region 1004 by a user operation on a user interface (not shown). At this time, candidates for the object of interest associated with each determination region are set. Specifically, the home base 1005 and the first base 1006 are associated with the first determination region 1001. Similarly, the first base 1006 and the second base 1007 are associated with the second determination region 1002, the second base 1007 and the third base 1008 are associated with the third determination region 1003, and the third base 1008 and the home base 1005 are associated with the fourth determination region 1004. Also, the object of interest determination unit 110 acquires the position information of the object of interest candidates associated with the determination regions, and obtains a direction vector connecting two object of interest candidates. For example, in the case of the second determination region 1002, a unit direction vector connecting the first base 1006 and the second base 1007 is obtained. At this time, the unit direction vector is a vector indicating the direction of the second base. Note that the position information of these determination regions and the object of interest candidates may be managed as condition information and stored in the object of interest unit 110, or may be stored in the storage unit 108. Note that S901 is performed only in the first frame for determining the object of interest, and is not performed for each frame. Note that the present invention is not limited to this, and S901 may be performed at an arbitrary timing of the user.

[0076] In step S902, the object of interest determination unit 110 acquires the object position information based on the time indicated by the time instruction unit 111 from the object position information detected by the object position detection unit 114 and recorded in the storage unit 108.

[0077] In step S903, similar to Embodiment 1, the velocity vector of the object is obtained. At this time, similar to Embodiment 1, filter processing is performed on the acquired object position information.

[0078] In step S904, the object of interest determination unit 110 determines in which of the four preset determination regions the subject is present. In this embodiment, the case where the subject is in the second determination region 1002 will be described. Next, a plurality of object of interest candidates associated with the second determination region 1002 are specified. In this embodiment, the first base 1006 and the second base 1007 are the object of interest candidates.

[0079] In step S905, it is determined whether the velocity vector calculated in S903 satisfies the object of interest determination condition. In the initial state where the object of interest has not been determined as in the first embodiment, regardless of whether the object of interest determination condition is satisfied, the process proceeds to S906. The object of interest determination unit 110 obtains the direction and magnitude of the vector obtained by orthogonally projecting the velocity vector of the subject onto the unit direction vector associated with the determination region. In this embodiment, the object of interest determination condition is whether the magnitude of the vector obtained by orthogonal projection is greater than a predetermined magnitude. If the magnitude of the vector obtained by orthogonal projection is greater than the predetermined magnitude, the process proceeds to S906. If the magnitude of the vector obtained by orthogonal projection is less than or equal to the predetermined magnitude, the process ends.

[0080] In step S906, one object of interest is determined based on the average velocity vector among the plurality of object of interest candidates specified in S904. The processing in the initial state where the object of interest has not been determined is the same as in the first embodiment. When the magnitude of the vector obtained by orthogonal projection is greater than the predetermined value and the direction of the vector obtained by orthogonal projection is in the same second base direction as the aforementioned unit direction vector, the second base 1007 is determined as the object of interest. Also, when the direction of the vector obtained by orthogonal projection is opposite to the unit direction vector, the first base 1006 is determined as the object of interest. By performing the processing of S902 to S906 for each frame, a virtual viewpoint that follows the movement of the player and focuses on an appropriate object of interest can be generated.

[0081] FIG. 11 is a diagram showing the movement of a subject, an object of interest, and a virtual viewpoint in Embodiment 2. Specifically, the movement of the player (runner) and the switching of the object of interest will be described with reference to FIG. 11. Here, it will be described with the runner 1101 being in the vicinity of the first base 1006 as shown in FIG. 11(a). When the runner 1101 is in the vicinity of the first base 1006, since the runner has passed through the first determination region 1001 before reaching there, the currently focused object of interest is the first base 1006. At this time, the runner 1101 makes a lead and enters the second determination region 1002. However, due to the low-speed movement, the object of interest remains the first base 1006. The virtual viewpoint generated by the viewpoint generation unit 109 at this time is a virtual viewpoint 1102 that captures the runner 1101 and the first base 1006 from diagonally behind the runner 1101 as shown in FIG. 11(a). Next, when the runner 1101 starts running to advance to the next base and the speed becomes equal to or higher than a predetermined speed, the object of interest to be focused on switches to the second base 1007. As a result, the virtual viewpoint 1102 following the runner 1101 rotates around the runner 1101 as indicated by the dashed arrow 1103 in FIG. 11(b), and automatically transitions to a virtual viewpoint 1102 that captures the second base 1007 over the shoulder of the runner 1101 while following the runner 1101. Also, for example, if the runner 1101 tries to return to the first base 1006 during the advancement, the object of interest to be focused on switches to the first base 1006, and the virtual viewpoint 1102 including the runner 1101 and the first base 1006 is transitioned to. Next, when the advancement to the next base is successful and the runner further tries to advance to the third base 1008, the runner 1101 enters the third determination region 1003. As a result, based on the unit direction vector defined by the second base 1007 and the third base 1008 associated with the third determination region 1003, the movement direction and speed are determined. As a result, when the runner enters the third determination region 1003, the object of interest to be focused on becomes the third base 1008, and as shown in FIG. 11(c), the virtual viewpoint 1102 moves to a position that includes the runner 1101 and the third base 1008.

[0082] By applying the present disclosure in this manner, among a plurality of candidates of objects of interest, an appropriate object of interest can be identified based on the position and speed information of the player, and based on the identified object of interest, the viewpoint generation unit 109 can generate an appropriate virtual viewpoint. As a result, even without an operator who operates the virtual viewpoint, it is possible to create appropriate camera work according to the situation.

[0083] (Other forms of Embodiment 2) Similar to Embodiment 1, the subject position detection unit 114 may use another position detection method such as GPS or image recognition.

[0084] In the present embodiment, the virtual viewpoint generated by the viewpoint generation unit 109 has been described as a virtual viewpoint that follows the subject from behind, but it is not limited to this. Any virtual viewpoint may be used as long as it generates a virtual viewpoint using information on the subject being focused on and the object of interest to be focused on. For example, the viewpoint generation unit 109 may be such that a virtual viewpoint is arranged near the base that is the object of interest, and camera work is created so as to lie in wait for the subject to be tracked and fit within the angle of view. In this case, it is desirable that as the object of interest changes, the arrangement of the virtual viewpoint transitions near the base that is the object of interest while capturing the subject to be tracked from the front.

[0085] In the present embodiment, it is assumed that the virtual viewpoint is automatically generated by the viewpoint generation unit 109 and the operator is not required, but it is not necessarily limited to this. For example, the distance between the virtual viewpoint generated by the viewpoint generation unit 109 and the player, the height at which it is arranged, the angle of view, etc. may be configured so that the operator can operate them using a user interface (not shown). By doing so, although it is the responsibility of the viewpoint generation unit 109 to fit the subject and the object of interest within the angle of view, it is more desirable to configure it so that the operator can control the composition such as the angle of view.

[0086] In the present embodiment, it is applied to baseball, but it may also be applied to sports competitions such as softball and other sports where the virtual viewpoint to be included in the virtual viewpoint changes depending on the position of the players.

[0087] In this embodiment, the candidate of interest is used as the background model, and the position and orientation of the virtual viewpoint are determined by the position of the subject specified by the user and the center-of-gravity position of the candidate of interest. However, the present invention is not limited to this. For example, a three-dimensional position different from the center-of-gravity position may be associated with the background model of the candidate of interest. In that case, the position and orientation of the virtual viewpoint are determined based on the position of the subject specified by the user and the three-dimensional position corresponding to the candidate of interest. Such processing enables easy setting of a virtual viewpoint suitable for the scene when there is a goal net at a position different from the center-of-gravity position of the background model, such as the goal of basketball. Note that the three-dimensional position corresponding to the background model may be set at the same height as the center-of-gravity position of the subject. For example, when the candidate of interest is a baseball base, it will be present on the floor surface, and there is a risk of generating a virtual viewpoint image corresponding to a virtual viewpoint facing the floor surface. Therefore, by adjusting the height of the three-dimensional position corresponding to the background model, it becomes possible to easily set a virtual viewpoint suitable for the scene.

[0088] (Other configurations) In the above embodiment, each processing unit shown in FIG. 1 is described as being configured by hardware. However, the processing performed by each of these processing units shown in these figures may be configured by a computer program.

[0089] FIG. 12 is a block diagram showing a configuration example of the hardware of a computer applicable to the indirect position estimation apparatus according to each of the above embodiments.

[0090] The CPU 1201 controls the entire computer using computer programs and data stored in the RAM 1202 and the ROM 1203, and executes each of the above-described processes as those performed by the indirect position estimation apparatus according to each of the above embodiments. That is, the CPU 1201 functions as each of the processing units shown in FIG. 1.

[0091] RAM 1202 has an area for temporarily storing computer programs and data loaded from the external storage device 1206, data acquired from the outside via the I / F (interface) 1207, and the like. Further, RAM 1202 has a work area used when the CPU 1201 executes various processes. That is, RAM 1202 can be allocated as, for example, a frame memory, or can appropriately provide various other areas.

[0092] The ROM 1203 stores the setting data of this computer, a boot program, and the like. The operation unit 1204 is composed of a keyboard, a mouse, and the like, and various instructions can be input to the CPU 1201 by the user of this computer operating it. The output unit 1205 displays the processing result by the CPU 1201. Also, the output unit 1205 is composed of, for example, a liquid crystal display. For example, the viewpoint generation unit 109 is composed of the operation unit 1204, and the display unit 113 is composed of the output unit 1205.

[0093] The external storage device 1206 is a large-capacity information storage device typified by a hard disk drive device. The external storage device 1206 stores an OS (operating system) and a computer program for causing the CPU 1201 to realize the functions of each unit shown in FIG. 1. Further, each image data to be processed may be stored in the external storage device 1206.

[0094] The computer programs and data stored in the external storage device 1206 are appropriately loaded into the RAM 1202 according to the control by the CPU 1201 and become the processing targets by the CPU 1201. The I / F 1207 can connect a network such as a LAN or the Internet, other devices such as a projection device and a display device, and this computer can acquire and send various information via this I / F 1207. In Embodiment 1, the imaging unit 101 is connected thereto to input the captured image and control each of them. 1208 is a bus connecting the above-described units.

[0095] The operation with the above configuration is controlled by the CPU 1201 centering around the operation described in the foregoing embodiment.

[0096] In addition, part or all of the control in the present embodiment may be supplied to an image processing system or the like via a network or various storage media by a computer program that realizes the functions of the foregoing embodiment. Then, a computer (or a CPU, MPU, etc.) in the image processing system or the like may read and execute the program. In that case, the program and the storage medium storing the program will constitute the present disclosure.

[0097] Note that the disclosure of the present embodiment includes the following configurations, methods, and programs.

[0098] (Configuration 1) Acquisition means for acquiring position information indicating the position of a subject included in an imaging area imaged by a plurality of imaging devices; Specification means for specifying one of a plurality of positions of interest based on the moving direction of the subject; Determination means for determining the position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and the line-of-sight direction from the virtual viewpoint based on the position of interest specified by the specification means and the position of the subject An information processing apparatus characterized by comprising:

[0099] (Configuration 2) The position information indicates the positions of a plurality of subjects, The information processing apparatus according to Configuration 1, wherein the specification means specifies one of the plurality of positions of interest based on the moving directions of the plurality of subjects.

[0100] (Configuration 3) The information processing apparatus according to Configuration 2, wherein the specification means specifies one of the plurality of positions of interest based on the moving directions and moving speeds of the plurality of subjects.

[0101] (Configuration 4) The plurality of subjects are classified into a plurality of types including a first type and a second type, The information processing apparatus according to Configuration 2, wherein the specifying means specifies one of the plurality of regions of interest based on the moving directions of the plurality of subjects corresponding to the first type.

[0102] (Configuration 5) The plurality of imaging devices image a competition, The first type is a player, The information processing apparatus according to Configuration 4, wherein the second type is a referee.

[0103] (Configuration 6) The acquisition means acquires condition information indicating the plurality of regions of interest corresponding to a predetermined region in a three-dimensional space, The information processing apparatus according to Configuration 1, wherein the specifying means specifies one of the plurality of regions of interest corresponding to the predetermined region when the subject is located within the predetermined region indicated by the condition information.

[0104] (Configuration 7) The information processing apparatus according to Configuration 1, wherein the moving direction of the subject is specified based on a change in position of the subject at a predetermined time.

[0105] (Configuration 8) The information processing apparatus according to Configuration 1, wherein the region of interest is set based on a user operation.

[0106] (Configuration 9) The information processing apparatus according to Configuration 1, wherein the region of interest is associated with a background model.

[0107] (Configuration 10) The information processing apparatus according to Configuration 9, wherein the background model is a stationary object.

[0108] (Configuration 11) The information processing apparatus according to Configuration 10, wherein the background model identifies a basketball goal.

[0109] (Configuration 12) The determination means determines the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint based on a half-line extending from the position of the subject to the position of interest, according to the information processing apparatus described in Configuration 1.

[0110] (Configuration 13) The determination means determines the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint such that the angle formed by the half-line extending from the position of the subject to the position of interest and the line-of-sight direction from the virtual viewpoint is a predetermined angle, according to the information processing apparatus described in Configuration 12.

[0111] (Configuration 14) The determination means determines the position of the virtual viewpoint and the line-of-sight direction from the virtual viewpoint such that the virtual viewpoint image includes the subject and the position of interest, according to the information processing apparatus described in Configuration 1.

[0112] (Method) An acquisition step of acquiring position information indicating the position of a subject included in an imaging region imaged by a plurality of imaging devices; A specification step of specifying one position of interest among a plurality of positions of interest based on the moving direction of the subject; A determination step of determining the position of a virtual viewpoint corresponding to a virtual viewpoint image including the subject and the line-of-sight direction from the virtual viewpoint based on the position of interest specified in the specification step and the position of the subject; An information processing method characterized by including these steps.

[0113] (Program) A program for causing a computer to function as the information processing apparatus according to any one of Configurations 1 to 14.

Explanation of Signs

[0114] 104 Shape extraction unit 105 Identification setting unit 106 Tracking unit 107 Subject position calculation unit 109 Viewpoint generation unit 110 Object of interest determination unit 111 Time indication unit

Claims

1. An acquisition means for acquiring position information indicating the position of a subject included in an imaging area captured by multiple imaging devices, A determination means for determining the position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and the direction of line of sight from the virtual viewpoint, based on the position of the subject and a position determined from a plurality of pre-set positions based on the position of the subject. An information processing device characterized by having the following features.

2. The information processing apparatus according to claim 1, wherein the determination means determines one position from among the plurality of positions based on the position of the subject.

3. The aforementioned location information indicates the locations of multiple subjects. The information processing apparatus according to claim 2, characterized in that the determination means determines one of the plurality of positions based on the movement direction of the plurality of subjects.

4. The information processing apparatus according to claim 3, wherein the determination means determines one of the plurality of positions based on the direction of movement and speed of movement of the plurality of subjects.

5. The aforementioned multiple subjects are classified into multiple types, including the first type and the second type. The information processing apparatus according to claim 3, characterized in that the determination means determines one of the plurality of positions based on the movement direction of the plurality of subjects corresponding to the first type.

6. The aforementioned plurality of imaging devices capture images of the competition, The first type is a player, The information processing device according to claim 5, characterized in that the second type is a referee.

7. The acquisition means acquires conditional information indicating the plurality of positions corresponding to a predetermined region in three-dimensional space, The information processing apparatus according to claim 2, characterized in that the determination means determines one position from a plurality of positions corresponding to the predetermined region when the subject is located within the predetermined region indicated by the condition information.

8. The information processing apparatus according to claim 3, characterized in that the direction of movement of the subject is determined based on the change in the position of the subject over a predetermined period of time.

9. The information processing apparatus according to claim 1, characterized in that the plurality of positions are set based on user operation.

10. The information processing apparatus according to claim 1, characterized in that the plurality of positions correspond to a background model.

11. The information processing apparatus according to claim 10, characterized in that the background model is a stationary object.

12. The information processing device according to claim 11, characterized in that the background model is the first base, second base, and third base of a baseball.

13. The information processing apparatus according to claim 1, wherein the determination means determines the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint based on the position of the subject and a straight line passing through a position determined based on the position of the subject.

14. The information processing apparatus according to claim 13, wherein the determination means determines the position of the virtual viewpoint and the direction of line of sight from the virtual viewpoint such that the angle formed by a straight line passing through the position of the subject and a position determined based on the position of the subject and the direction of line of sight from the virtual viewpoint becomes a predetermined angle.

15. The information processing apparatus according to claim 1, wherein the determination means determines the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint so that the virtual viewpoint image includes the subject and a position determined based on the position of the subject.

16. An acquisition step to acquire position information indicating the position of a subject included in an imaging area captured by multiple imaging devices, A determination step of determining the position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and the line of sight direction from the virtual viewpoint, based on the position of the subject and a position determined based on the position of the subject from among a plurality of predetermined positions. An information processing method characterized by having the following features.

17. A program for causing a computer to function as an information processing device according to any one of claims 1 to 15.

Images