Image processing device, image processing method, and program

Abstract

To set a virtual viewpoint suitable for a scene.SOLUTION: An information processing apparatus has: acquisition means that acquires position information indicating the position of a subject included in an imaging area, images of which are picked up by a plurality of imaging apparatuses; specification means that specifies one position of interest of a plurality of positions of interest on the basis of the direction of movement of the subject; and determination means that, on the basis of the position of interest specified by the specification means and the position of the subject, determines the position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and the direction of line of sight from the virtual viewpoint.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present disclosure relates to an image processing device that generates a virtual viewpoint image. [Background technology]

[0002] A technology that generates an image (virtual viewpoint image) from any virtual camera (virtual viewpoint) based on a user's operation by using the multiple images obtained by installing multiple cameras in different positions and capturing images simultaneously is attracting attention. With such a technology, for example, it becomes possible to watch highlight scenes of soccer or basketball from various angles, which can give the user a high sense of realism compared to normal images.

[0003] In order to facilitate the operation of the virtual viewpoint, Patent Document 1 describes a method for determining the position and orientation of the virtual viewpoint based on position information of a first subject of interest and position information of a second subject that is to be included in the virtual viewpoint image. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2022-171436 A Summary of the Invention [Problem to be solved by the invention]

[0005] However, when generating a virtual viewpoint image of a subject performing complex movements, it was not possible to determine a virtual viewpoint appropriate for the scene. For example, in a basketball game, when trying to generate a virtual viewpoint image focusing on a certain player, the offense and defense frequently switch, so the appropriate virtual viewpoint position changes with the change in the scene, and it was not possible to set the position of the virtual viewpoint and the line of sight from the virtual viewpoint appropriate for the scene.

[0006] An object of the present disclosure is to enable a virtual viewpoint to be set that is appropriate for a scene. [Means for solving the problem]

[0007] The information processing device according to the present disclosure has the following configuration: an acquisition means for acquiring position information indicating a position of a subject included in an imaging area captured by a plurality of imaging devices, an identification means for identifying one of a plurality of positions of interest based on a moving direction of the subject, and a determination means for determining a position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and a line of sight direction from the virtual viewpoint based on the position of the subject and the position of the subject identified by the identification means. Effect of the Invention

[0008] According to the present disclosure, a virtual viewpoint suitable for a scene can be set. [Brief description of the drawings]

[0009] [Figure 1] 1 is a block diagram showing an example of a configuration of an image processing system according to a first embodiment. [Diagram 2] 5A to 5C are diagrams for explaining the process of tracking a subject position in the first embodiment. [Diagram 3] 6A to 6C are diagrams for explaining the process of assigning an identifier to a subject position in the first embodiment. [Figure 4] 4 is a diagram showing an example of a representative position of a subject in the first embodiment. FIG. [Diagram 5] 11 is a flowchart showing a process in which the tracking unit 106 tracks the position of a subject in the first embodiment. [Figure 6] 4 is a flowchart showing a process of determining an interest subject by the interest subject determining unit 110 according to the first embodiment. [Figure 7] 5A to 5C are diagrams showing examples of the velocity vectors and average velocity vectors of each subject in the first embodiment. [Figure 8] 4 shows an example of the position of a virtual viewpoint generated by a viewpoint generating unit 109 in the first embodiment. [Figure 9]10 is a flowchart showing a process of determining an interest subject by the interest subject determining unit 110 according to the second embodiment. [Figure 10] 13 is a diagram showing an example of a target of interest and a determination region in the second embodiment. FIG. [Figure 11] 13A to 13C are diagrams illustrating an example of the movements of a subject, a target of interest, and a virtual viewpoint in the second embodiment. [Figure 12] 2 is a block diagram showing an example of a hardware configuration of an information processing device 3. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] <Embodiment 1> (System configuration and operation of image processing device) The image processing system is a system that generates a virtual viewpoint image that represents a scene from a specified virtual viewpoint based on a plurality of images based on images captured by a plurality of imaging devices and a specified virtual viewpoint. The virtual viewpoint image in this embodiment is also called a free viewpoint video, but is not limited to an image corresponding to a viewpoint freely (arbitrarily) specified by a user, and includes, for example, an image corresponding to a viewpoint selected by a user from a plurality of candidates. In addition, although the present embodiment will mainly describe a case where the virtual viewpoint image is a moving image, the virtual viewpoint image may also be a still image.

[0011] The viewpoint information used to generate the virtual viewpoint image is information indicating the position and orientation (line of sight direction) of the virtual viewpoint. Specifically, the viewpoint information is a parameter set including a parameter indicating the three-dimensional position of the virtual viewpoint and a parameter indicating the orientation of the virtual viewpoint in the pan, tilt, and roll directions. Note that the content of the viewpoint information is not limited to the above. For example, the parameter set as the viewpoint information may include a parameter indicating the size of the visual field (angle of view) of the virtual viewpoint. Furthermore, the viewpoint information may have a plurality of parameter sets. For example, the viewpoint information may have a plurality of parameter sets corresponding to a plurality of frames constituting a video of the virtual viewpoint image, and may be information indicating the position and orientation of the virtual viewpoint at each of a plurality of consecutive time points.

[0012] The image processing system has a plurality of imaging devices that capture images of an imaging area from a plurality of directions. The imaging area is, for example, a stadium where sports such as soccer and karate are held, or a stage where a concert or a play is held. The plurality of imaging devices are installed at different positions so as to surround such an imaging area, and capture images synchronously. Note that the plurality of imaging devices do not need to be installed all around the imaging area, and may be installed only in a part of the periphery of the imaging area depending on restrictions on the installation location, etc. Also, the number of imaging devices is not limited to the example shown in the figure, and for example, when the imaging area is a soccer stadium, about 30 imaging devices may be installed around the stadium. Also, imaging devices with different functions, such as a telephoto camera and a wide-angle camera, may be installed.

[0013] In this embodiment, the multiple imaging devices are cameras each having an independent housing and capable of capturing images from a single viewpoint. However, this is not limited to this, and two or more imaging devices may be configured in the same housing. For example, a single camera having multiple lens groups and multiple sensors and capable of capturing images from multiple viewpoints may be installed as the multiple imaging devices.

[0014] FIG. 1 is a block diagram showing the configuration of an image processing system in the first embodiment. The image processing system includes, for example, an imaging unit 101, a synchronization unit 102, a three-dimensional shape estimation unit 103, a shape extraction unit 104, an identification setting unit 105, a tracking unit 106, a subject position calculation unit 107, a storage unit 108, a viewpoint generation unit 109, a subject of interest determination unit 110, and a time indication unit 111. The image processing system further includes an image generation unit 112 and a display unit 113. In this embodiment, the shape extraction unit 104, the identification setting unit 105, the tracking unit 106, and the subject position calculation unit 107 are collectively referred to as a subject position detection unit 114. The image processing system may be configured by one image processing device, or may be a system configured by multiple image processing devices. For example, the viewpoint generation unit 109, the subject of interest determination unit 110, the time indication unit 111, and the subject position detection unit 114 may be included in a single information processing device different from the image processing device, or the storage unit 108 may be configured by a dedicated server device. In the following description, the image processing system will be described as one image processing device.

[0015] The imaging units 101 capture images in synchronization with each other based on a synchronization signal from the synchronization unit 102. Then, the imaging units 101 output the captured images to the three-dimensional shape estimation unit 103. Note that the imaging units 101 are installed so as to surround an imaging area including a subject in order to capture the subject from multiple directions.

[0016] The synchronization unit 102 outputs a synchronization signal to the multiple image capture units 101 .

[0017] The three-dimensional shape estimation unit 103 generates a silhouette image of the subject using the input captured images from multiple viewpoints. Furthermore, it generates a 3D model of the subject using a volume intersection method or the like. The three-dimensional shape estimation unit 103 also outputs the generated 3D model of the subject and the captured images to the storage unit 108 and the shape extraction unit 104. Here, the subject is an object for which a 3D model is to be generated, and includes people and items handled by people.

[0018] The shape extraction unit 104 extracts a part of the 3D model of the subject acquired from the three-dimensional shape estimation unit 103. By extracting a part of the 3D model of the subject, it is possible to reduce the possibility that, when there are 3D models of multiple subjects, the subjects cannot be identified due to contact or adjacency between them. Furthermore, the shape extraction unit 104 generates a two-dimensional image by projecting the extracted 3D model of the subject from one direction. In this embodiment, the two-dimensional image is generated by projecting from a direction perpendicular to the floor surface. Then, extraction information of the 3D model of the subject is extracted from the generated two-dimensional image and transmitted to the tracking unit 106 and the identification setting unit 105.

[0019] The identification setting unit 105 assigns an identifier to the extracted information obtained from the shape extraction unit 104. The identifier assigned to each extracted shape is transmitted to the tracking unit .

[0020] In response to an identifier input from the identification setting unit 105, the tracking unit 106 assigns the identifier to each extracted shape as an initial state. After that, the tracking unit 106 tracks the extracted shape to which this identifier has been assigned. Each tracked extracted shape and its identifier are transmitted to the subject position calculation unit 107.

[0021] The subject position calculation unit 107 calculates the representative position of each subject (the position of the subject) using each extracted shape and identifier acquired from the tracking unit 106. The calculated representative position of the subject is transmitted to the storage unit 108.

[0022] The storage unit 108 saves and stores the following data group as data (virtual viewpoint material data) used to generate a virtual viewpoint image. In this embodiment, the data used to generate a virtual viewpoint image is specifically a 3D model of a subject and a captured image input from the three-dimensional shape estimation unit 103. In addition, the data used to generate a virtual viewpoint image includes camera parameters such as the position and orientation and optical characteristics of each imaging unit, and subject position information acquired by the subject position detection unit 114. It is assumed that a 3D model of the background and a background texture image are saved (stored) in the storage unit 108 in advance as data used to generate the background of the virtual viewpoint image. Furthermore, the 3D models stored in the storage unit 108 are stored in association with a type. Here, the type is set for each imaging target, and for example, in the case of basketball, it is a player, a ball, a referee, etc. It is to be noted that the type of each 3D model may be specified by a user, or may be automatically set according to a preset condition. In addition, the storage unit 108 stores a candidate target of interest, which will be described later. The candidate of interest may be a background model or a specific three-dimensional coordinate (position of interest) in a three-dimensional space. The candidate of interest is described below as being set in advance by a user, but may be set automatically by the device. The candidate of interest automatically set by the device may be, for example, the center position of an image capturing area by a plurality of image capturing locations or a subject at that center position, or the position of a boundary area between an image capturing area and other areas or a subject at that position. In this embodiment, the candidate of interest is described as a background model. The background model is, for example, a basketball goal, a soccer goal, or a finish line in track and field.

[0023] The viewpoint generating unit 109 generates a virtual viewpoint based on the subject position information detected by the subject position detecting unit 114 and the position information of the subject of interest output by the subject of interest determining unit 110. Specifically, as shown in FIG. 8, the viewpoint generating unit 109 sets the position of the virtual viewpoint at a position at a predetermined distance from the subject in a direction rotated at a predetermined rotation angle θ from a straight line connecting the subject and the center of gravity of a background model that is the subject of interest, with the position of the subject specified by the user as the center. Then, the viewpoint generating unit 109 sets the line of sight and angle of view from the virtual viewpoint so that the subject and the background model are included in the field of view of this virtual viewpoint (virtual camera). Then, the viewpoint generating unit 109 generates virtual viewpoint information indicating parameters such as the position of the virtual viewpoint, the line of sight from the virtual viewpoint, and the angle of view of the virtual viewpoint, and outputs the virtual viewpoint information to the image generating unit 112. Note that the position of the virtual camera may be set on a straight line connecting the position of the subject specified by the user and the center of gravity of the background model that is the subject of interest. Also, the rotation angle θ may be set by the user. The viewpoint generating unit 109 also includes a viewpoint operating unit, which is a physical user interface such as a joystick or an operation button (not shown), and a display unit for displaying a virtual viewpoint image. The viewpoint operating unit can set parameters for generating virtual viewpoint information, such as the distance between a subject included in the virtual viewpoint image and the virtual viewpoint, the direction from the virtual viewpoint, and the height of the virtual viewpoint. In response to changes in these parameters, the virtual viewpoint image is updated as needed by the image generating unit 112 (described later) and displayed on the display unit. This display unit may share the display unit 113 (described later), or may include a separate display device. The virtual viewpoint information output by the viewpoint generating unit 109 includes information equivalent to external parameters of the camera, such as the position and direction of the virtual viewpoint, information equivalent to internal parameters of the camera, such as the focal length and angle of view, and time information specifying the shooting time to be played back.

[0024] The subject of interest determining unit 110 determines one of a plurality of pre-set subject of interest candidates as the subject of interest based on a subject of interest determining process described below, and outputs it to the viewpoint generating unit 109 .

[0025] A time indication unit 111 generates time information for indicating the time of a virtual viewpoint image to be generated by an image generation unit 112 described below, and outputs the information to the viewpoint generation unit 109 and the interest subject determination unit 110. The time indication unit 111 is provided with at least one user interface, such as a physical user interface including a plurality of buttons and a jog dial (not shown) or a GUI, and is capable of changing the time at which an image is generated by a user operation.

[0026] Based on the time information included in the input virtual viewpoint information, the image generation unit 112 acquires material data at the shooting time from the storage unit 108. The image generation unit 112 generates a virtual viewpoint image at the set virtual viewpoint using the 3D model of the subject and the captured image from the acquired material data, and outputs the generated virtual viewpoint image to the display unit 113.

[0027] The display unit 113 is a display means for displaying the image input from the image generating unit 112. The display unit 113 is configured with a display, a head mounted display (HMD), or the like.

[0028] (How to track the subject position) Next, a method for tracking the three-dimensional position of a subject in this embodiment will be described.

[0029] First, three-dimensional shape estimation section 103 generates a 3D model of the subject, and outputs the generated 3D model to storage section 108 and also outputs the generated 3D model to shape extraction section 104.

[0030] FIG. 2 is a diagram for explaining the tracking process of the subject position in the first embodiment. The shape extraction unit 104 cuts out the lower part of the 3D model of the subject as shown in FIG. 2(b) from the 3D model of the subject as shown in FIG. 2(a). In this embodiment, the cutout is made from the bottom surface of the circumscribed rectangular parallelepiped of the 3D model of the subject to a predetermined height (e.g., a height equivalent to 50 cm). For example, as shown in FIG. 2(c), when one of the subjects is standing and one of the subjects is away from the floor surface of the shooting area by jumping or the like, the 3D model of the subject is cut out as shown in FIG. 2(d). That is, for both of the 3D models of the subjects, the 3D model is cut out from the part equivalent to the feet to a predetermined height. Note that the area cut out from the 3D model is not limited to the lower part of the 3D model. For example, the central part of the 3D model of the subject (e.g., an area from 50 cm to 100 cm from the bottom surface of the circumscribed rectangular parallelepiped of the 3D model) may be cut out.

[0031] Next, as shown in FIG. 2(e), the shape extraction unit 104 projects the cut-out 3D model onto the floor to generate a two-dimensional image. In this embodiment, the projected image is a binary image in which the cut-out 3D model part is white and the other part is black. The shape extraction unit 104 divides this two-dimensional image into independent regions and obtains circumscribing rectangles 201 to 204 shown in FIG. 2(e). The shape extraction unit 104 outputs vertex information of this circumscribing rectangle as the extracted shape of the extracted 3D model. Specifically, the shape extraction unit 104 converts the vertex information of the circumscribing rectangle into the same coordinate system and unit as the three-dimensional space of the shooting area and outputs it. In addition, to determine the independent shape in the shape extraction unit 104, a method such as continuous component analysis is used for the projected two-dimensional image. By using such a method, the shape extraction unit 104 can divide the 3D model into individual regions.

[0032] FIG. 3 is a diagram for explaining the process of assigning an identifier to the subject position in the first embodiment. The identification setting unit 105 assigns an identifier to the extracted shape output by the shape extraction unit 104. Specifically, the identification setting unit 105 calculates the distance between each extracted shape, and assigns an identifier according to the distance between the extracted shapes. For example, as shown in FIG. 3(a), the identification setting unit 105 assigns the same identifier to extracted shapes whose distance between the extracted shapes is less than a predetermined distance (solid line arrow), and assigns a different identifier to extracted shapes whose distance between the extracted shapes is equal to or greater than the predetermined distance (dashed line arrow). The threshold value of the predetermined distance used as the criterion for judgment is preferably a distance equivalent to the width of the subject's legs when standing. In this embodiment, the explanation will be given assuming that the threshold value of the predetermined distance is set to 50 cm.

[0033] The identification setting unit 105 displays the assigned identifiers on a display unit provided in the identification setting unit 105 by a graphical user interface (GUI) as shown in FIG. 3(b). The user operates the image processing system while viewing this GUI. Specifically, the identification setting unit 105 displays the current identifier assignment (identifier assignment in the initial state) on the GUI by distinguishing it with at least one of characters and color coding. In FIG. 3(b), the identification setting unit 105 displays each identifier by using both characters and color coding. The user checks the GUI and checks whether the desired identifier assignment is set as the initial state. If the desired identifier assignment is not set, the user instructs the subject to change its standing position or to close its legs, and repeats this until the desired assignment is set. Alternatively, the user operates the image processing system via the GUI and issues a change instruction so that the desired identifier assignment is set. If the desired identifier assignment is set, the user presses, for example, a decision button (initial identifier decision button) on a graphical user interface as shown in FIG. 3(b). In response to this operation, the identification setting unit 105 decides the initial state identifier. Furthermore, the type of each subject may be set for the identifier in the initial state. It is assumed that this type is set for each imaging target (for example, baseball, live performance, etc.), and in this embodiment, a basketball is assumed, and one of the types of a player, a referee, or a ball is set for each subject. It is assumed that this type is set by user input, but is not limited to this. For example, in the initial state, a condition may be set in advance such that a subject corresponding to an extracted shape that does not have other extracted shapes within a predetermined distance is set as a ball, and the type may be set at the same time as the identifier is determined. Then, the identification setting unit 105 outputs information indicating the identifier and type determined for each extracted shape to the tracking unit 106.

[0034] In response to receiving an identifier from the identification setting unit 105, the tracking unit 106 assigns the identifier to each extracted shape as an initial state. After that, the tracking unit 106 tracks the extracted shape to which the identifier has been assigned. The identifier assigned to the extracted shape during tracking is not the identifier determined by the identification setting unit 105, but an identifier determined based on the tracking result of the position of each extracted shape by the tracking unit 106. In tracking the extracted shape, the tracking unit 106 tracks the extracted shape based on the position of each extracted shape at the time immediately before the shooting time of the extracted shape, the identifier of each extracted shape, and information on the subject position input from the subject position calculation unit described later. The specific processing of tracking by the tracking unit 106 will be described later. The tracking unit 106 assigns an identifier to each extracted shape at the time based on the tracking result, and outputs each extracted shape to the subject position calculation unit 107.

[0035] Fig. 4 is a diagram showing the representative position of a subject in embodiment 1. The subject position calculation unit 107 obtains a representative position for each extracted shape to which an identifier obtained from the tracking unit 106 has been assigned. For example, as shown in Fig. 4, the subject position calculation unit 107 obtains a position indicating each extracted shape group for each extracted shape group to which the same identifier has been assigned, such as representative position 401 and representative position 402. In this embodiment, the representative position is the center position of the extracted shape group.

[0036] However, since this representative position is affected by shape estimation errors and fluctuations in boundary parts when the shape is cut out by the shape extraction unit 104, the position may fluctuate at each time even if the subject is stationary. Therefore, in this embodiment, the subject position calculation unit 107 performs processing such as low-pass filtering or moving average in the time direction on the center position information at each time to generate position information in which high-frequency components are suppressed. Then, the subject position calculation unit 107 outputs the position information of the representative position together with the identifier as the position of the subject to the tracking unit 106. In addition, the subject position calculation unit 107 records (stores) information in which information on the time when the subject was photographed is added to the position information of the representative position in the storage unit 108 as position information of the subject (subject position information).

[0037] (Tracking process by tracking unit 106) FIG. 5 is a flowchart showing a process in which the tracking unit 106 tracks the position of a subject in the first embodiment.

[0038] In step S501, the tracking unit 106 receives an input from the identification setting unit 105 and acquires the identifier and type of each extracted shape.

[0039] In step S 502 , the tracking unit 106 acquires the extracted shape that is next input from the shape extraction unit 104 .

[0040] In step S503, the tracking unit 106 assigns the identifiers and types acquired from the identification setting unit 105 to each of the acquired extracted shapes, and outputs each extracted shape to which the identifiers and types have been assigned, to the subject position calculation unit 107.

[0041] In step S504, the subject position calculation unit 107 calculates the subject position from the group of extracted shapes to which the same identifier is assigned, and outputs the subject position to the tracking unit .

[0042] The above processing in steps S501 to S504 corresponds to the initialization processing.

[0043] The subsequent steps S505 to S509 are executed every time while the imaging unit 101 is imaging the subject. When the imaging process of the subject by the imaging unit 101 is completed, the process of step S509 is completed and the process of this flowchart is terminated.

[0044] In step S505, tracking unit 106 acquires the extracted shape input from shape extraction unit 104 and the subject position at the previous time (previous time) calculated by subject position calculation unit 107. The previous time is, for example, the shooting time of the extracted shape generated one frame before the extracted shape currently being processed. Here, for comparison, the current time is also referred to as the current time. Here, the current time refers to the shooting time of the image used to generate the extracted shape currently being processed.

[0045] In step S506, when the subject position at the previous time overlaps with the representative position of each extracted shape at the current time, the tracking unit 106 assigns to the extracted shape an identifier that is assigned to the subject position that overlaps with the representative position. Here, in step S506, when the representative position of one extracted shape overlaps with multiple subject positions, the tracking unit 106 assigns to the extracted shape an identifier indicating "undeterminable" at the current time. This is because, for example, multiple extracted shapes with different identifiers assigned may overlap at the current time, such as when two subjects are close to each other, so an identifier indicating "undeterminable" is assigned in the processing of this step. The extracted shape to which an identifier including an identifier indicating "undeterminable" is assigned is subjected to the processing of S509 described later.

[0046] In step S507, if the representative position of an extracted shape to which an identifier has not yet been assigned overlaps with the extracted shape of the previous time, the tracking unit 106 assigns the identifier assigned to the extracted shape of the previous time to the extracted shape of the current time.

[0047] In step S508, if there is another extracted shape to which an identifier has already been assigned at the current time within a predetermined range from an extracted shape to which an identifier has not yet been assigned, the tracking unit 106 assigns the identifier assigned to the other extracted shape. The predetermined range is preferably a range equivalent to the width of the subject's legs when standing. For example, the predetermined range is a range of a radius of 50 cm from the center of the extracted shape. Here, if there are multiple other extracted shapes to which an identifier has been assigned within a predetermined range from a certain extracted shape, the tracking unit 106 assigns the identifier of the extracted shape that is closest among the other extracted shapes to the extracted shape. For an extracted shape to which an identifier has not been assigned at the stage of completing the processing up to step S508, the tracking unit 106 determines that the extracted shape is not a tracking target. In this case, the tracking unit 106 does not output the extracted shape determined to be not a tracking target to the subject position calculation unit 107.

[0048] In step S509, the tracking unit 106 outputs to the subject position calculation unit 107 the extracted shape to which an identifier has been assigned in the processes of steps S506 to S508, and the identifier assigned thereto.

[0049] In step S510, a control unit (not shown) determines whether or not the imaging process of the subject by the imaging unit 101 has ended. If it is determined that the imaging process of the subject by the imaging unit 101 has not ended, the process of step S508 is executed. If it is determined that the imaging process of the subject by the imaging unit 101 has ended, the process of this flowchart ends.

[0050] Note that the processes from step S506 to step S508 are performed for each extracted shape. By repeating the processes from step S506 to step S509, the identifier set by the identification setting unit 105 is linked to the extracted shape at each time. Using this identifier, the subject position calculation unit 107 can distinguish between each subject and calculate the subject position.

[0051] In addition, when the tracking unit 106 assigns an "undeterminable" identifier to the extracted shape, there is a possibility that some of the identifiers set in the initial setting will not be assigned at a certain time. In such a case, the subject position calculation unit 107 does not update the subject position information that is the same as the identifier that is not assigned to the extracted shape. As a result, even if the extracted shapes overlap due to multiple subjects approaching each other, the multiple subject position information will not be in the same position. In this case, the multiple subject positions are maintained at their respective positions up to the previous time. After that, when the subjects move away from each other and the multiple overlapping extracted shapes are separated again, an identifier is assigned to each extracted shape based on the most recent subject position. In other words, in response to the elimination of the overlap of the multiple extracted shapes, the update of each subject position information is resumed.

[0052] By the above processing, the image processing system can track each subject and obtain the position information of each subject even when multiple subjects are in the shooting area. Furthermore, by the above processing, the image processing system can track each subject even when the subjects approach or move away from each other, causing overlapping or separation in the generated 3D model.

[0053] (Interest Determination Process) 6 is a flowchart showing the process of determining an object of interest by the object of interest determining unit 110 in the first embodiment. In this embodiment, an example of determining an object of interest when generating a virtual viewpoint image of basketball will be described. In this case, the object of interest candidates are the basketball goals of both teams, and position information of the object of interest candidates is acquired in advance from the storage unit 108. This process is performed for each frame.

[0054] 7A and 7B are diagrams showing the velocity vectors and average velocity vectors of each subject in this embodiment. As shown in Fig. 7A, coordinate axes x and y are defined, the center of the court is set as the origin (0,0), and the basketball goal is located on the x-axis.

[0055] In step S601, the subject of interest determining unit 110 acquires subject position information based on the time specified by the time specifying unit 111 from the subject position information detected by the subject position detecting unit 114 and recorded in the storage unit 108. At this time, not only the subject position information at the specified time but also the subject position information for several to several tens of frames before and after that time is acquired. The following steps S602 to S605 are processed for each subject.

[0056] In step S602, it is determined whether a certain subject is a subject to be calculated. Specifically, it is determined whether to perform subsequent processing based on whether the type corresponding to the identifier for identifying the subject included in the position information matches the calculation target information. The calculation target information is information that specifies the calculation target of the interest target determination process, and is information that specifies the type corresponding to the subject's identifier. For example, calculation target information is generated such that subjects whose type corresponding to the identifier is a player are calculated, and is recorded in advance in the storage unit 108 or the interest target determination unit 110. In this case, subjects whose types are umpire and ball can be excluded from the interest target determination process by excluding them from the calculation target.

[0057] In step S603, a filtering process is performed on the position information of the object determined to be the object of calculation. Specifically, position information for multiple frames acquired in S601 is averaged to calculate position information. This makes it possible to reduce blurring due to detection errors of the object position detection unit 114.

[0058] In step S604, subject position information from a time earlier than the time of the subject to be calculated, for example, one second ago and several to several tens of frames (predetermined time) before and after that time is acquired and an average value is calculated.

[0059] In step S605, the position information after averaging at a past time is subtracted from the position information after averaging at the current time, and then the difference in position information between the current time and the past time is divided by the time difference between the current time and the past time, here by one second, to obtain the position deviation (position change) at the current time unit time. The magnitude of this position deviation at the unit time is the moving speed, and the direction of the vector on the two-dimensional plane is the moving direction of the subject. The above steps S602 to S605 are performed for each subject. When the process has been completed for all subjects, proceed to S606.

[0060] In step S606, an average velocity vector of the obtained velocity vectors of all the objects to be calculated is calculated. Fig. 7(b) shows the calculated average velocity vector of all the objects to be calculated. In this embodiment, the component of the goal direction contained in this average velocity vector is obtained. Specifically, since the goal direction is the positive and negative directions of the x-axis, the direction and magnitude of the vector obtained by orthogonally projecting the average velocity vector onto the vector (1,0) are obtained.

[0061] In step S607, it is determined whether the average velocity vector calculated in S606 satisfies the target of interest determination condition. It is assumed that the target of interest determination condition differs for each imaging target. In this embodiment, it is determined whether the vector obtained by projecting the average velocity vector in the goal direction satisfies the target of interest determination condition. In the initial state where the target of interest has not been determined, the process proceeds to S608 regardless of whether the target of interest determination condition is satisfied. In this embodiment, the target of interest determination condition is whether the direction of the vector obtained by projecting the average velocity vector in the goal direction is toward the current target of interest, and whether the magnitude of the vector is greater than a predetermined magnitude. When the current target of interest has been determined, a change in the target of interest is determined based on the direction and magnitude of the orthogonally projected vector. If the orthogonally projected vector is toward the positive direction, is greater than or equal to a predetermined magnitude, and the current target of interest is not the x-axis positive side goal 71, the process proceeds to S608. If the vector is toward the negative direction, is greater than or equal to a predetermined magnitude, and the currently focused goal is not the x-axis negative side goal 72, the process proceeds to S608. If neither of the two conditions is met, the current subject of interest is left unchanged, and the process of determining the subject of interest in the current frame ends.

[0062] In step S608, one of the multiple candidates for the target of interest is determined based on the average velocity vector. In the initial state where the target of interest has not been determined, a straight line is extended in the direction of the average velocity vector, and the target of interest closest to the line is identified. In addition, if there are multiple targets of interest closest to the line, the target of interest closest to the position of the subject of interest is identified. Alternatively, the initial value of the target of interest may be set in advance by a user operation. In this embodiment, a determination is made as to whether or not to change the target of interest based on the direction and magnitude of the orthogonally projected vector. If the orthogonally projected vector is oriented in the positive direction, is equal to or greater than a predetermined magnitude, and the current target of interest is not the x-axis positive side goal 71, the goal 71 is determined (specified) as the target of interest. In addition, if the vector is oriented in the negative direction, is equal to or greater than a predetermined magnitude, and the currently focused goal is not the x-axis negative side goal 72, the goal 72 is determined as the target of interest.

[0063] By performing the processes of S601 to S608 for each frame, the movements of all subjects are detected and a subject of interest is determined. The subject of interest determining unit 110 outputs information on the determined subject of interest to the viewpoint generating unit 109.

[0064] As described above, the viewpoint generator 109 places a virtual viewpoint in a direction rotated by a predetermined angle from the straight line connecting the subject and the object of interest. When the object of interest switches from goal 71 to goal 72, the viewpoint does not switch immediately, but rather calculates the current angle and the angle after the switch, and rotates by this angle difference over a predetermined time to complement the virtual viewpoint. This allows the viewer to correctly recognize that the object of interest he or she is currently focusing on has switched when the object of interest switches.

[0065] In this way, the present disclosure makes it possible to detect a change in offense and defense from the movements of all subjects in sports such as basketball, and to automatically generate a virtual viewpoint that captures the goal of the attacking team by the viewpoint generating unit 109 accordingly. This makes it possible to create a virtual viewpoint image suited to the scene, while tracking a subject specified by the user and including it in the virtual viewpoint image.

[0066] In this embodiment, the goal direction component of the average velocity vector is calculated in S606, but which direction component is calculated varies depending on the imaging subject and the candidate interest subject. For example, when the imaging subject is a 100m sprint, the straight line from the start to the goal is set as the x-axis direction. In this way, which direction component of the average velocity vector is calculated varies depending on the imaging subject and the candidate interest subject. Therefore, for example, when a combination of multiple candidate interest subjects is determined, a predetermined component direction may be set and stored in advance for the combination.

[0067] In this embodiment, the component of the average velocity vector in the goal direction is calculated in S606, but this is not limiting. The target of interest may be identified from the direction of the average velocity vector. Specifically, a straight line is set in the direction of the average velocity vector, and the target of interest closest to the straight line is identified. There is also a possibility that there are multiple target of interest candidates closest to the straight line. In this case, the target of interest closest to the position of the subject of interest is identified. This makes it possible to identify one target of interest from multiple target of interest candidates.

[0068] (Other aspects of embodiment 1) In this embodiment, subject position detection means based on shape estimation results has been described as subject position detection unit 114, but the present disclosure is not limited to this method of detecting the subject position. For example, a configuration may be used in which a position sensor such as a GPS is attached to the player and the sensor value is acquired, or a configuration may be used in which the subject position is detected using image recognition technology from images obtained by multiple imaging means.

[0069] In addition, in this embodiment, the subject of interest determination unit 110 is configured to calculate the speed and movement direction for each frame when generating a virtual viewpoint image, but this is not necessarily limited to this. For example, the subject position detection unit 114 may detect the position, calculate the speed vector of each subject, and record it in the storage unit 108. In this case, the subject of interest determination unit 110 may obtain the speed vector of each subject from the storage unit 108 and determine the subject of interest.

[0070] In this embodiment, the filtering process of the position information is performed by averaging, but the present invention is not limited to this. For example, a low-pass filter such as an IIR filter or an FIR filter may be used. However, in the case of a configuration in which the speed or the like is calculated each time, if a low-pass filter is used, the value becomes incorrect when the time at which the virtual viewpoint image is reproduced is changed discontinuously. Therefore, in such a configuration, it is desirable to obtain information of nearby times and then average the information as described above.

[0071] In this embodiment, basketball is used as an example, but it may be applied to sports such as soccer and rugby. For example, in the case of soccer, the game progresses more slowly than in basketball, and the players may be stationary for a relatively long period of time. In such a case, it is possible to determine which side of the field there are more subjects (players), and determine the subject of interest by combining the result of this determination with the velocity vectors of the subjects.

[0072] <Embodiment 2> In the first embodiment, the object of interest is determined from the average velocity vector of all subjects that are players. However, the object of interest may be determined based on the movement of one player. For example, when generating a virtual viewpoint image that follows a baseball runner, the object of interest may be determined by focusing on the movement of the runner. In this embodiment, an example in which the imaging object is a baseball will be described. In this embodiment, the block diagram is the same as in FIG. 1, and components that are not particularly described are the same as those in the first embodiment, so their description will be omitted.

[0073] 9 is a flowchart showing the process of determining an object of interest by the object of interest determining unit 110 in embodiment 2. This process is performed for each frame.

[0074] 10 is a diagram showing an example of interest object candidates and a determination region (predetermined region) in embodiment 2. In this embodiment, a home base 1005 and first base 1006, second base 1007, and third base 1008 in baseball are set as interest object candidates.

[0075] In step S901, the subject of interest determining unit 110 sets a first judgment area 1001, a second judgment area 1002, a third judgment area 1003, and a fourth judgment area 1004 by a user operation on a user interface (not shown). At this time, a subject of interest candidate to be linked to each judgment area is set. Specifically, a home base 1005 and a first base 1006 are linked to the first judgment area 1001. Similarly, a first base 1006 and a second base 1007 are linked to the second judgment area 1002, a second base 1007 and a third base 1008 are linked to the third judgment area 1003, and a third base 1008 and a home base 1005 are linked to the fourth judgment area 1004. In addition, the subject of interest determining unit 110 acquires position information of the subject of interest candidates linked to the judgment areas and obtains a directional vector connecting the two subject of interest candidates. For example, in the case of the second determination area 1002, a unit direction vector connecting first base 1006 and second base 1007 is obtained. At this time, the unit direction vector is a vector indicating the direction of second base. Note that position information of these determination areas and the candidate target of interest is managed as condition information, and may be stored in the target of interest unit 110 or in the storage unit 108. Note that S901 is not performed for every frame, but is performed only for the first frame in which the target of interest is determined. Note that this is not limited to this, and S901 may be performed at any timing of the user's choice.

[0076] In step S902, the subject of interest determining unit 110 acquires subject position information based on the time specified by the time specifying unit 111 from the subject position information detected by the subject position detecting unit 114 and recorded in the storage unit .

[0077] In step S903, the velocity vector of the subject is obtained, as in embodiment 1. At this time, as in embodiment 1, filter processing is performed on the obtained position information of the subject.

[0078] In step S904, the subject of interest determination unit 110 determines in which of the four preset determination regions the subject is present. In this embodiment, a case will be described in which the subject is present in the second determination region 1002. Next, a plurality of subject of interest candidates corresponding to the second determination region 1002 are identified. In this embodiment, first base 1006 and second base 1007 are the subject of interest candidates.

[0079] In step S905, it is determined whether the velocity vector calculated in S903 satisfies the target of interest determination condition. In the initial state where the target of interest has not been determined as in the first embodiment, the process proceeds to S906 regardless of whether the target of interest determination condition is satisfied. The target of interest determination unit 110 obtains the direction and magnitude of the vector obtained by orthogonally projecting the speed vector of the subject onto the unit direction vector associated with the determination region. In this embodiment, the target of interest determination condition is whether the magnitude of the vector obtained by orthogonal projection is greater than a predetermined magnitude. If the magnitude of the vector obtained by orthogonal projection is greater than the predetermined magnitude, the process proceeds to S906, and if the magnitude of the vector obtained by orthogonal projection is equal to or less than the predetermined magnitude, the process ends.

[0080] In step S906, one of the multiple candidate objects of interest identified in S904 is determined based on the average velocity vector. The processing in the initial state where the object of interest has not been determined is the same as that in the first embodiment. If the magnitude of the vector obtained by orthogonal projection is larger than a predetermined value and the direction of the vector obtained by orthogonal projection is the same as the unit direction vector, toward second base, second base 1007 is determined as the object of interest. If the direction of the vector obtained by orthogonal projection is opposite to the unit direction vector, first base 1006 is determined as the object of interest. By performing the processing of S902 to S906 for each frame, a virtual viewpoint that focuses on an appropriate object of interest while following the movement of the player can be generated.

[0081] FIG. 11 is a diagram showing the movements of a subject, an object of interest, and a virtual viewpoint in the second embodiment. Specifically, the movement of a player (runner) and switching of an object of interest will be described with reference to FIG. 11. Here, a state will be described in which a runner 1101 is in the vicinity of first base 1006 as shown in FIG. 11(a). When the runner 1101 is in the vicinity of first base 1006, the runner has passed through the first judgment area 1001 before reaching the first base 1006, and therefore the object of interest currently being observed is first base 1006. At this time, the runner 1101 makes a lead and enters the second judgment area 1002, but the object of interest remains first base 1006 due to the low speed of movement. The virtual viewpoint generated by the viewpoint generating unit 109 at this time is a virtual viewpoint 1102 that shows the runner 1101 and the first base 1006 from diagonally behind the runner 1101 as shown in FIG. 11(a). Next, when the runner 1101 starts running to advance and his speed exceeds a predetermined speed, the target of interest is switched to second base 1007. As a result, the virtual viewpoint 1102 following the runner 1101 rotates around the runner 1101 as indicated by the dashed arrow 1103 in FIG. 11(b) and automatically transitions to a virtual viewpoint 1102 that shows second base 1007 over the shoulder of the runner 1101 while following the runner 1101. Also, for example, if the runner 1101 tries to return to first base 1006 while advancing, the target of interest is switched to first base 1006 and transitions to a virtual viewpoint 1102 that includes the runner 1101 and the first base 1006. Next, if the runner 1101 successfully advances and then attempts to advance to third base 1008, the runner 1101 enters the third judgment area 1003. As a result, the moving direction and speed are determined based on the unit direction vector defined by second base 1007 and third base 1008, which are linked to the third determination area 1003. As a result, the object of interest becomes third base 1008 when the runner enters the third determination area 1003, and the virtual viewpoint 1102 moves to a position that includes the runner 1101 and third base 1008, as shown in FIG.

[0082] By applying the present disclosure in this manner, it becomes possible to identify an appropriate target of interest from among multiple target of interest candidates based on the position and speed information of the player, and the viewpoint generation unit 109 can generate an appropriate virtual viewpoint based on the identified target of interest. This makes it possible to create appropriate camera work suited to the situation even without an operator to operate the virtual viewpoint.

[0083] (Other aspects of the second embodiment) As in the first embodiment, the subject position detection unit 114 may use another position detection method such as GPS or image recognition.

[0084] In this embodiment, the virtual viewpoint generated by the viewpoint generating unit 109 has been described as a virtual viewpoint that follows the subject from behind, but is not limited to this. Any virtual viewpoint may be generated using information on the subject being watched and the target of interest to be noted. For example, the viewpoint generating unit 109 may be configured to place a virtual viewpoint near the base that is the target of interest, wait for the subject to be tracked, and create camerawork so that the subject fits within the angle of view. In this case, it is desirable that, as the target of interest is switched, the placement of the virtual viewpoint transitions to near the base that is the target of interest while capturing the subject to be tracked from the front.

[0085] In this embodiment, the virtual viewpoint is automatically generated by the viewpoint generation unit 109, and an operator is not required, but this is not necessarily limited to the above. For example, the distance of the virtual viewpoint generated by the viewpoint generation unit 109 from the player, the height at which it is placed, the angle of view, etc. may be controlled by the operator through a user interface (not shown). In this way, the viewpoint generation unit 109 is responsible for fitting the subject and the object of interest within the angle of view, but the operator can control the composition, such as the angle of view, which is more desirable.

[0086] In this embodiment, the present invention is applied to baseball, but it may also be applied to softball or other sports in which the virtual viewpoint to be included in the virtual viewpoint is switched depending on the position of the player.

[0087] In this embodiment, the candidate object of interest is used as a background model, and the position and orientation of the virtual viewpoint are determined based on the position of the subject specified by the user and the center of gravity of the object of interest, but this is not limited thereto. For example, a three-dimensional position different from the center of gravity may be associated with the background model of the candidate object of interest. In this case, the position and orientation of the virtual viewpoint are determined based on the position of the subject specified by the user and the three-dimensional position corresponding to the object of interest. This process makes it possible to easily set a virtual viewpoint suitable for a scene when a goal net is located at a position different from the center of gravity of the background model, such as a basketball goal. The three-dimensional position corresponding to the background model may be set at the same height as the center of gravity of the subject. For example, when the object of interest is a baseball base, it is present on the floor surface, and there is a risk of generating a virtual viewpoint image corresponding to a virtual viewpoint facing the floor surface. Therefore, by adjusting the height of the three-dimensional position corresponding to the background model, it is possible to easily set a virtual viewpoint suitable for a scene.

[0088] (Other configurations) In the above embodiment, each processing unit shown in Fig. 1 is configured as hardware. However, the processing performed by each processing unit shown in these figures may be configured as a computer program.

[0089] FIG. 12 is a block diagram showing an example of the hardware configuration of a computer applicable to the indirect position estimation device according to each of the above embodiments.

[0090] The CPU 1201 controls the entire computer using computer programs and data stored in the RAM 1202 and the ROM 1203, and executes the processes described above as being performed by the indirect position estimation device according to each of the above embodiments. That is, the CPU 1201 functions as each of the processing units shown in FIG.

[0091] The RAM 1202 has an area for temporarily storing computer programs and data loaded from an external storage device 1206, and data acquired from the outside via an I / F (interface) 1207. Furthermore, the RAM 1202 has a work area used when the CPU 1201 executes various processes. That is, the RAM 1202 can be allocated as a frame memory, for example, or can provide various other areas as appropriate.

[0092] The ROM 1203 stores setting data for the computer, a boot program, and the like. The operation unit 1204 is composed of a keyboard, a mouse, and the like, and can be operated by a user of the computer to input various instructions to the CPU 1201. The output unit 1205 displays the results of processing by the CPU 1201. The output unit 1205 is composed of, for example, a liquid crystal display. For example, the viewpoint generation unit 109 is composed of the operation unit 1204, and the display unit 113 is composed of the output unit 1205.

[0093] The external storage device 1206 is a large-capacity information storage device, such as a hard disk drive. The external storage device 1206 stores an operating system (OS) and computer programs for causing the CPU 1201 to realize the functions of each unit shown in Fig. 1. Furthermore, the external storage device 1206 may store image data to be processed.

[0094] Computer programs and data stored in the external storage device 1206 are loaded into the RAM 1202 as appropriate under the control of the CPU 1201, and become the subject of processing by the CPU 1201. A network such as a LAN or the Internet, and other devices such as a projector or display device can be connected to the I / F 1207, and the computer can obtain and transmit various information via the I / F 1207. In the first embodiment, the imaging unit 101 is connected to this, and inputs captured images and controls each of them. 1208 is a bus connecting the above-mentioned units.

[0095] The operation of the above-mentioned configuration is controlled mainly by the CPU 1201 as described in the above embodiment.

[0096] A computer program for implementing all or part of the control in the present embodiment may be provided to an image processing system or the like via a network or various storage media. A computer (or a CPU, MPU, or the like) in the image processing system or the like may then read and execute the program. In this case, the program and the storage medium storing the program constitute the present disclosure.

[0097] The disclosure of this embodiment includes the following configurations, methods, and programs.

[0098] (Configuration 1) An acquisition means for acquiring position information indicating a position of a subject included in an imaging area captured by a plurality of imaging devices; A specifying means for specifying one of a plurality of positions of interest based on a moving direction of the subject; a determining means for determining a position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and a line of sight direction from the virtual viewpoint based on the position of interest and the position of the subject specified by the specifying means; 13. An information processing device comprising:

[0099] (Configuration 2) The position information indicates positions of a plurality of subjects, 2. The information processing apparatus according to configuration 1, wherein the identifying means identifies one of the plurality of points of interest based on a moving direction of the plurality of subjects.

[0100] (Configuration 3) The information processing device according to configuration 2, wherein the identification means identifies one of the plurality of positions of interest based on a moving direction and a moving speed of the plurality of subjects.

[0101] (Configuration 4) The plurality of subjects are classified into a plurality of types including a first type and a second type; 3. The information processing device according to configuration 2, wherein the identification means identifies one of the plurality of positions of interest based on a moving direction of the plurality of subjects corresponding to the first type.

[0102] (Configuration 5) The plurality of imaging devices capture images of a competition, The first type is a player; 5. The information processing device according to configuration 4, wherein the second type is a referee.

[0103] (Configuration 6) The acquiring means acquires condition information indicating the plurality of positions of interest corresponding to a predetermined area in a three-dimensional space, The information processing device described in configuration 1, characterized in that when the subject is located within the specified area indicated by the condition information, the identification means identifies one position of interest from the multiple positions of interest corresponding to the specified area.

[0104] (Configuration 7) The information processing device according to configuration 1, wherein the moving direction of the subject is identified based on a change in position of the subject over a predetermined period of time.

[0105] (Configuration 8) The information processing device according to configuration 1, wherein the position of interest is set based on a user operation.

[0106] (Configuration 9) The information processing device according to configuration 1, wherein the position of interest corresponds to a background model.

[0107] (Configuration 10) The information processing device according to configuration 9, wherein the background model is a stationary object.

[0108] (Configuration 11) The information processing device according to configuration 10, which identifies the background model as a basketball goal.

[0109] (Configuration 12) The information processing device according to configuration 1, characterized in that the determination means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint based on a half line extended from the position of the subject to the position of interest.

[0110] (Configuration 13) The information processing device described in Configuration 12, characterized in that the determination means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint so that the angle formed by a half line extended from the position of the subject to the position of interest and the line of sight from the virtual viewpoint is a predetermined angle.

[0111] (Configuration 14) The information processing device according to configuration 1, wherein the determination means determines the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint so that the subject and the position of interest are included in the virtual viewpoint image.

[0112] (Method) An acquisition step of acquiring position information indicating a position of a subject included in an imaging area captured by a plurality of imaging devices; identifying one of a plurality of positions of interest based on a moving direction of the subject; a determining step of determining a position of a virtual viewpoint corresponding to a virtual viewpoint image including the subject and a line of sight direction from the virtual viewpoint based on the position of interest and the position of the subject identified by the identifying step; 13. An information processing method comprising:

[0113] (Program) A program for causing a computer to function as the information processing device according to any one of configurations 1 to 14. [Explanation of symbols]

[0114] 104 Shape extraction part 105 Identification setting section 106 Tracking Department 107 Subject position calculation section 109 Viewpoint Generation Unit 110 Interest Determination Division 111 Time indicator

Claims

1. an acquisition means for acquiring position information indicating a position of a subject included in an imaging area captured by a plurality of imaging devices; A specifying means for specifying one of a plurality of positions of interest based on a moving direction of the subject; a determining means for determining a position of a virtual viewpoint corresponding to a virtual viewpoint image related to the subject and a line of sight direction from the virtual viewpoint based on the position of interest and the position of the subject specified by the specifying means; 13. An information processing device comprising:

2. the position information indicates positions of a plurality of subjects; The information processing apparatus according to claim 1 , wherein the specifying unit specifies one of the plurality of positions of interest based on a moving direction of the plurality of subjects.

3. The information processing apparatus according to claim 2 , wherein the specifying means specifies one of the plurality of positions of interest based on a moving direction and a moving speed of the plurality of subjects.

4. The plurality of subjects are classified into a plurality of types including a first type and a second type, The information processing apparatus according to claim 2 , wherein the specifying unit specifies one of the plurality of positions of interest based on a moving direction of the plurality of subjects corresponding to the first type.

5. The plurality of imaging devices capture images of a sport; the first type being a player; 5. The information processing apparatus according to claim 4, wherein the second type is a referee.

6. the acquiring means acquires condition information indicating the plurality of positions of interest corresponding to a predetermined area in a three-dimensional space; The information processing device according to claim 1 , characterized in that, when the subject is located within the specified area indicated by the condition information, the identification means identifies one position of interest from the multiple positions of interest corresponding to the specified area.

7. 2 . The information processing apparatus according to claim 1 , wherein the moving direction of the subject is identified based on a change in position of the subject over a predetermined period of time.

8. The information processing apparatus according to claim 1 , wherein the position of interest is set based on a user operation.

9. The information processing apparatus according to claim 1 , wherein the position of interest corresponds to a background model.

10. The information processing apparatus according to claim 9 , wherein the background model is a stationary object.

11. The information processing apparatus according to claim 10 , wherein the background model is specified to be a basketball goal.

12. 2 . The information processing apparatus according to claim 1 , wherein the determining means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint based on a half line extending from the position of the subject to the position of interest.

13. The information processing device according to claim 12, characterized in that the determination means determines the position of the virtual viewpoint and the line of sight from the virtual viewpoint so that an angle formed by a half line extended from the position of the subject to the position of interest and the line of sight from the virtual viewpoint is a predetermined angle.

14. 2 . The information processing apparatus according to claim 1 , wherein the determining means determines the position of the virtual viewpoint and the line of sight direction from the virtual viewpoint so that the subject and the position of interest are included in the virtual viewpoint image.

15. an acquisition step of acquiring position information indicating a position of a subject included in an imaging area captured by a plurality of imaging devices; a step of identifying one of a plurality of positions of interest based on a moving direction of the subject; a determining step of determining a position of a virtual viewpoint corresponding to a virtual viewpoint image including the subject and a line of sight direction from the virtual viewpoint based on the position of interest and the position of the subject identified by the identifying step; 13. An information processing method comprising:

16. A program for causing a computer to function as the information processing device according to any one of claims 1 to 14.

Images