Information processing device, information processing method, and program

Abstract

To display a virtual viewpoint image from the viewpoint of a person whose viewpoint is assumed to be of high viewing value in a scene in which the person is present.SOLUTION: The present invention controls a virtual camera corresponding to a virtual viewpoint image generated on the basis of a plurality of captured images captured by a plurality of imaging devices, determines whether an object included in a virtual space exists in a first predetermined area, and controls the position of the virtual camera on the basis of the position of the object when it is determined that the object exists in the first predetermined area.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present invention relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] In recent years, technology has been used in fields such as sports viewing, in which 3D data is generated from multiple 2D images captured by multiple cameras, and a 2D image is then generated based on a virtual viewpoint set for that 3D data. Unlike regular sports broadcasts using television images, this technology makes it possible to set a virtual viewpoint independent of the position of the capturing camera, making it possible to watch the sport from a viewpoint close to that of the athletes playing. Furthermore, by detecting the position of the athletes' faces and eyes, it is possible to view images from the athletes' point of view.

[0003] Patent Document 1 discloses a method for providing images with a field of view seen from the perspective of a player and at least one referee in a match when the player and at least one referee in the match are moving around. In other words, the technology described in Patent Document 1 provides images from the player's perspective on the condition that the player and at least one referee in the match are moving around. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Patent No. 6761811 Summary of the Invention [Problem to be solved by the invention]

[0005] Compared to viewing images with a bird's-eye view of the playing area, images from the player's perspective can provide a particularly realistic image. However, viewing from the player's perspective is not always appropriate during a game. For example, in baseball, during a batter's change, the player's perspective may be directed toward a direction with low viewing value, such as the spectator seats, the coach, or the ground, since the game is not currently in progress. In such scenes, the player's perspective is hardly considered to be a scene with high viewing value for the viewer, and using a virtual viewpoint from the player's perspective may result in a decrease in viewing value.

[0006] Furthermore, in baseball, for example, players and umpires hardly move until the pitcher throws the ball, and the pitching motion is performed while standing on the pitcher's plate, so it is difficult to say that they are moving around. Furthermore, scenes with high viewing value for viewers begin before the pitch starts, so with the technology of Patent Document 1, viewers may miss high-viewing scenes if they are shown after the pitching motion has started. Furthermore, during batter changes, the batter moves around, but such scenes are difficult to say that they are high-viewing value for viewers.

[0007] The present invention aims to display a virtual viewpoint image from the viewpoint of a person in a scene in which the person is present and whose viewpoint is assumed to be of high viewing value. [Means for solving the problem]

[0008] To achieve the object of the present invention, for example, an information processing device according to one embodiment has the following configuration: That is, the information processing device controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, and includes a first determination means that determines whether an object exists in a first predetermined area, and a control means that, when it is determined that the object exists in the first predetermined area, controls the position of the virtual camera based on the position of the object. [Effects of the Invention]

[0009] In a scene in which a person whose viewpoint is assumed to be of high viewing value is present, a virtual viewpoint image from the viewpoint of that person is displayed. [Brief explanation of the drawings]

[0010] [Figure 1] FIG. 1 is a diagram showing an example of an application environment for processing by an information processing apparatus. [Figure 2] FIG. 1 is a diagram showing an example of a system including an information processing apparatus. [Figure 3] FIG. 1 is a diagram showing an example of a hardware configuration of an information processing apparatus. [Figure 4] FIG. 1 is a diagram showing an example of the functional configuration of an information processing apparatus according to a first embodiment. [Figure 5] 4 is a flowchart showing an example of information processing according to the first embodiment. [Figure 6] FIG. 2 is a diagram for explaining an application example of the information processing device according to the first embodiment. [Figure 7] FIG. 10 is a diagram showing an example of an image of a scoreboard. [Figure 8] A diagram to explain VR goggles. [Figure 9] FIG. 10 is a diagram showing an example of a GUI for mode selection. [Figure 10] FIG. 10 is a diagram showing an example of the functional configuration of an information processing device according to a second embodiment. [Figure 11] 10 is a flowchart showing an example of information processing according to the second embodiment. [Figure 12] FIG. 10 is a diagram for explaining an application example of an information processing device according to a second embodiment. [Figure 13] FIG. 10 is a diagram showing an example of the functional configuration of an information processing device according to a third embodiment. [Figure 14] 11 is a flowchart showing an example of information processing according to the third embodiment. [Figure 15] FIG. 10 is a diagram for explaining an application example of an information processing device according to a third embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0011] Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the invention according to the claims. Although multiple features are described in the embodiments, not all of these multiple features are necessarily essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same reference numerals are used to designate the same or similar components, and redundant explanations will be omitted.

[0012] [Embodiment 1] FIG. 1 is a diagram showing a baseball stadium (sports venue) as an example of an application environment for processing by an information processing device according to this embodiment. The field 100 is the field on which baseball is played, and includes bases, batter's boxes, foul lines, etc. In this embodiment, part or all of the field 100 is considered to be the area to be imaged. The cameras 101 to 112 are imaging devices that each capture part or all of the field 100 from various directions.

[0013] FIG. 2 is a diagram illustrating an example of the configuration of a system including an information processing device 201 according to this embodiment. In the system illustrated in FIG. 2, images captured by the above-described cameras 101 to 112 are input to the information processing device 201 via a relay device 200 and subjected to various processes. The information processing device 201 is, for example, a server, and controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices. The processes executed by the information processing device 201 will be described later. Each process according to this embodiment will be described later. The display 202 is a display unit that displays the processing results by the information processing device 201, including the virtual viewpoint image. The display 202 may be, for example, a PC monitor or a screen of a mobile terminal, and is not particularly limited as long as it is capable of displaying each piece of information. Furthermore, the processing results by the information processing device 201 may not only be displayed on the display 202, but may also be transmitted to another computer or display device using a communication unit (not shown) and displayed thereon.

[0014] FIG. 3 is a block diagram showing an example of the hardware configuration of the information processing device 201. The CPU 300 is a processing device that executes the operation of a predetermined program. The ROM 301 is a memory that stores predetermined programs, etc. The RAM 302 is a memory that stores programs or data that are currently being executed. The memory unit 303 is a storage device for storing programs and data, and is configured from an HDD, SSD, etc. The GPU 304 performs output processing to the display 202 as needed, and also performs some processing such as matrix calculations on behalf of the CPU 300. The communication unit 305 is an interface for communicating with other devices, and performs communication when inputting images from the cameras 101 to 112 or outputting generated virtual viewpoint images to another computer.

[0015] Each process described below as being executed by the information processing device 201 is executed by the CPU 300 or the GPU 304 in accordance with a program stored in the ROM 301, the storage unit 303, or the like. However, some or all of such processes may be realized by a circuit (not shown), such as an ASIC. Furthermore, the configuration shown in FIG. 1 uses a large number of cameras, which may result in a very high processing load. Therefore, although the information processing device 201 will be described as a single device in this embodiment, the information processing device 201 may be configured with multiple devices. Furthermore, the information processing device 201 may include multiple CPUs 300 or GPUs 304.

[0016] 4 is a block diagram showing an example of the configuration of the functional units of the information processing device 201 according to this embodiment. An image input terminal 400 acquires an image from the outside (here, an image capturing device such as cameras 101 to 112). A data generation unit 401 generates three-dimensional data of an object captured by a virtual camera. Here, the data generation unit 401 generates three-dimensional data based on a plurality of images input via the image input terminal 400, with an object shown in the image being the image capturing object. In the following description, a person is used as the image capturing object, but the invention is not particularly limited to this, and the image capturing object may be, for example, a vehicle or an animal.

[0017] The person information detection unit 402 detects person information including the position of a person in an image, or the position or orientation of a face or eyes. The person information detection unit 402 according to this embodiment detects person information by image processing of a captured image input via the image input terminal 400, but is not limited to such processing as long as the person information described above can be acquired. For example, the person information detection unit 402 may detect person information based on the three-dimensional data generated by the data generation unit 401.

[0018] The area information input terminal 403 is an input unit that acquires information about the determination area. The "determination area" according to this embodiment is an area in which the determination unit 404 determines whether or not at least one of the objects (people in the image) is present. The determination unit 404 determines whether or not at least one of the objects is present in the determination area. Here, the determination unit 404 determines whether or not a person is present in the determination area based on the person information and the information about the determination area.

[0019] Here, the determination area is an area that can be arbitrarily set by the user (for example, in advance). For example, by setting an area corresponding to the batter's box as the determination area, it becomes possible to determine whether or not the batter is in the batter's box (i.e., in this case, it is a scene of high value to the viewer). When a predetermined determination area (determined before the start of the game) is used, for example, the results of measurement within the range set as the predetermined set area may be stored in advance in the storage unit 303 or an external server (not shown) and read.

[0020] When it is determined that a person is present in the determination area, the virtual viewpoint setting unit 405 controls the virtual camera based on the position of the person. As will be described in detail later, when it is determined that a person is present in the determination area, the virtual viewpoint setting unit 405 can, for example, match the position of the virtual camera with the position of the person and control the imaging direction (posture) of the virtual camera to match the orientation of the person's face. Hereinafter, the virtual camera to be controlled in this way may be referred to as a "virtual viewpoint." Furthermore, hereinafter, determining that the virtual camera can be controlled based on the position of the person may be expressed as "a person's viewpoint is possible" or "a person's viewpoint is suitable."

[0021] The image generation unit 406 generates a virtual viewpoint image based on the three-dimensional data and the virtual viewpoint. The image output terminal 407 outputs the virtual viewpoint image to a display or an external device.

[0022] Fig. 5 is a flowchart showing an example of information processing by the information processing device 201 according to this embodiment. The processing shown in Fig. 5 is started when, for example, a start operation is performed by the user.

[0023] In S500, the region information input terminal 403 acquires region information. In S501, the image input terminal 400 acquires images from the cameras 101 to 112. Here, the region information input terminal 403 acquires one frame of image and sets that image as the processing target, but it may also acquire multiple frames of image and set one of those frames that has not yet been set as the processing target.

[0024] In S502, the data generation unit 401 generates three-dimensional data of the subject in the image based on the acquired image, using a known algorithm such as Visual Hull.

[0025] In S503, the person information detection unit 402 detects the position of a person in the captured image, or the position or orientation of their face or eyes, as person information. The position and posture of a person may be expressed in a spatial coordinate system that defines a virtual space, or in a spatial coordinate system centered on each camera, and can be defined in any manner. Here, the position of the person's feet in the image is detected as the person's position. The position of the person's feet can be detected using a known method, such as converting the position of the person's feet into floor coordinates using projective transformation, taking into account the camera's imaging range. The "foot position" in this embodiment may be a position set based on the position of each foot, such as the center of gravity of both feet (for example, by statistical calculation processing), or may be a predetermined area including both feet (for example, a minimum area including both feet, or a perfect circle area surrounding both feet). Here, the person information detection unit 402 extracts the positions of both feet from the 3D data generated in S502, and calculates the center of gravity of these positions or the minimum area surrounding the feet to determine the position of the person. When the feet are off the floor, such as during jumping or running, the foot positions can be determined based on the projected positions of the feet onto the floor. If it is difficult to project from a two-dimensional image such as a captured image, the position of the person can be determined based on the three-dimensional data generated in S502.

[0026] In S504, the determination unit 404 determines whether or not a person is present in the determination area based on the position of the person and the area information described above. Taking the baseball stadium shown in FIG. 1 as an example, the area information is information indicating the range of the batter's box, which is the determination area, and in S504, it is determined whether or not a person is present in the batter's box. For example, in baseball, if the batter is not in the batter's box, it is considered that the batter is being replaced (i.e., the game is suspended). On the other hand, after the batter enters the batter's box, the play essentially begins from the state before the pitcher threw the ball, and the determination of whether or not a person has entered the batter's box can be considered to determine whether or not a scene is worth watching.

[0027] In S505, based on the determination result of S504, if a person is present in the determination area, the determination unit 404 determines that the person is suitable as the viewpoint position of the virtual viewpoint and proceeds to S506; if not, the determination unit 404 proceeds to S507. In S506, the virtual viewpoint setting unit 405 sets the virtual viewpoint at a position and direction corresponding to the person's line of sight determined by a method described below. The virtual viewpoint setting unit 405 may set the virtual viewpoint to the person's line of sight using the position or direction of the face or eyes detected in S503. Furthermore, if it is difficult to detect the line of sight, the direction of the face may be detected as the line of sight.

[0028] In S507, the virtual viewpoint setting unit 405 controls the virtual camera regardless of the position of the person being processed (assuming that the person being processed is not suitable as the viewpoint position of the virtual viewpoint). For example, the virtual viewpoint setting unit 405 can control the virtual camera so that the viewpoint of the virtual camera is a bird's-eye view from a position different from that of the person being processed, or a virtual third-party view that is not the viewpoint of any person. An example of a bird's-eye view is a view that overlooks the entire or partial area of ​​the competition venue from a distance. Examples of virtual third-party views include a virtual viewpoint from a predetermined position within the competition venue, a virtual viewpoint that follows a specific person from behind so that the entire person is captured, or a virtual viewpoint that follows a baseball and captures one or more people nearby. For example, by applying a virtual viewpoint from the spectator seats within the competition venue, it is possible to give viewers who are not at the competition venue a sense of realism as if they were watching from the spectator seats at the competition venue. Furthermore, various virtual viewpoints other than a person's line of sight may be applied, such as a virtual viewpoint determined by manual operation, and the virtual viewpoint may be set taking into consideration the rules of the competition, the situation at the stadium, or the viewer's orientation.

[0029] In S508, the image generation unit 406 generates a virtual viewpoint image from the above-mentioned three-dimensional data based on the set virtual viewpoint. In S509, the image output terminal 407 outputs the generated virtual viewpoint image to the display 202 or other device. In S510, the information processing device 201 determines whether or not to end the processing of Fig. 5, and if so, ends the processing, or if not, returns to S501. Here, it is assumed that the processing ends when an end instruction is given by the user.

[0030] Note that S503 and S504 are processes related to the position of the person. Here, if the image capture is performed at a frame rate that is high enough relative to the movement of the person (for example, if the frame rate is equal to or higher than a predetermined threshold value), S503 and S504 may be performed once every few frames rather than every frame (every loop of S501 to S509).

[0031] Next, an application example of the information processing device 201 according to this embodiment will be described with reference to FIG. 6. In FIGS. 6(a) to 6(d), the same objects are denoted by the same reference numerals. In FIG. 6, a baseball game is captured, and in FIG. 6(a), a pitcher 600, a catcher 601, a umpire 602, a batter 603, a right batter's box 604, and a left batter's box 605 are captured. In FIG. 6(a), the batter 603 is located within the right batter's box 604, so a human viewpoint is determined to be appropriate. In the situation in FIG. 6(a), the battle between the pitcher 600 and the batter 603 has already begun, creating a situation of high viewing value for viewers. In such a situation, by setting a virtual viewpoint at the line of sight of the pitcher 600, the catcher 601, or the batter 603, it is possible to provide a game video with high viewing value for viewers.

[0032] On the other hand, Figure 6(b) shows a situation in which batter 603 has struck out and is being replaced by batter 606. In Figure 6(b), there are no people within the range of batter's boxes 604 and 605. In this situation, neither pitcher 600, catcher 601, nor umpire 602 are in a play state, and the game is suspended. In a situation like Figure 6(b), pitcher 600, catcher 601, nor batters 603 and 606 are in a play state, so they may be looking at objects that are not of interest to the viewer, such as the ground, bat, ball, or audience seats. Furthermore, in this situation, if you try to set the virtual viewpoint from the batter's perspective, you may not know whether to set the virtual viewpoint to batter 603 or batter 606. Furthermore, a subject who is not looking at a specific object may move their gaze unpredictably, and sudden, unexpected gaze movements may cause visual sickness. Therefore, in a situation where a game is not in progress, such as in Figure 6(b), it is considered undesirable to set a virtual viewpoint at the line of sight of a person, and it is preferable to display an image from a virtual viewpoint other than that of the person in question. An example of a virtual viewpoint other than that of the person in question that is used here is a bird's-eye view virtual viewpoint, as shown in Figure 6(b), that allows one to see the situation during the transition to the next play.

[0033] Then, when the batter 606 enters the batter's box 605 from the situation in Figure 6(b) as shown in Figure 6(c), the game resumes, making it possible to provide a game video with high viewing value for viewers. When the situation in Figure 6(c) occurs, the virtual viewpoint is again set to the line of sight. Figure 6(d) is an example of an output image when the virtual viewpoint is set to the line of sight of the batter, and since the batter's line of sight is focused on the ball thrown by the pitcher 600, an image from the virtual viewpoint focusing on the pitcher 600 is displayed.

[0034] Note that when a person's line of sight is not appropriate, the virtual camera need not be controlled based on the person's position; it can be positioned at various other positions and postures. For example, a predetermined virtual viewpoint, such as a viewpoint overlooking the entire infield or a viewpoint looking at the ground from a specific seat in the stadium, may be set. Alternatively, the viewpoint may be set according to the ball's position. For example, as shown in FIG. 7, an image captured by the virtual camera may be displayed as an image showing the progress of the game, such as an image captured of a scoreboard or a graphics image showing the scoreboard. Furthermore, the image showing the progress of the game may include an advertising image. Because such images are not necessarily generated based on three-dimensional data or a virtual viewpoint, it is preferable to perform processing such as selecting a target image rather than performing the virtual viewpoint image generation processing of S508.

[0035] When viewing from a virtual eye-line viewpoint, a viewing method using VR goggles 800 as shown in Fig. 8 can be used as a means for viewing images with a sense of realism. In particular, when using VR goggles as a service within a competition venue and controlling a virtual camera with the information processing device 201, if a person's eye line is appropriate, an image is displayed on the VR goggles with the eye line of a specific athlete set as the virtual viewpoint.

[0036] On the other hand, as a display when a person's line of sight is not suitable, a see-through display (transparent display) may be provided using the image capture results from the imaging devices 801 and 802 that capture the front of the VR goggles. In particular, when VR goggles are used as a service within a competition venue and the information processing device 201 controls a virtual camera, as a display when a person's line of sight is not suitable, a see-through display (transparent display) may be provided using the image capture results from the imaging devices 801 and 802 that capture the front of the VR goggles. Here, see-through display refers to transparently superimposing a virtual viewpoint image on the image captured in front of the VR goggles. This makes it possible to provide a highly realistic virtual viewpoint image when a person's line of sight is suitable, while, when a person's line of sight is not suitable, it makes it possible for a viewer to experience the realism of being in the spectator seats at the competition venue without putting on or taking off the VR (Virtual Reality) goggles. Furthermore, when viewing from outside the competition venue using the VR goggles 800, the viewpoint from the spectator seats may be set as a virtual viewpoint instead of the see-through display, and an image that gives the viewer the feeling of being in the spectator seats at the competition venue may be displayed with a sense of realism.

[0037] Next, an example of a setting screen when the processing by the information processing device 201 according to this embodiment is applied to a baseball stadium will be described with reference to FIG. 9 . FIG. 9 is a diagram showing an example of a GUI displayed on the display 202 according to this embodiment for user settings. In FIG. 9 , a setting item 900 displays each item of a mode used by the user to determine a virtual viewpoint. Examples of modes used here include a preset viewpoint selection mode in which the position and orientation of the virtual viewpoint are registered in advance and the user selects one of the registered virtual viewpoints. Another example of a mode used here includes a position specification mode in which the user selects a pre-registered viewpoint position (e.g., a viewpoint from behind the player at each position) used when watching a sport where positions are fixed, such as baseball. Another example of a mode used here includes a free selection mode in which the user can freely move the viewpoint using a controller or the like. The “user” described here may be the viewer watching the virtual viewpoint video or a content creator, such as a broadcaster. Furthermore, when the virtual viewpoint is set to a position other than the person's line of sight in S507 described above, an image from the virtual viewpoint set using the setting item 900 may be displayed as the display content.

[0038] On the other hand, setting item 901 displays each item of a mode used to select a person's line of sight based on which the virtual viewpoint should be set when a person's line of sight is appropriate (the virtual viewpoint is set to the person's line of sight). Examples of modes used here include a team-specifying mode, a position-specifying mode, and a player-specifying mode. For example, when the team-specifying mode is selected, if the selected team is the offensive team in a baseball game, the virtual viewpoint is set to the line of sight of the batter (the player in the batter's box). If the selected team is the defensive team in a baseball game, the virtual viewpoint is set to the line of sight of either the pitcher or the catcher. This processing enables viewing from a virtual viewpoint focusing on a player on the selected team without the need to perform settings during viewing (e.g., every time the offense and defense change places). Furthermore, when the position-specifying mode is selected, the virtual viewpoint is set to the line of sight of a person playing a specific position. This processing makes it possible to provide a viewing experience that makes the viewer feel as if they are constantly experiencing the batter's position, or as if they are experiencing the speed of a pitch or the curve of a curveball from the catcher's line of sight. When a player-specifying mode is selected, the target person is identified by selecting information identifying the player, such as the team name and uniform number set by the user, and the virtual viewpoint is set to the target person's line of sight. This processing makes it possible to provide a viewing experience that makes the viewer feel as if they are the player, viewing from the perspective of their favorite player.

[0039] Note that the selection items shown in FIG. 9 are examples considered to be suitable for this embodiment, but the present invention is not limited to these, and other setting forms and setting contents may be used. For example, a virtual viewpoint capable of displaying at least a portion of the stadium and a display for displaying the virtual viewpoint may be provided, and when a position or range within the stadium is specified, the virtual viewpoint may be set to the line of sight of a person present within the specified range during the game. Furthermore, in ball games, for example, it may be expected that the viewing value of a video from the line of sight of a person identified based on the position or direction of the ball, such as holding the ball or being closest to the ball, is high. From this perspective, in addition to the above processing, a mode may be provided among the options in which the position or direction of the ball is detected and the line of sight of a person identified based on the position or direction of the ball (e.g., closest to the ball) is set as the virtual viewpoint.

[0040] Note that if the movement of a person whose line of sight is the virtual viewpoint is reflected in detail in the virtual viewpoint image, there is a possibility that visually induced motion sickness may occur. From this perspective, conditions may be imposed on the person used to set the virtual viewpoint, such as reducing the frequency of virtual viewpoint changes (for example, updating the virtual viewpoint every five frames), setting a time-averaged eye position and direction within a predetermined time, or not allowing the virtual viewpoint to follow rapid and large viewpoint changes (for example, the position of a person whose moving speed exceeds a predetermined threshold).

[0041] With this configuration, it is possible to determine whether at least one object is present in a predetermined area, and if it is determined that the object is present in the predetermined area, it is possible to control the virtual camera based on the position of the object. Therefore, in a scene in which a person whose viewpoint is assumed to have high viewing value is present, it is possible to display a virtual viewpoint image from the perspective of that person. Furthermore, in a scene in which a person whose viewpoint is assumed to have high viewing value is not present, a different image is displayed without using the virtual viewpoint image from the perspective of that person, thereby making it possible to display an image with high viewing value regardless of the scene.

[0042] [Embodiment 2] In the first embodiment, the description has been given assuming that a game at a baseball stadium is mainly captured. In particular, in the first embodiment, an example has been described in which a batter's box is used as the determination area, and when a person is in the batter's box, the virtual camera is controlled based on the position of the person. However, it is conceivable that the conditions for determining that the gaze direction of the person to be processed is appropriate will change depending on the content and progress of the game. Below, an example will be described in which the information processing device 201 is applied to a captured scene of a game different from that described in the first embodiment (particularly, in this case, a volleyball game scene is captured instead of a baseball game).

[0043] 10 is a block diagram showing an example of the functional configuration of an information processing device 201 according to embodiment 2. The information processing device 201 according to this embodiment has the same configuration and functions as that of embodiment 1, except that it additionally has a progress status input terminal 1000 and has a determination unit 1001 instead of the determination unit 404, so duplicated explanations will be omitted. The progress status input terminal 1000 acquires progress information indicating the progress of the game, such as the score obtained in the match or which team has the right to serve.

[0044] The determination unit 1001 determines whether or not a person is present in the determination area based on the person information and the information on the determination area, and also determines whether or not the person's line of sight is appropriate based on the progress information. Here, the determination unit 1001 can determine that the person's line of sight is appropriate when the person is present in the determination area and the progress status satisfies a predetermined condition.

[0045] Fig. 11 is a flowchart showing an example of information processing by the information processing device 201 according to this embodiment. The processing shown in Fig. 11 is performed in the same manner as the processing shown in Fig. 5 except that S1100 to S1102 are performed instead of S505, and therefore a duplicated description will be omitted.

[0046] In S1100, the progress status input terminal 1000 acquires progress information related to the competition. In S1101, the determination unit 1001 determines whether the progress status of the competition satisfies a predetermined condition. Here, the determination unit 1001 determines whether a change in the progress status has occurred. Note that, in the example of volleyball, a change in the progress status in this embodiment refers to the acquisition of a point or the transfer of the serving right (the team having the serving right has changed). Next, in S1102, the determination unit 1001 determines whether the person's line of sight is appropriate based on the determination result of S504 and the detection result of S1101. If it is determined to be appropriate here, the process proceeds to S506; if not, the process proceeds to S507.

[0047] Next, an application example of the information processing device 201 according to this embodiment will be described with reference to FIG. 12 . First, consider a scene with high viewing value in volleyball. In volleyball, a serve by one of the teams triggers the start of play. Next, if the ball falls, whether inside or outside the court, or if a foul occurs, such as a player touching the net, a point is scored or the serving right is transferred, and play is stopped. Here, similar to the pitcher's delivery in baseball, in volleyball, the scenes before the serve are considered to be of high viewing value to viewers. Furthermore, since volleyball rules stipulate that serves must be made from a predetermined area within the court, this predetermined area can be used as the determination area used in the determination performed in embodiment 1.

[0048] Figure 12 is an overhead view of a volleyball court. In Figure 12, a net 1200, sidelines 1201, and endlines 1202 are shown, and the area enclosed by these lines forms the court. In Figure 12, a service line 1203 is located on an extension of the sidelines 1201 and behind the endlines 1202. The area behind the endlines 1202 and between the service lines 1203 is a service zone 1204 in which a serve can be made. A condition for making a serve is that a person 1205 who will be making the serve must be present in the service zone 1204, and other people 1206 to 1215 must also be present on the court.

[0049] In this embodiment, the area information input terminal 403 acquires, as area information, information indicating that the service zone 1204 is the determination area. The determination unit 1001 first determines whether or not a person serving is in the service zone 1204. Here, since it is conceivable that the person serving 1205 will serve while people 1206-1215 other than the person serving 1205 are present on the court for both teams, an additional condition may be added that the other people 1206-1215 are present on each team's own side of the court, in addition to whether the person serving 1205 is present in the service area.

[0050] Furthermore, in the case of imaging a volleyball, when the serving right is transferred or a point is scored, play is suspended until the next serve. This is similar to the batter's change in baseball described with reference to FIG. 6 in the first embodiment. However, while a batter's change can be determined based on the presence or absence of a person in the batter's box, it is difficult to determine whether a pause in volleyball has occurred based solely on the presence or absence of a person within a predetermined range. Therefore, as described above, the determination unit 1001 according to this embodiment determines that a person's line of sight is appropriate when the person is present in the determination area (service zone 1204) and the progress satisfies a predetermined condition. To this end, the progress input terminal 1000 inputs progress information indicating the dynamic progress of the game, such as the transfer of the serving right or the scoring of a point. The determination unit 1001 then determines the start of a pause in play based on the progress information when it detects the transfer of the serving right or the scoring of a point. By making such a determination, it becomes possible to determine whether play has been suspended, not only based on whether a person is present in the determination area, but also based on whether the progress of the game satisfies a predetermined condition. Note that if it is determined that play has been suspended in this way, and thereafter it is detected that a person is present in the determination area (service zone 1204) and that the progress of the game satisfies a predetermined condition, as described above, the suspension of play is deemed to have been lifted, and the line of sight of that person is again set as the virtual viewpoint.

[0051] In volleyball, a serve can be made after the referee 1216 gives a gesture or whistle signal (a serve-enabled signal). Therefore, the determination unit 1001 may acquire serve-enabled information indicating that such a serve-enabled signal has been given as progress information, and determine that the person's line of sight is appropriate when the serve-enabled signal has been given. Here, the serve-enabled information may be, for example, a result of detecting whether or not the referee 1216 has given a signal (here, whether or not there is a gesture) by analyzing image or three-dimensional data, or a result of detecting whether or not there has been a whistle sound by analyzing audio. Such detection of a serve-enabled signal can be performed by commonly used image analysis processing or audio analysis processing, and a detailed description thereof will be omitted here.

[0052] According to this process, it is possible to determine whether or not to set the virtual viewpoint at the line of sight based on the gestures of the referee 1216, etc.

[0053] Furthermore, by determining whether or not a signal related to the progress of the game has been given, it is expected that it will be possible to determine whether or not a game is being played without making a determination using the position of a person. From this perspective, the determination unit 1001 according to this embodiment may determine whether or not a signal indicating that a player is ready to serve has been given, and if a signal indicating that a player is ready to serve has been given, the virtual camera may be controlled based on the position of the person.

[0054] In addition, since the condition for serving is that the person serving has the ball, it is possible to detect not only the person but also the ball, and to add the condition that "a person in the service zone has the ball" as an additional condition for determining that the person's line of sight is appropriate.

[0055] Although the example described here is one in which volleyball is imaged as a sport, the sport is not limited to this as long as a determination can be made using the conditions for the progression of the sport. For example, the technology for imaging volleyballs described above can be applied to sports such as tennis or table tennis, where play begins with a serve from a predetermined position. Note that, since the rules for the transfer of the serving right in tennis and table tennis differ from those in volleyball, the transfer of the serving right may not be used as the predetermined condition, and only whether or not a point has been scored may be used as the predetermined condition.

[0056] With this configuration, it is possible to acquire information indicating the progress of the competition, and determine whether or not to use the position of a person to control the virtual camera, depending on the progress as well as whether or not a person is present in the determination area. Therefore, in a scene where a person whose viewpoint is assumed to have high viewing value is present, it is possible to display a virtual viewpoint image from the perspective of that person. Furthermore, in a scene where a person whose viewpoint is assumed to have high viewing value is not present, a different image is displayed without using the virtual viewpoint image from the perspective of that person, making it possible to display an image with high viewing value regardless of the scene.

[0057] [Embodiment 3] In the second embodiment, an example was described in which whether or not to set the virtual viewpoint to the line of sight of a specific person is controlled based on progress information indicating the progress of the game in addition to the presence or absence of a person in the determination area. In the present embodiment, an example will be described in which the information processing device 201 is used in capturing an image of a basketball game.

[0058] 13 is a block diagram showing an example of the functional configuration of an information processing device 201 according to embodiment 3. The information processing device 201 according to this embodiment has the same configuration and functions as that of embodiment 1 except that it additionally has a competition time input terminal 1300 and has a determination unit 1301 instead of the determination unit 404, so duplicated explanations will be omitted. The competition time input terminal 1300 acquires information (competition time information) indicating whether the competition time during the competition is progressing or stopped.

[0059] The determination unit 1301 determines whether or not a person is present in the determination area based on the person information and the determination area information, and also determines whether or not the person's line of sight is appropriate based on the competition time information. Here, the determination unit 1301 can determine that the person's line of sight is appropriate when the person is present in the determination area and the competition time is progressing during the competition.

[0060] Fig. 14 is a flowchart showing an example of information processing by the information processing device 201 according to this embodiment. The processing shown in Fig. 11 is performed in the same manner as the processing shown in Fig. 5 except that S1400 to S1402 are performed instead of S505, and therefore a duplicated description will be omitted.

[0061] In S1400, the competition time input terminal 1300 acquires information indicating whether the competition time is progressing or stopped during the competition. In S1401, the determination unit 1301 determines whether the competition time is progressing or not. Next, in S1402, the determination unit 1301 determines whether the person's line of sight is appropriate based on the determination results of S504 and S1401. If it is determined to be appropriate here, the process proceeds to S506, and if not, the process proceeds to S507.

[0062] Next, with reference to FIG. 15 , an application example of the information processing device 201 according to this embodiment will be described. First, consider a scene with high viewing value in basketball. In a basketball game, there are four 10-minute periods called quarters, with a total time limit of 40 minutes. If the game is interrupted due to the ball going out of the court or a foul being committed, the game time stops. Therefore, in this case, the game time input terminal 1300 may acquire game time information, and the determination unit 1301 may make a determination based on the game time information. Here, the determination unit 1301 may determine that a person's line of sight is appropriate when the game is progressing, and may determine that a person's line of sight is not appropriate when the game is stopped.

[0063] Furthermore, in a captured scene of a basketball game, for example, when a person steps outside the court to take a throw-in, the presence or absence of a person in a specific area may be related to whether the scene is of high viewing value. From this perspective, the determination unit 1301 may determine whether the person's line of sight is appropriate based on whether a person is present in a predetermined area different from the determination area, in addition to whether the game is progressing.

[0064] FIG. 15(a) shows a basketball game in which a person 1509 is throwing in a ball 1519. FIG. 15 shows a center circle 1500, sidelines 1501, and end lines 1502, with the area enclosed by the sidelines 1501 and end lines 1502 forming the court. FIG. 15 also shows a goal ring 1503, a backboard 1504, and a three-point line 1505. A free throw line 1507 is a line in a paint area 1506 closer to the center circle 1500. A semicircle 1508 is drawn on the side of the free throw line 1507 closer to the center circle 1500. People 1509 to 1518 represent people participating in the game, and ball 1519 is the ball being used.

[0065] In FIG. 15 , a throw-in is a play in which person 1509, who is outside the court and holding ball 1519, throws ball 1519 to one of people 1510-1518 on the court. As with the volleyball serve described above, the tactics have already begun, such as searching for a place to pass the ball before throwing it in. Therefore, even though the game time is not progressing immediately before the ball is thrown in by the throw-in, it is considered to be a scene with high viewing value for viewers. Therefore, by determining that game time has stopped during the game and that person 1509 is present in a predetermined area (here, an area outside the court), the determination unit 1301 can determine that the person's line of sight is appropriate. Note that in addition to detecting person 1509, ball 1519 may also be detected, and the fact that person 1509 is in an area outside the court and that person 1509 is holding ball 1519 may be used as further conditions for the above-mentioned determination.

[0066] Furthermore, in basketball, the game progresses quickly, and a throw-in or other action may occur immediately after a pause. In such cases, the game is stopped for only a short period of time, so if the virtual viewpoint is switched in a short period of time, the viewer will end up viewing images with frequent changes in viewpoint. Therefore, if the game is stopped for a predetermined period of time or longer, the determination unit 1301 may determine that the eye-directed virtual viewpoint is not suitable.

[0067] Furthermore, in a basketball game, a free throw is an example of a scene that is expected to have high viewing value even when the game is stopped. During a free throw, a person participating in the game needs to be in a predetermined position. From this perspective, if it is determined that the game is stopped during the game and a person is present in such a predetermined position, it may be determined that a free throw is being taken, and the person's line of sight may be appropriate.

[0068] FIG. 15(b) shows the situation during a free throw, with each person satisfying the positional requirements for a free throw. In FIG. 15(b), the same elements as in FIG. 15(a) are assigned the same reference numerals, and redundant explanations will be omitted. In FIG. 15(b), a person 1509 carrying a ball 1519 is present alone in the area enclosed by the free throw line 1507 and semicircle 1508. Also, people 1513-1515, 1510, and 1511 are present outside the paint area 1506 on both sides of the sidelines. Other people 1512, 1513, 1517, and 1518 are present on the center circle 1500 side of the free throw line 1507 and three-point line 1505. When game time is stopped during a game and the person positional requirements for a free throw are satisfied as shown in FIG. 15(b), the determination unit 1301 may determine that the person's line of sight is appropriate.

[0069] Note that the above description of throw-ins and free throws is merely an example, and similar processing may be performed for other events during a game. For example, a jump ball at the start of a basketball game is also a scene that is expected to have high viewing value even though the game is stopped. Therefore, the predetermined area used for the determination here can be set arbitrarily depending on the content of such expected events. Furthermore, with regard to the positions of people, the above description has been given assuming that each processing is performed using predetermined positions within the court, but the determination unit 1301 may also make a determination based on the relative positions of people.

[0070] Furthermore, the game here is not limited to basketball, but can be applied to any time-limited sport, such as American football, where the game time stops when the game is interrupted. In soccer, the publicly displayed game time does not stop, but extra time is displayed as injury time. However, the referee has a clock to manage the game time, and when an interruption occurs, he stops the clock to stop the game time. Therefore, by providing a communication means, such as a wireless connection, to the clock that the referee keeps to manage the time, and by having the game time input terminal 1300 acquire such game time information, it becomes possible to execute each of the above-mentioned processes.

[0071] Furthermore, by determining whether or not competition time is progressing during a competition in this manner, it is expected that it will be possible to determine whether or not a game is in progress without making a determination using the position of a person. From this perspective, the determination unit 1301 according to this embodiment may determine whether or not competition time is progressing during a competition, and if so, control of the virtual camera may be performed based on the position of a person.

[0072] According to this processing, information indicating whether or not the competition time is progressing during the competition can be acquired, and in addition to whether or not a person is present in the determination area, it is also possible to determine whether or not to use the position of the person to control the virtual camera in accordance with this information. Therefore, in a scene in which a person whose viewpoint is assumed to have high viewing value is present, it is possible to display a virtual viewpoint image from the viewpoint of that person. Furthermore, in a scene in which a person whose viewpoint is assumed to have high viewing value is not present, a different image is displayed without using the virtual viewpoint image from the viewpoint of that person, thereby making it possible to display an image with high viewing value regardless of the scene.

[0073] The disclosure of this specification includes the following information processing device, information processing method, and program. (Item 1) An information processing device that controls a virtual camera corresponding to a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, a first determination means for determining whether at least one of the objects is present in a first predetermined area; a control means for controlling the virtual camera based on a position of the object when it is determined that at least one of the objects is present in the first predetermined area; An information processing device comprising: (Item 2) 2. The information processing device according to item 1, wherein the control means places the virtual camera at the position of the object. (Item 3) 3. The information processing device according to item 2, wherein the control means sets the imaging direction of the virtual camera to the direction of the face or the line of sight of the object. (Item 4) the object is a person participating in a competition; a first acquisition means for acquiring the progress of the competition; The method further comprises a second determination means for determining whether or not the progress of the competition satisfies a predetermined condition; The information processing device described in any one of items 1 to 3, characterized in that the control means controls the virtual camera based on the position of the object when it is determined that at least one of the objects is present in the first predetermined area and it is determined that the progress of the competition satisfies the predetermined condition. (Item 5) the object is a person participating in a competition; The game system further includes a second acquisition means for acquiring information indicating whether the game time is progressing or stopped during the game, The information processing device described in any one of items 1 to 3 is characterized in that, when it is determined that at least one of the objects is present in the first predetermined area, the control means changes whether or not to control the virtual camera based on the position of the object depending on whether competition time is progressing or stopped during the competition. (Item 6) Item 6. The information processing device according to item 5, wherein the control means controls the virtual camera based on the position of the object when it is determined that at least one of the objects is present in the first predetermined area and it is determined that competition time is progressing during the competition. (Item 7) a third determination means for determining whether or not at least one of the objects is present in a second predetermined area different from the first predetermined area; Item 6. The information processing device according to item 5, wherein the control means controls the virtual camera based on the position of the object when it is determined that at least one of the objects is present in the first predetermined area, it is determined that competition time is not progressing during the competition, and it is determined that at least one of the objects is present in the second predetermined area. (Item 8) Further, a designation means for designating one of the objects is provided, 8. The information processing device according to any one of items 1 to 7, wherein the control means controls the virtual camera based on the position of the object designated by the designation means. (Item 9) an output means for outputting the virtual viewpoint image to a VR goggle; a third acquisition means for acquiring a captured image of the front of the VR goggles; a display control means for, when it is not determined that at least one of the objects is present in the first predetermined area, causing the virtual viewpoint image to be transparently superimposed on the captured image of the front side in the VR goggles; 9. The information processing device according to any one of items 1 to 8, further comprising: (Item 10) An information processing device that controls a virtual camera corresponding to a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, a second acquisition means for acquiring information indicating whether the competition time is progressing or stopped during the competition in which the subject of the image is participating; a control means for controlling the virtual camera based on the position of the object when it is determined that competition time is progressing during the competition; An information processing device comprising: (Item 11) An information processing device that controls a virtual camera corresponding to a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, a first acquisition means for acquiring the progress of a competition in which the subject of the image capture is participating; a second determination means for determining whether or not the progress of the competition satisfies a predetermined condition; a control means for controlling the virtual camera based on the position of the object when it is determined that the progress of the competition satisfies a predetermined condition; An information processing device comprising: (Item 12) An information processing method by an information processing device that controls a virtual camera corresponding to a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, determining whether at least one of the objects is present in a first predetermined region; When it is determined that at least one of the objects is present in the first predetermined area, controlling the virtual camera based on the position of the object; An information processing method comprising: (Item 13) An information processing method by an information processing device that controls a virtual camera corresponding to a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, acquiring information indicating whether the competition time is progressing or stopped during the competition in which the imaged subject is participating; a step of controlling the virtual camera based on the position of the object when it is determined that competition time is progressing during the competition; An information processing method comprising: (Item 14) An information processing method by an information processing device that controls a virtual camera corresponding to a virtual viewpoint image based on a plurality of captured images captured by a plurality of imaging devices, acquiring the progress of a competition in which the subject of the image is participating; a step of determining whether the progress of the competition satisfies a predetermined condition; a step of controlling the virtual camera based on the position of the object when it is determined that the progress of the competition satisfies a predetermined condition; An information processing method comprising: (Item 15) A program for causing a computer to function as each of the means of the information processing device described in any one of items 1 to 11.

[0074] (Other Examples) The present invention can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0075] The invention is not limited to the above-described embodiments, and various changes and modifications can be made without departing from the spirit and scope of the invention. Accordingly, the following claims are appended to apprise the public of the scope of the invention. [Explanation of symbols]

[0076] 101: camera, 200: relay device, 201: information processing device, 202: display

Claims

1. An information processing device that controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, a first determination means for determining whether an object included in the virtual space exists in a first predetermined area; a control means for controlling a position of the virtual camera based on a position of the object when it is determined that the object is present in the first predetermined area; An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the control means places the virtual camera at the position of the object.

3. The information processing apparatus according to claim 2 , wherein the control means sets the imaging direction of the virtual camera to the direction of the face or the line of sight of the object.

4. the object is a person participating in a competition; a first acquisition means for acquiring the progress of the competition; The method further comprises a second determination means for determining whether or not the progress of the competition satisfies a predetermined condition; 2. The information processing device according to claim 1, wherein the control means controls the virtual camera based on the position of the object when it is determined that at least one of the objects is present in the first predetermined area and the progress of the competition satisfies the predetermined condition.

5. the object is a person participating in a competition; The game system further includes a second acquisition means for acquiring information indicating whether the game time is progressing or stopped during the game, 2. The information processing device according to claim 1, wherein the control means changes whether or not to control the virtual camera based on the position of the object, depending on whether competition time is progressing or stopped during the competition, when it is determined that at least one of the objects is present in the first predetermined area.

6. 6. The information processing device according to claim 5, wherein the control means controls the virtual camera based on the position of the object when it is determined that at least one of the objects is present in the first predetermined area and it is determined that competition time is progressing during the competition.

7. a third determination means for determining whether or not at least one of the objects is present in a second predetermined area different from the first predetermined area; 6. The information processing device according to claim 5, wherein the control means controls the virtual camera based on the position of the object when it is determined that at least one of the objects is present in the first predetermined area, it is determined that competition time is not progressing during the competition, and it is determined that at least one of the objects is present in the second predetermined area.

8. Further, a designation means for designating one of the objects is provided, 2. The information processing apparatus according to claim 1, wherein said control means controls said virtual camera based on the position of said object designated by said designation means.

9. an output means for outputting the virtual viewpoint image to a VR goggle; a third acquisition means for acquiring a captured image of the front of the VR goggles; a display control means for, when it is not determined that at least one of the objects is present in the first predetermined area, causing the virtual viewpoint image to be transparently superimposed on the captured image of the front side in the VR goggles; The information processing device according to claim 1 , further comprising:

10. An information processing device that controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, a second acquisition means for acquiring information indicating whether the competition time is progressing or stopped during the competition in which the subject of the image is participating; a control means for controlling the position of the virtual camera based on the position of the object when it is determined that competition time is progressing during the competition; An information processing device comprising:

11. An information processing device that controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, a first acquisition means for acquiring the progress of a competition in which the subject of the image capture is participating; a second determination means for determining whether or not the progress of the competition satisfies a predetermined condition; a control means for controlling the position of the virtual camera based on the position of the object when it is determined that the progress of the competition satisfies a predetermined condition; An information processing device comprising:

12. An information processing method by an information processing device that controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, comprising: determining whether an object included in the virtual space exists in a first predetermined area; When it is determined that the object is present in the first predetermined area, controlling the position of the virtual camera based on the position of the object; An information processing method comprising:

13. An information processing method by an information processing device that controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, comprising: acquiring information indicating whether the competition time is progressing or stopped during the competition in which the imaged subject is participating; a step of controlling the position of the virtual camera based on the position of the object when it is determined that competition time is progressing during the competition; An information processing method comprising:

14. An information processing method by an information processing device that controls a virtual camera corresponding to a virtual viewpoint image generated based on a plurality of captured images captured by a plurality of imaging devices, comprising: acquiring the progress of a competition in which the subject of the image is participating; a step of determining whether the progress of the competition satisfies a predetermined condition; a step of controlling the position of the virtual camera based on the position of the object when it is determined that the progress of the competition satisfies a predetermined condition; An information processing method comprising:

15. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 11.

Images