Information processing device, information processing method, and program

Abstract

To obtain information allowing appropriate assistance for a referee's determination.SOLUTION: An information processing apparatus generates three-dimensional shape data representing a three-dimensional shape of an object by using a plurality of images obtained by a plurality of imaging apparatuses picking up images of the object. The information processing apparatus estimates the size of the object by using the three-dimensional shape data. The information processing apparatus specifies the position of a specific object which is picked up by the plurality of imaging apparatuses based on a result of comparison between the estimated size of the object and a predetermined reference value.SELECTED DRAWING: Figure 3

Description

[Technical Field]

[0001] The present disclosure relates to an information processing device, an information processing method, and a program. [Background technology]

[0002] Recently, in order to confirm the accuracy of a referee's decision in a sports competition, a referee assistance system has been introduced that displays on a display device an image including a target play from images captured by multiple imaging devices installed around the stadium. Patent Document 1 describes a technology that calculates the movement speed and acceleration of each subject, such as a player and a ball, from two-dimensional images captured by multiple imaging devices in a soccer match, and determines whether or not the subjects have come into contact with each other based on the calculation results. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Application Laid-Open No. 2013-232181 Summary of the Invention [Problem to be solved by the invention]

[0004] For example, in basketball, there is a so-called 24-second rule, which states that a team must complete the shot within 24 seconds from when they get the ball to when they shoot. The standard for determining whether the ball has left the player's hand after 24 seconds has elapsed is whether the ball has left the player's hand, so an image capturing the moment the ball leaves the player's hand is required. For example, when occlusion occurs due to the relative positions of the players, it is difficult to identify the moment the ball leaves the player's hand, even with the technology of Patent Document 1.

[0005] The present disclosure has been made in consideration of the above-mentioned problems, and aims to provide a technology for obtaining information that can appropriately assist a referee in making a decision in a sports competition, for example. [Means for solving the problem]

[0006] An information processing device according to one aspect of the present disclosure includes a generation unit that generates three-dimensional shape data representing a three-dimensional shape of an object using a plurality of images obtained by a plurality of image capture devices capturing the object; an estimation unit that estimates a size of the object using the three-dimensional shape data; and an information processing device that estimates a size of the object captured by the plurality of image capture devices based on a comparison result between the size of the object estimated by the estimation unit and a predetermined reference value. Of the multiple objects, Specific Objects Whether or not the object is in contact with other objects different from the specific object in question. and a specifying means for specifying the [Effects of the Invention]

[0007] According to the present disclosure, information that can appropriately assist the judge in making a decision can be obtained. [Brief explanation of the drawings]

[0008] [Figure 1] Block diagram showing an example of the configuration of a virtual viewpoint image generation system [Figure 2] Schematic diagram showing an example of the placement of the imaging unit [Figure 3] Block diagram showing an example of the server's functional configuration [Figure 4] Schematic diagram showing example subjects [Figure 5] Diagram illustrating the generation of a 3D model of a basketball court [Figure 6] Diagram explaining the generation of a 3D model of a basketball [Figure 7] A diagram explaining the generation of a 3D model of a basketball player shooting. [Figure 8] Diagram explaining 3D model generation [Figure 9] A diagram explaining the presence or absence of voxels when modeling a subject [Figure 10] A flowchart showing the process of estimating the size of an object. [Figure 11] A flowchart showing the process of determining a basketball violation [Figure 12] Table showing examples of fouls and violations for each sport [Figure 13] Diagram showing an example of server hardware configuration DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, embodiments of the present disclosure will be described with reference to the drawings. Note that the following embodiments do not limit the scope of the present disclosure according to the claims, and not all of the combinations of features described in the embodiments are necessarily essential to the solutions of the present disclosure. Identical components are designated by the same reference numerals, and their description will be omitted.

[0010] [Embodiment 1] The virtual viewpoint image generation system according to this embodiment will be described with reference to the drawings. Fig. 1 is a diagram showing an example of the hardware configuration of the virtual viewpoint image generation system according to this embodiment. The virtual viewpoint image system according to this embodiment includes camera systems 110a-110j, a switching hub 120, a server 130, a database 140, a control device 150, an operation device 160, a display device 170, and a timer device 180.

[0011] Each camera system 110a-110j includes an imaging unit 111a-111j, which is composed of a lens, an imaging sensor, and the like, and a camera adapter 112a-112j, which controls the imaging unit and performs predetermined image processing in accordance with instructions from a control device 150. The imaging units 111a-111j are, for example, imaging devices such as cameras. The camera adapters 112a-112j include a processor (CPU or ASIC) and memory (RAM and ROM) required for control and image processing. The camera systems 110a-110j are connected to each other in a daisy chain configuration via network cables 113a-113i. Image data captured by the camera systems 110a-110j is transmitted via the network cables 113a-113i. A switching hub (hereinafter referred to as "HUB") 120 routes data transmission over the network. The HUB 120 and the camera system 110a are connected by a network cable 114a, and the HUB 120 and the camera system 110j are connected by a network cable 114b.

[0012] The server 130 processes image data from multiple viewpoints (multiple viewpoint frames) transmitted from the camera systems 110a-110j, and generates a three-dimensional model (hereinafter referred to as a "3D model") by estimating the shape from the image data from multiple viewpoints. The server 130 also generates a time synchronization signal and controls synchronization of the entire system. In other words, the server 130 can also be said to be an information processing device because it processes information. The HUB 120 and the server 130 are connected by a network cable 114c.

[0013] Database (hereinafter referred to as "DB") 140 stores image data and model information sent from server 130, and provides the stored image data and model information to server 130 as needed. Server 130 and DB 140 are connected by network cable 114d.

[0014] The control device 150 comprehensively controls each of the camera systems 110a-110j and the server 130. It also performs control for generating a virtual viewpoint video. The operation device 160 provides a user with a user interface unit (hereinafter referred to as a UI unit) through which the user operating this camera system controls the control device 150. The HUB 120 and the control device 150 are connected by a network cable 114e.

[0015] The display device 170 is a device that displays images, such as a liquid crystal display, and displays the image sent from the control device 150. The control device 150 and the display device 170 are connected by a video cable 114f.

[0016] The timer device 180 is a timer that generates time information when judging the rules of the game used in the game. That is, the timer device 180 measures a predetermined time. In this embodiment, the timer device 180 generates time information when judging the 24-second rule used in a basketball game. The timer device 180 is set to a predetermined time by a referee assistant (not shown) or is set automatically. The timer device 180 automatically counts down and outputs the count value (time information). That is, the timer device 180 measures time and outputs the measured time. The server 130 and the timer device 180 are connected by a network cable 114h.

[0017] 1, multiple camera systems are connected in a daisy chain, but a star connection in which each of the camera systems 110a-110j is directly connected to the HUB 120 may also be used. Also, the number of camera systems in the virtual viewpoint image generation system is not limited to 10, and may be more or less than 10.

[0018] The operation of the virtual viewpoint image generation system will now be described. An image captured by the imaging unit 111a undergoes image processing in the camera adapter 112a, such as separating the subject from the background, and is then transmitted to the camera adapter 112b of the camera system 110b via the network cable 113a. An image of the subject with the background separated is specifically called a silhouette image, and for example, the foreground is a white image and the background is a black image. The camera adapter 112 generates and transmits these silhouette, foreground, and background images. For example, the silhouette image can be generated using a common method, such as background subtraction, in which a difference between an image captured by capturing the subject and a background image captured in advance, such as before the start of a game, when the subject is not present, is calculated, and any difference above a threshold is defined as a silhouette (foreground region). Of course, the method for generating a silhouette image is not limited to this. For example, a silhouette image may be generated using a method such as human body recognition. Similarly, the camera system 110b transmits the image captured by the imaging unit 111b, together with the image acquired from the camera system 110a, to the camera system 110c. By continuing the above-described operation, images acquired by the imaging units of camera systems 110a to 110j are transmitted from camera system 110j to HUB 120 via network cable 114b, and then transmitted to server 130.

[0019] Next, the configuration and operation of the server 130 will be described. The server 130 processes data acquired from the camera system 110j. The server 130 has a time server function and transmits time and synchronization signals to each camera system 110 and the timer device 180. Upon receiving the time and synchronization signal, each of the ten camera systems 110, namely the imaging units 111a-111j, synchronizes the time and captures images frame by frame. The timer device 180 synchronizes the value of a 24-second timer displayed on its own display with the signal received from the server 130. Further details will be described later using the drawings.

[0020] (Arrangement of imaging unit) The arrangement of the imaging units of the camera system will be explained using the drawings. FIG. 2 is a diagram showing an example of the arrangement of the imaging units of the camera system. Note that FIG. 2 shows a basketball court (hereinafter referred to as "court") 201 as viewed from directly above. As shown in FIG. 2, imaging units 111a-111j of camera systems 110a-110j are arranged around court 201 so as to be able to capture images of desired positions on court 201. It is assumed that imaging units 111a-111j are each installed at a certain height from the ground. Imaging units 111a-111j are arranged so as to be able to capture images of the entire area (three-dimensional area including height) in which a subject moves on court 201.

[0021] (Server configuration) The configuration of the server 130 will be described with reference to the diagram. Fig. 3 is a diagram showing an example of the configuration of the server 130. As shown in Fig. 3, the server 130 has a time server 301, a camera information acquisition unit 302, a 3D model generation unit 303, a shape estimation unit 304, and a ball detection unit 305.

[0022] The time server 301 acquires time information for time synchronization in order to synchronize the images (videos) captured by the ten camera systems 110 shown in Fig. 1. The time server 301 receives Universal Time (UTC) from a GPS antenna (not shown), for example, or generates its own time within the time server 301 and transmits that time to each camera system 110. In this embodiment, the time server 301 also receives a counter value from the timer device 180 used in the basketball game, combines that value with the time for time synchronization, and sends the combined value to the camera information acquisition unit 302.

[0023] The camera information acquisition unit 302 acquires, from the ten imaging units 111a-111j, camera parameter information related to imaging, such as the current zoom value, focus value, and aperture value of each imaging unit, and image data obtained by imaging by each imaging unit. Here, the images obtained by imaging by each imaging unit 111a-111j are images obtained by time-synchronized imaging of the imaging units based on the time synchronization time transmitted from the time server 301.

[0024] The 3D model generation unit 303 generates a 3D model (three-dimensional shape data) of the subject using the camera parameter information acquired by the camera information acquisition unit 302 and images captured from multiple viewpoints. That is, the 3D model generation unit 303 generates a 3D model corresponding to each of multiple objects in a frame using frames from multiple viewpoints. For example, the 3D model is generated using a silhouette image of the subject generated by the camera adapter 112. The 3D model of the subject is generated using a silhouette image generated from images captured by all the imaging units and the camera parameter information. For example, the Visual Hull method can be used to generate the 3D model. As a result of this processing, 3D data (a set of points having three-dimensional coordinates) representing the 3D model of the subject is obtained.

[0025] The shape estimation unit 304 estimates the size of the 3D model using the camera parameter information acquired by the camera information acquisition unit 302 and the 3D model of the subject generated by the 3D model generation unit 303.

[0026] Ball detection unit 305 determines whether the 3D model is a basketball or not, using the size of the 3D model estimated by shape estimation unit 304 and preset information about the size of the basketball to be detected. Details of the determination method will be described later.

[0027] Here, we will use diagrams to explain the positions of a target player and the ball controlled by the target player, as well as one violation, the 24-second rule, using a basketball game as the imaging subject. Figure 4 is a schematic diagram of a basketball game in which a player takes a shot. Figure 4(a) shows the state in which the ball has not yet left the target player's hand just before the shot, and Figure 4(b) shows the state in which the ball has left the target player's hand just after the shot. Basketball's rules stipulate a violation, such as the 24-second rule. Under the 24-second rule, if a target player controls a live ball on the court and does not meet the following conditions, the violation is deemed a violation and the ball becomes the opponent's ball. That is, the target player's team must shoot within 24 seconds and the ball must either touch the rim or go into the basket before the opponent's ball becomes the opponent's ball. The important thing here is whether the ball leaves the player's hand before the 24-second timer reaches "0." Therefore, as shown in FIG. 4(a), if the timer device 180 counts down to "0" while the hand 412 of the focused player 411 is still in contact with the ball 413, it is a violation of the 24-second rule and the ball becomes the opponent's ball. On the other hand, in the situation shown in FIG. 4(b), even if the timer device 180 counts down to "0" the moment the ball 423 leaves the hand 422 of the focused player 421, it is not a violation of the 24-second rule if the following situation occurs. In other words, if the ball 423 touches the rim 424 or enters the basket 425 and the shot is successful, it is not a violation of the 24-second rule. In this way, the positional relationship between the count value of the timer device 180 and the focused player and the ball controlled by the focused player is important.

[0028] Here, the generation of a 3D model of a court will be explained using figures. Figures 5(a) and 5(b) are diagrams for explaining the generation of a 3D model of a court. As shown in Figure 5(a), the floor of the court is represented by the XY coordinate system of the three-dimensional coordinate system, and the height direction is represented by the Z coordinate system. A cube consisting of the court surface on the XY axis and a height Z1 represents the target space 510 in which the subject is modeled. Note that Figure 5(a) does not show the multiple camera systems arranged around the court. Figure 5(b) is a schematic diagram showing voxels 520. Image data captured by the multiple camera systems fills the target space 510 with voxels where the subject exists, while areas where the subject does not exist are empty. The voxel 520 is the smallest unit used to model a subject. As shown in Figure 5(b), a voxel 520 is a regular hexahedron per unit volume. In this embodiment, the voxel 520 is the smallest unit used to model a subject.

[0029] Here, the generation of a 3D model of a basketball will be explained using diagrams. FIG. 6 is a diagram for explaining the generation of a 3D model of a basketball. FIG. 6(a) shows a schematic diagram of a basketball, FIG. 6(b) shows a 3D model 620 of the basketball on the XY plane, and FIG. 6(c) shows a 3D model (voxel set) 630 of the basketball in XYZ space. FIGS. 6(b) and 6(c) are schematic diagrams showing a basketball in three-dimensional coordinates when modeling a subject in three-dimensional coordinates, and are views from different viewpoints. Note that FIGS. 6(b) and 6(c) show the basketball on the same coordinate axes.

[0030] As shown in FIG. 6(a), a basketball 610 has a diameter of M. As shown in FIG. 6(b), the basketball is voxelized, and the size of a 3D model 620 on the XY plane is L. However, the size M of the basketball 610 and the size L of the 3D model 620 of the basketball on the XY plane are assumed to satisfy the relationship L>>M. The size L of the 3D model 620 of the basketball on the XY plane is a size that includes an error when the size M of the actual basketball 610 is voxelized, and is assumed to be sufficiently larger than the size M of the basketball 610.

[0031] FIG. 7 is a schematic diagram of a player shooting a basketball game. FIGS. 7(a), 7(b), and 7(c) correspond to FIG. 4(a). FIGS. 7(d), 7(e), and 7(f) correspond to FIG. 4(b). FIG. 7(a) shows a silhouette image 710 from FIG. 4(a) mapped onto a three-dimensional coordinate system. FIG. 7(b) shows an image 720 from FIG. 7(a) that extracts the player's hands and the ball and maps it onto a three-dimensional coordinate system. FIG. 7(c) shows a 3D model (voxel set) 730 of image 720 from FIG. 7(b) mapped onto a three-dimensional coordinate system.

[0032] Similarly, Fig. 7(d) shows the silhouette image 740 of Fig. 4(b) mapped onto three-dimensional coordinates. Fig. 7(e) shows the image 750 corresponding to Fig. 7(d) and extracting the player's hands and ball, mapped onto three-dimensional coordinates. Fig. 7(f) shows the 3D model (voxel set) 760 of image 750 of Fig. 7(e) mapped onto three-dimensional coordinates. Note that the voxel sizes shown in Figs. 6 and 7 are larger than the actual sizes for ease of understanding.

[0033] Here, the three-dimensional coordinate positions of voxels when they are mapped onto a three-dimensional coordinate system will be explained using diagrams. Similar to FIG. 5(b), FIG. 8 is a schematic diagram of voxels mapped onto a three-dimensional coordinate system. Each of voxels V1, V2, and V3 is a regular hexahedron with each side having a length of "1," and the position of the voxel is represented by the three-dimensional coordinate of the vertex closest to the origin 0 among the eight vertices. According to this rule, the three-dimensional coordinates (x, y, z) of voxels V1, V2, and V3 can be (0, 0, 0), (2, 0, 0), and (2, 0, 1), respectively.

[0034] (Whether or not voxels are included when modeling the subject) The presence or absence of voxels when a subject is modeled will be explained using the drawings. Fig. 9 is a diagram for explaining the presence or absence of voxels when a subject is modeled. Fig. 9(a) shows a table summarizing information about voxels for each three-dimensional coordinate of the voxels explained in Fig. 8, which is information corresponding to one frame, and Fig. 9(b) shows the relationship between a voxel of interest and surrounding voxels.

[0035] As shown in FIG. 9(a), table 910 includes voxel presence / absence 912, surrounding voxel presence / absence 913, and object group 914 for all three-dimensional coordinates 911 in the space to be processed. Voxel presence / absence 912 indicates whether a voxel is present at the target coordinates, and stores "1" if a voxel is present at the target coordinates, and stores "0" if no voxel is present. Surrounding voxel presence / absence 913 indicates whether a voxel is present at each of 26 coordinates (SV1-SV26) surrounding the target coordinates. Surrounding voxel presence / absence 913 stores "1" if a voxel is present at each of 26 coordinates surrounding the target coordinates, and stores "0" if no voxel is present. Object group 914 indicates the group to which this voxel belongs if a voxel is present at the target coordinates. Table 910 is data generated by shape estimation unit 304 with reference to point cloud data generated by 3D model generation unit 303, and is stored in DB 140. Table 910 (information about voxels) stored in DB 140 is read by ball detection unit 305 and used when executing the ball detection process. Each voxel has 26 adjacent voxel positions in the three-dimensional direction around the voxel itself (voxel of interest). When the voxel itself is (0,0,0), the relative coordinates are (-1,-1,-1), (0,-1,-1), (1,-1,-1), (-1, 0, -1), (0, 0, -1), (1, 0, -1), (-1, 1, -1), (0, 1, -1), (1,1,0), (-1, -1, 0), (0, -1, 0), (1, -1, 0), (-1, 0, 0), (1, 0, 0), (-1, 1, -1), (0, 1, -1), (1,1,0), (-1, 1, 0), (0, 1, 0), (1, 1, 0), (-1, -1, 1), (0, -1, 1), (1, -1, 1), (-1, 0, 1), (0, 0, 1), (1, 0, 1), The relationship between SV1-SV26 and the surrounding voxels (also called adjacent voxels) is shown in the order of (-1, 1, 1), (0, 1, 1), and (1, 1, 1). However, the end points of the 3D coordinates are excluded. This is shown in Figure 9(b). In Figure 9(b), for ease of explanation, voxels are shown at a fixed distance from each other, with the voxel itself (the voxel of interest) shown in gray and the 26 surrounding voxels shown in white. For example, in Figure 8, voxel V1 is an end point of the 3D coordinates, so there are only seven voxel positions around it. However, considering the example shown in Figure 8, there are no surrounding voxels, so in the table in Figure 9(a), all the "Absence of Surrounding Voxels" columns are "0." Similarly, voxel V2 is located at the end point of the three-dimensional coordinate system, so the number of surrounding voxels is 11. Of these, only voxel V3 exists as a surrounding voxel, and it is assigned a value of "1"; the others are assigned a value of "0." The coordinates of the 26 surrounding voxels for each voxel in the table are combinations of x, y, and z coordinate values ​​of ±1 relative to the voxel itself, but we will not discuss them here. The same is true for voxel V3. Here, when each voxel exists as a surrounding voxel, it is considered to be part of the same subject, and the subject group is represented by the same symbol. Here, voxel V1 is group A, and voxels V2 and V3 are group B.

[0036] (Object size estimation) The process of estimating the size of an object, which is executed by the shape estimation unit 304, will be described using the drawings. FIG. 10 is a flowchart showing the flow of the process of estimating the size of an object. The series of processes shown in the flowchart in FIG. 10 are performed by the CPU by loading program code stored in ROM into RAM and executing it. In addition, some or all of the functions of the steps in FIG. 10 may be realized by hardware such as an ASIC or electronic circuit. Note that the symbol "S" in the description of each process indicates a step in the flowchart, and this also applies to subsequent flowcharts.

[0037] First, shooting is started with multiple cameras, and when the operations of the camera information acquisition unit 302 and the 3D model generation unit 303 are completed, the shape estimation unit 304 starts operation (S1001). That is, the shape estimation unit 304 acquires the camera parameter information acquired by the camera information acquisition unit 302 and the 3D model of the subject generated by the 3D model generation unit 303. The camera parameter information and the 3D model of the subject may be acquired from each functional unit or from DB 140.

[0038] In S1002, the shape estimation unit 304 determines whether or not there are any unprocessed voxels in the space to be processed, on a voxel-by-voxel basis. In the example of FIG. 5(a), the space to be processed is the volume of a cube consisting of the court surface on the X and Y axes and a value of "Z1" in the +z-axis direction, which is the height direction. The voxel unit is, for example, the voxel shown in FIG. 5(b). Based on the table of FIG. 9(a), it is determined whether or not there is a voxel at each position. Note that the search for the presence or absence of voxels is performed, for example, in the following order: First, a search is performed in the X-axis direction from the origin (0, 0, 0). Then, after completing the search at (Y, Z) = (0, 0), a search is performed in the X-axis direction from (Y, Z) = (1, 0). After completing the search at the end of the Y-axis in this order, a search is performed in the X-axis direction from (Y, Z) = (0, 1). Then, after completing the search for (Y, Z) = (0, 1), a search is performed in the X-axis direction from (Y, Z) = (1, 1). After completing the search at the end of the Y-axis in this order, a search is performed from the Z-axis coordinates incremented by 1. The entire space to be processed is searched in this order. The order in which the search for the presence or absence of voxels is not limited to this. If the determination result indicates that a voxel exists (YES in S1002), the process proceeds to S1003. On the other hand, if the determination result indicates that a voxel does not exist (NO in S1002), the process proceeds to S1013, and the flow shown in FIG. 10 ends.

[0039] In S1003, the shape estimation unit 304 identifies a voxel of interest to be processed.

[0040] In S1004, the shape estimation unit 304 determines whether or not there are voxels (surrounding voxels) at 26 positions surrounding the voxel of interest identified in S1003. However, depending on the position of the voxel of interest, there may not be 26 voxels. If the determination result indicates that there are no voxels at all at 26 positions surrounding the voxel of interest (NO in S1004), the process proceeds to S1006. If the determination result indicates that there is at least one voxel at all at 26 positions surrounding the voxel of interest (YES in S1004), the process proceeds to S1005.

[0041] In S1005, the shape estimation unit 304 sets surrounding voxels (adjacent voxels) at the position determined to have a voxel in S1004 to the same group as the voxel of interest.

[0042] In S1006, the shape estimation unit 304 determines whether or not the presence or absence of surrounding voxels has been confirmed for all voxels detected from the space to be processed. If the determination result indicates that the presence or absence of surrounding voxels has been confirmed for all voxels (YES in S1006), the process proceeds to S1007. On the other hand, if the determination result indicates that the presence or absence of surrounding voxels has not been confirmed for all voxels (NO in S1006), the process returns to S1003. Then, in S1003, a voxel of interest is identified from among the unprocessed voxels, and a series of processes from S1004 to S1006 is executed.

[0043] In S1007, the shape estimation unit 304 derives the size of the voxel set for each group. Within the voxel set, which is a cluster of voxels set to the same group in S1005, the maximum length of each axis is calculated from the coordinate points at the extreme ends of the ±X, ±Y, and ±Z directions. Here, the X-axis direction will be used as an example. For a certain group, if the coordinate point (3, 0, 1) has the smallest x value and the coordinate point (10, 4, 3) has the largest x value, the length in the x direction is 10-3 = 7, or "7." Similarly, the lengths in the y-axis and z-axis directions can be calculated to derive the size of the group as a cube.

[0044] In S1008, the shape estimation unit 304 identifies a group of interest to be processed.

[0045] In S1009, the shape estimation unit 304 determines whether the size of the target group is smaller than the size L of the object (target object) acquired in advance. If the result of the determination is that the size of the target group is smaller than the size L of the object (target object) (YES in S1009), the process proceeds to S1010. On the other hand, if the result of the determination is that the size of the target group is not smaller than the size L of the object (target object) (NO in S1009), the process proceeds to S1011. Specifically, it is confirmed whether the size of the voxel set corresponding to the target group is smaller than the size "L" of the basketball represented by voxels in FIG. 6(b). To confirm the size of the basketball, the maximum length of each axis is calculated from the coordinate points of the three-dimensional coordinates at the extreme ends of the ±X, ±Y, and ±Z directions within the group of voxel blocks set in the same group in S1005. If there is an object smaller than "L," that is, smaller than L, it means that the object is smaller in all x, y, and z directions.

[0046] In S1010, the shape estimation unit 304 estimates the shape of the target group as the shape of the object.

[0047] In S1011, the shape estimation unit 304 stores position information indicating the position of the group of interest estimated as an object (a basketball in this embodiment) in S1010. The shape estimation unit 304 also stores time information, linked to the frame to be processed and including the time generated by the server 130 and the count value counted by the timer device 180, in the DB 140 or the like. That is, the position information of the object of interest, the count value, and the game time of the imaged object are associated and stored in the DB 140 or the like. The position information may be, for example, coordinate information of any voxel in the voxel set corresponding to the group of interest, but as described in FIG. 8, the position information is the voxel closest to the origin of the three-dimensional coordinate system, and is the coordinate information of the vertex of that voxel closest to the origin. The position information is not limited to this, and the position of the object may be, for example, the center of gravity of the positions of each voxel.

[0048] In S1012, the shape estimation unit 304 determines whether or not the object sizes have been confirmed for all groups. If the determination result indicates that the object sizes have not been confirmed for all groups (NO in S1012), the process returns to S1008. Then, in S1008, a group of interest is identified from among the unprocessed groups, and a series of processes from S1009 to S1012 is executed. On the other hand, if the determination result indicates that the object sizes have been confirmed for all groups (YES in S1012), the process proceeds to S1013, and the flow shown in FIG. 10 ends.

[0049] By comparing the size of the group (voxel set) with the size of the object obtained in advance for each frame in the flow shown in FIG. 10 described above, the position of the basketball at each time and whether or not the basketball is being held by a player can be determined. From this, it can be said that the shape estimation unit 304 can also identify contact between specific objects. In other words, the position of the basketball can be identified in each frame. In other words, based on the comparison result between the size of the object estimated by the shape estimation unit 304 and a predetermined reference value, the position of a specific object captured by multiple image capture devices can be identified.

[0050] Returning to the explanation of Figure 3, the violation determination unit 306 determines whether or not a foul has been committed using the relative positions of the player and the basketball and the count value of the timer device 180, and sends the determination result to the display device 170 via the HUB 115 and the control device 118. The display device 170 displays the determination result that has been sent. A case where the violation is based on the 24-second rule will be explained using the diagram.

[0051] (Processing performed by the server) The processing executed by the server 130 will be described with reference to the drawings. Fig. 11 is a flowchart showing the flow of the processing executed by the server 130. Here, a case where the 24-second rule of basketball is applied as a violation will be described.

[0052] First, the flow shown in FIG. 11 starts when one of the teams in the basketball game to be processed starts an attack (S1101).

[0053] In S1102, when any team starts an offense in a basketball game, the counter value of timer device 180 is set to 24 seconds. The 24-second timer in timer device 180 is manually set by, for example, a referee assistant. When the counter value is set in this way, timer device 180 starts counting.

[0054] In S1103, the camera information acquisition unit 302 acquires a silhouette image from the camera system 110j.

[0055] In S1104, the 3D model generation unit 303 estimates the shape of the subject. Specifically, the 3D model generation unit 303 generates a 3D model of the subject using the silhouette image acquired by the camera information acquisition unit 302.

[0056] In S1105, the shape estimation unit 304 estimates the size of the subject. Specifically, the shape estimation unit 304 determines the size from the 3D model (voxel set) of the subject generated in S1104. The determination method is as follows. First, the presence or absence of voxels and the presence or absence of surrounding voxels is confirmed at all coordinate positions of the cube shown in FIG. 5(a). The confirmation method is performed by focusing on a gray voxel in FIG. 9(b) and checking the presence or absence of 26 surrounding white voxels. This is represented by table 910 in FIG. 9(a). Next, a group is identified for each voxel, and the size of the voxel set (subject) is determined for each group. The detailed processing of S1105 is the same as S1007 described in the flowchart of FIG. 10 above.

[0057] In S1106, the shape estimation unit 304 checks whether there is an object smaller than the size "L" of the basketball represented in voxels shown in Fig. 6(b). The detailed processing of S1106 is the same as that of S1009 and S1010 described in the flowchart of Fig. 10.

[0058] In S1107, it is determined whether a basketball has been detected. Specifically, if there is an object smaller than "L" (YES in S1107), the ball detection unit 305 determines that the object is a basketball, and the process proceeds to S1110. "Smaller than L" means that the object is small in all of the x, y, and z directions. On the other hand, if there is no object smaller than "L" (NO in S1107), the process proceeds to S1108. In other words, since consecutive frames are processed in order, the detection of a basketball indicates that the basketball has left the player.

[0059] In S1108, the violation determination unit 306 acquires the counter value of the timer device 180.

[0060] In S1109, the violation determination unit 306 determines whether the counter value acquired in S1108 is not "0." In other words, the violation determination unit 306 determines whether a predetermined time has elapsed. If the determination result obtained in S1108 is that the counter value is not "0" (YES in S1109), the process returns to S1103, and the series of processes from S1103 onwards are executed for the next frame. On the other hand, if the determination result obtained in S1108 is that the counter value is "0" (NO in S1109), the process proceeds to S1115.

[0061] In S1110, the violation determination unit 306 determines whether the basketball and the offensive player have just separated, for example, immediately after the offensive player has shot, based on the time before and after the generation of the 3D model. Specifically, the violation determination unit 306 determines whether the basketball and the player have just separated, based on the results of the ball detection process performed on the data associated with the frame to be processed (the frame of interest) and the data associated with the previous frame. If the ball is not detected in the process performed on the data of the previous frame, but is detected in the process performed on the data of the frame of interest, and the determination is made that the basketball and the player have just separated (YES in S1110), the process proceeds to S1112. Then, in S1112, the violation determination unit 306 obtains the counter value of the timer device 180 and records it in DB 140. On the other hand, if the basketball is detected in both the process performed on the data of the previous frame and the process performed on the data of the frame of interest, and the determination is made that the basketball and the player have not just separated (NO in S1110), the process proceeds to S1111.

[0062] In S1111, the violation determination unit 306 determines whether the basketball thrown by the player has touched or scored a goal with the goal ring. Whether the basketball has touched or scored a goal can be determined by checking the position information of each and the 3D model. Because the goal is located at known coordinates on the xyz coordinate system, whether a voxel the size of a basketball has touched or scored a goal with the goal can be easily determined by determining the ball's position coordinates, for example, from Table 910 shown in FIG. 9(a). Specifically, using the position information of the basketball used in the determination in S1106 and the position information of the goal ring previously stored in DB 117, if the two pieces of position information contain adjacent position information, it is determined that the basketball has touched or scored a goal. Furthermore, if the area enclosed by the position information of the goal ring intersects with the area enclosed by the position information of the basketball, it is determined that the basketball has scored a goal. If these conditions are not met, it is determined that the basketball did not touch the goal ring and did not score a goal.

[0063] In S1113, violation determination unit 306 determines whether or not the basketball has come into contact with the goal ring. If the determination result indicates that the basketball has come into contact with the goal ring (YES in S1113), the process proceeds to S1114. On the other hand, if the determination result indicates that the basketball has not come into contact with the goal ring (NO in S1113), the process proceeds to S1108. In this process, it is determined whether or not the player will throw (shoot or pass) the basketball and it will come into contact with the goal ring within a certain period of time.

[0064] In S1114, violation determination unit 306 determines whether the counter value of timer device 180 was "0" when the basketball contacted the goal ring. If the determination result indicates that the counter value of timer device 180 was "0" (YES in S1114), the process proceeds to S1115. On the other hand, if the determination result indicates that the counter value of timer device 180 was not "0" (NO in S1114), the process proceeds to S1116.

[0065] In S1115, the violation determination unit 306 records that a violation has occurred and notifies the referee of the violation. The method of notifying the referee may be a notification using a device capable of displaying an image or message such as a user interface image, or a notification using an alarm device (not shown) that issues an alarm. After the notification is completed, the process proceeds to S1117.

[0066] In S1116, the violation determining unit 306 resets the counter value. That is, the violation determining unit 306 determines that no violations that fall under the 24-second rule have occurred in the series of plays.

[0067] In S1117, the violation determination unit 306 ends the flow shown in FIG.

[0068] In S1111, the violation determination unit 306 can determine the xyz coordinate position of the basketball from Fig. 9(a), but this is not limiting. For example, using information such as a voxel set, the subject may be identified from the size of the subject in a voxel cluster (voxel set) as long as the subject has a specific size, not limited to a basketball, and its position may be determined.

[0069] As described above, the system for generating virtual viewpoint images using multiple cameras arranged around a subject can achieve the following effects. That is, in the process (stage) before generating the virtual viewpoint image, by correlating the size of the subject's 3D model at the time of generating the model with the duration of the game, it is possible to obtain an image that can appropriately assist the referee in making a decision. This makes it possible to judge violations of the basketball game rules.

[0070] Furthermore, according to this embodiment, by sequentially measuring the size of each subject model generated when generating a virtual viewpoint image from image data of the subject captured by multiple imaging units, the device can be used as a referee assistance device to assist the referee in the competition.

[0071] The application of the system of this embodiment is not limited to the 24-second rule in basketball. The system of this embodiment can also be applied to other violations in basketball, such as three-second overtime. The system of this embodiment can also be used for judgment processing in sports other than basketball, such as offside in soccer. Furthermore, it is also possible to track the trajectory of the ball in time series.

[0072] Fouls and violations for each sport to which the system of this embodiment can be applied will be described using the drawings. FIG. 12 is a table listing fouls and violations for each sport. As shown in FIG. 12, table 1200 contains information on sport 1201 to be imaged and fouls and violations 1202 to be imaged. In table 1200, sports such as basketball, soccer, and handball are stored for sport 1201.

[0073] Regarding fouls and violations 1202, in basketball, information such as the "24-second rule," "8-second rule," "5-second rule," "3-second rule," "traveling," and "double dribble" is stored. That is, when the timer device 180 times 8 seconds after one team starts their attack and the ball detected by the ball detection unit 305 is not in a predetermined area of ​​the court, information indicating a violation of the basketball rules is stored. When the timer device 180 times 5 seconds after the ball is handed over from the umpire to a player and the ball detected by the ball detection unit 305 is in contact with the player, information indicating a violation of the basketball rules is stored.

[0074] In soccer, information such as "offside" and "handball" is stored.

[0075] For handball, information such as "7m throw," "overtime," "overstep," "double dribble," etc. is stored. When the timer device 180 times that three seconds have passed since the ball detected by the ball detection unit 305 came into contact with a handball player, information indicating that the ball is in violation of the handball rules is stored if the ball is in contact with the player.

[0076] By defining a flow that corresponds to each rule in advance and storing it in the system, for example, in DB 140, it becomes possible to efficiently assist referees in accordance with each sport. That is, as shown in Fig. 12, data that associates predetermined rules that use position information and time information for referee decisions for each sport such as basketball or handball may be stored in advance in DB 140 in the form of a table or the like.

[0077] In this embodiment, a case where an event occurring in a basketball game is interpreted in more detail has been described, but the present invention is not limited to this. The present invention may also be applied to a case where an event occurring in other ball games, such as soccer or tennis, is interpreted in more detail. The present invention may also be applied to a case where an event occurring in a sport that does not involve a ball, such as archery or kyudo, is interpreted in more detail. When applied to the interpretation of an event occurring in a sport that does not involve a ball, the ball detection unit 305 can detect the object by determining its size and elongated shape.

[0078] Furthermore, the judgment is not limited to judgments that exceed the time limit. For example, when it is determined that a handball has been touched by a hand, information indicating that the ball has made contact may be displayed on the display device 170 or the like to present to the referee. Furthermore, a virtual viewpoint image may be generated by the control device 150 based on the corresponding frame, and the generated virtual viewpoint image may be displayed on the display device 170 or the like to present to the referee. This can assist the referee's judgment. Furthermore, when assisting the referee's judgment, it is also possible to set the spatial position that is thought to be the target, such as contact, in the center of the image, making it easier for the referee to see the area that is the target of the judgment.

[0079] It can also be applied to outputting information about the speed of a player dribbling in sports such as soccer, basketball, and handball. When the ball detection unit 305 detects the ball in a positional relationship with the same player for a predetermined period of time after timing by the timer device 180, it is also possible to output information indicating the speed of the player dribbling. In other words, it is also possible to output the speed of the player dribbling derived using information generated based on two frames immediately after contact between the ball and the player within a certain period of time, where the frame interval is equal to or greater than a predetermined value.

[0080] [Other embodiments] The present disclosure can also be realized by providing a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions.

[0081] Although the embodiments of the present disclosure have been described in detail above, the present disclosure is not limited to the above-described embodiments, and various modifications and changes are possible within the scope of the gist of the present disclosure as set forth in the claims. Fig. 13 is a block diagram showing an example of the configuration of computer hardware applicable to devices such as the server 130 included in the system of the above-described embodiment.

[0082] The CPU 1301 controls the entire computer using computer programs and data stored in the RAM 1302 and the ROM 1303, and also executes the processes described above as being performed by devices such as the server 130 included in the system of the above embodiment. That is, the CPU 1301 functions as each processing unit shown in FIG.

[0083] The RAM 1302 has an area for temporarily storing computer programs and data loaded from an external storage device 1306, data acquired from the outside via an I / F (interface) 1307, etc. The RAM 1302 also has a work area used when the CPU 1301 executes various processes. That is, the RAM 1302 can be allocated as a frame memory, for example, or can provide various other areas as needed.

[0084] The ROM 1303 stores setting data for the computer, a boot program, etc. The operation unit 1304 has a keyboard, a mouse, etc., and can be operated by a user of the computer to input various instructions to the CPU 1301. The output unit 1305 has, for example, a liquid crystal display, and displays the results of processing by the CPU 1301. The operation unit 1304 and the output unit 1305 are not necessarily required, and data may be input / output to / from a connected external device via the I / F 1307.

[0085] The external storage device 1306 is a large-capacity information storage device such as a hard disk drive. The external storage device 1306 stores an OS (operating system) and computer programs for causing the CPU 1301 to realize the functions of each processing unit shown in Fig. 3. Furthermore, the external storage device 1306 may also store image data to be processed.

[0086] Computer programs and data stored in the external storage device 1306 are loaded into the RAM 1302 as appropriate under the control of the CPU 1301, and become the subject of processing by the CPU 1301. The I / F 1307 can be connected to networks such as a LAN or the Internet, and other devices such as a projector or display device, and the computer can obtain and send various information via this I / F 1307. The bus 1308 connects each part of the device such as the server 130 to transmit information.

[0087] The operation of the above-described configuration is controlled mainly by the CPU 1301 as explained in the above-described embodiment.

[0088] The object of the present disclosure can also be achieved by providing a storage medium containing computer program code for implementing the above-described functions to a system, and having the system read and execute the computer program code. In this case, the computer program code itself read from the storage medium implements the functions of the above-described embodiments, and the storage medium containing the computer program code constitutes the present disclosure. This also includes cases where an operating system (OS) running on a computer performs some or all of the actual processing based on the instructions of the program code, thereby implementing the above-described functions.

[0089] Furthermore, the present invention may be realized in the following form: That is, computer program code read from a storage medium is written to memory in a function expansion card inserted into a computer or in a function expansion unit connected to the computer, and the CPU in the function expansion card or function expansion unit then performs some or all of the actual processing based on the instructions of the computer program code, thereby realizing the above-mentioned functions.

[0090] When the present disclosure is applied to the storage medium, the storage medium stores computer program code corresponding to the processes described above. [Explanation of symbols]

[0091] 303 3D model generation unit 304 Shape estimation part 305 Ball detection unit

Claims

1. a generating means for generating three-dimensional shape data representing a three-dimensional shape of an object using a plurality of images obtained by capturing images of the object using a plurality of imaging devices; an estimation means for estimating a size of the object using the three-dimensional shape data; an identification means for identifying whether or not a specific object among the plurality of objects captured by the plurality of imaging devices is in contact with another object different from the specific object, based on a comparison result between the size of the object estimated by the estimation means and a predetermined reference value; An information processing device comprising:

2. an output means for outputting information for notifying a contact between the specific object identified by the identification means and the other object; 2. The information processing apparatus according to claim 1, further comprising:

3. The specific object and the other objects are objects in a ball game captured by the plurality of imaging devices.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. 4. The information processing apparatus according to claim 1, wherein the specifying means specifies the position of the specific object when it is specified that the specific object and the other object are not in contact with each other.

5. An acquisition means for acquiring time information relating to time; an information output means for outputting predetermined information relating to the specific object based on position information representing the position of the specific object and the time information; 5. The information processing apparatus according to claim 4, further comprising:

6. the location information, the time information, and a local time of an object imaged by the imaging device are associated with each other; The information output means outputs predetermined information based on the location information corresponding to the time indicated by the time information and the local time.

6. The information processing apparatus according to claim 5,

7. the imaging target is a basketball game, the particular object is a basketball, further comprising a detection means for detecting the basketball; The information output means outputs information about the basketball.

7. The information processing apparatus according to claim 6,

8. further comprising a determination means for determining whether or not there is a violation of the rules of the basketball game based on at least one of the identification result regarding the contact between the basketball, which is the specific object, and the other object, and the position information of the basketball, which is the specific object, information about the local time, and information about the rules of the basketball game; When it is determined that the violation has occurred, the information output means outputs information indicating that the violation has occurred as information about the basketball.

8. The information processing apparatus according to claim 7,

9. The determination means determines, based on the information about the local time, that the basketball detected by the detection means is in contact with an offensive player or is not in contact with a goal ring at a time when a predetermined time has elapsed since one team started attacking in the basketball game.

9. The information processing apparatus according to claim 8,

10. The determination means determines, based on the information about the local time, that the basketball detected by the detection means is not in a predetermined area of ​​the basketball court at a time when a predetermined time has elapsed since one team started their attack in the basketball game, that a violation of the rules of the game has occurred.

10. The information processing device according to claim 8, wherein the information processing device is a computer.

11. The determining means determines, based on the information about the local time, that the basketball detected by the detecting means is in contact with a player at a time when a predetermined time has elapsed since the basketball was handed over from the referee to the player in the basketball game, that a violation of the rules of the game of the basketball game has occurred.

11. The information processing device according to claim 8, wherein the information processing device is a computer.

12. the imaging target is a soccer ball game, the specific object is a soccer ball, further comprising a detection means for detecting the soccer ball; The information output means outputs information about the soccer ball.

7. The information processing apparatus according to claim 6,

13. a determination means for determining whether or not there is a violation of a game rule of the soccer ball game based on at least one of the determination result regarding the contact between the specific object, the soccer ball, and the other object, and the position information of the specific object, the soccer ball, and information regarding the local time; When it is determined that the violation has occurred, the information output means outputs information indicating that the violation has occurred as information about the soccer ball.

13. The information processing apparatus according to claim 12.

14. the imaging target is a handball match, the specific object is a handball, further comprising a detection means for detecting the handball; The information output means outputs information about handball.

7. The information processing apparatus according to claim 6,

15. further comprising a determination means for determining whether or not there is a violation of the rules of the handball match based on at least one of the identification result regarding the contact between the handball as the specific object and the other object and the position information of the handball as the specific object, information regarding the local time, and When it is determined that a violation has occurred, the information output means outputs information indicating that a violation of the rules of the handball game has occurred.

15. The information processing apparatus according to claim 14,

16. The determining means determines, based on the information about the local time, that a violation of the rules of the handball match has occurred when the handball detected by the detecting means is in contact with a player in the handball match at a time when a predetermined time has elapsed since the handball made contact with the player.

16. The information processing apparatus according to claim 15,

17. The determining means determines whether or not there is a violation of a competition rule by using a table that records the competition rule.

17. The information processing device according to claim 8, 9, 10, 11, 13, 15, or 16.

18. the generating means generates the three-dimensional shape data corresponding to each of a plurality of objects in a frame using frames from a plurality of viewpoints obtained by capturing images from a plurality of directions; the estimation means estimates sizes of a plurality of objects in the frame using the three-dimensional shape data; The specifying means specifies whether or not the specific object is in contact with the other object by comparing the size of the object estimated by the estimating means with the predetermined reference value for each frame.

18. The information processing device according to claim 1, wherein the information processing device is a computer.

19. the three-dimensional shape data is data expressed in voxels, The estimation means estimates the size of the object based on the number of voxels in three-dimensional coordinates.

19. The information processing device according to claim 1, wherein the information processing device is a computer.

20. a generation step of generating three-dimensional shape data representing a three-dimensional shape of the object using a plurality of images obtained by capturing images of the object using a plurality of imaging devices; an estimation step of estimating a size of the object using the three-dimensional shape data; a specifying step of specifying whether or not a specific object among the plurality of objects captured by the plurality of image capturing devices is in contact with another object different from the specific object, based on a comparison result between the size of the object estimated in the estimating step and a predetermined reference value; An information processing method comprising:

21. A program for causing a computer to function as the information processing device according to any one of claims 1 to 19.

Images