Information processing device, information processing method, and program

Abstract

To properly identify a plurality of objects.SOLUTION: An information processing device is configured to: identify each of a plurality of objects on the basis of a first type of feature until a distance between the plurality of objects falls below a threshold; and identify each of the plurality of objects on the basis of a second type of feature when the distance between the plurality of objects is no longer below the threshold after having fallen below the threshold.SELECTED DRAWING: Figure 1

Description

[Technical Field]

[0001] The present disclosure relates to generating data based on captured images. [Background technology]

[0002] There is a method for generating three-dimensional shape data (hereinafter sometimes referred to as a three-dimensional model) representing the three-dimensional shape of an object based on a plurality of captured images obtained by a plurality of imaging devices arranged around the object. There is also a method for generating a virtual viewpoint image, which is an image from an arbitrary viewpoint, using texture information obtained from the captured images and the three-dimensional model. It is also sometimes required to manage which object is in the virtual viewpoint image.

[0003] Patent Document 1 discloses a method for identifying multiple objects in a three-dimensional space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] International Publication No. 2019 / 021375 Summary of the Invention [Problem to be solved by the invention]

[0005] Patent Document 1 describes identifying multiple objects in a three-dimensional space using the color characteristics of the objects, their uniform numbers, or signals emitted from sensors attached to the objects. However, extracting color characteristics or uniform numbers from an image requires image processing for extraction, which increases the processing load. Furthermore, methods that use sensors increase the cost of introducing the sensors. [Means for solving the problem]

[0006] The information processing device of the present disclosure includes an acquisition means for acquiring information for identifying multiple types of features for each of multiple objects included in an imaging space of an imaging device, and an identification means for identifying each of the multiple objects based on at least one of the information for identifying the multiple types of features, wherein the identification means identifies each of the multiple objects based on a first type of feature among the multiple types of features until the distance between the multiple objects falls below a threshold, and when the distance between the multiple objects falls below the threshold and is no longer below the threshold, identifies each of the multiple objects based on a second type of feature among the multiple types of features that is different from the first type. [Effects of the Invention]

[0007] According to the present disclosure, multiple objects can be appropriately identified. [Brief explanation of the drawings]

[0008] [Figure 1] FIG. 1 is a block diagram showing a schematic configuration of an image processing system. [Figure 2] FIG. 1 is a block diagram showing a hardware configuration of an information processing device. [Figure 3] 1A and 1B are diagrams showing examples of a three-dimensional model of an object and position information of the object. [Figure 4] FIG. 10 is a diagram for explaining a method for identifying an object using coordinate information. [Figure 5] FIG. 10 is a diagram illustrating an example of color information of an object. [Figure 6] FIG. 10 is a diagram for explaining an example of character information of an object. [Figure 7] FIG. 10 is a diagram for explaining the distance state between objects. [Figure 8] 10 is a flowchart illustrating an object identification process. [Figure 9] FIG. 10 is a diagram for explaining an example of object identification information. DETAILED DESCRIPTION OF THE INVENTION

[0009] Hereinafter, the technology of the present disclosure will be described in detail based on embodiments with reference to the accompanying drawings. Note that the configurations shown in the following embodiments are merely examples, and the technology of the present disclosure is not limited to the configurations shown in the drawings.

[0010] In addition, terms that differ only in the alphabet following the reference number indicate different instances of devices with the same function, and the alphabet in the reference number may be omitted when referring to any of the devices with the same function.

[0011] <Embodiment 1> [System Configuration] FIG. 1 is a diagram illustrating an example of an image processing system 1 that generates a virtual viewpoint image. A virtual viewpoint image is an image that represents a view from a virtual viewpoint that is independent of the viewpoint from an actual imaging device. A virtual viewpoint image is generated using a plurality of images obtained by capturing images time-synchronized from a plurality of viewpoints by installing a plurality of imaging devices at different positions. The virtual viewpoint image allows a user to view highlight scenes of a sport such as soccer from various angles, thereby providing the user with a higher sense of realism than a normal captured image. Note that the virtual viewpoint image may be a video or a still image. In the following embodiment, the virtual viewpoint image will be described as a video.

[0012] The image processing system 1 includes a plurality of image capturing devices 111, silhouette image extracting devices 112 connected to each of the image capturing devices 111, a three-dimensional shape generating device 113, a three-dimensional shape storage device 114, and an information processing device 100. The image processing system 1 further includes a virtual viewpoint image generating device 130, an image display device 140, and an input device 120.

[0013] The imaging device 111 is a digital video camera equipped with an image signal interface, such as a serial digital interface (SDI). The imaging device 111 of this embodiment outputs captured image data to the silhouette image extraction device 112 via the video signal interface.

[0014] 1(b) is a bird's-eye view of the arrangement of the multiple imaging devices 111, looking directly from above at the space (imaging space) to be imaged by the multiple imaging devices 111. As shown in FIG. 1(b), the imaging devices 111 are made up of, for example, imaging devices 111a to 111p, and are arranged around the periphery of a field where a game such as soccer is played, and capture images of objects such as players or a ball from various angles in a time-synchronized manner.

[0015] The silhouette image extraction devices 112 are image processing devices corresponding to the respective imaging devices 111. Captured images obtained as a result of imaging by the imaging devices 111 corresponding to the silhouette image extraction devices 112 are input to the respective silhouette image extraction devices 112. The silhouette image extraction devices 112 perform image processing on the input captured images. The image processing performed by the silhouette image extraction devices 112 includes processing to extract foreground regions that indicate the silhouettes of objects included in the input captured images. The silhouette image extraction devices 112 then generate silhouette images that indicate, in binary form, the foreground regions included in the captured image and the background regions that are regions other than the foreground regions. The silhouette image extraction devices 112 also generate texture information of the object, which is image data corresponding to the silhouette of the object.

[0016] The object shown in the foreground of the captured image is a subject that can be seen from a virtual viewpoint, such as a person (player) on the field of a stadium. Alternatively, the object may be an object with a predetermined image pattern, such as a ball or a goal.

[0017] One method for extracting the foreground from a captured image is to use background difference information. In this method, for example, an environmental space in which no objects exist is captured and stored in advance as a background image. Then, an area in which the difference in pixel values ​​between the captured image and the background image is greater than a threshold is determined to be the foreground. Note that the method for extracting the foreground is not limited to the method using background difference information. Other methods for extracting the foreground may also be used, such as a method using parallax, a method using feature values, or a method using machine learning. The generated silhouette image and texture information are output to the three-dimensional shape generation device 113.

[0018] In this embodiment, the silhouette image extraction device 112 and the image capture device 111 are described as being separate devices, but they may be an integrated device, or may be realized by different devices for each function.

[0019] The three-dimensional shape generation device 113 is an image processing device realized by a computer such as a PC, workstation, or server. The three-dimensional shape generation device 113 acquires silhouette images based on captured images (frames) obtained by capturing different field of view ranges from the silhouette image extraction device 112. Based on the silhouette images, it generates data (referred to as three-dimensional shape data or a three-dimensional model) that represents the three-dimensional solid shape of an object included in the captured space.

[0020] One example of a method for generating a 3D model is the commonly used volume intersection method. Volume intersection is a method for obtaining 3D shape information of an object by back-projecting silhouette images corresponding to multiple image capture devices into 3D space and determining the intersection of the volumetric volumes derived from each silhouette image. The generated 3D model is represented as a collection of voxels in 3D space.

[0021] The three-dimensional shape storage device 114 is a device that stores three-dimensional models and texture information. The three-dimensional shape storage device 114 is a storage device that includes a hard disk or the like that can store three-dimensional models and texture data. The three-dimensional shape storage device 114 stores the three-dimensional models and texture information in association with time code information that indicates information about the image capture time. Alternatively, the three-dimensional shape generation device 113 may output data directly to the information processing device 100. In this case, the image processing system 1 may be configured without the three-dimensional shape storage device 114.

[0022] The information processing device 100 is connected to a three-dimensional shape storage device 114. Furthermore, the information processing device 100 is connected to a virtual viewpoint image generation device 130. The information processing device 100 reads out the three-dimensional model and texture information stored in the three-dimensional shape storage device 114, adds object identification information, and outputs the read out information to the virtual viewpoint image generation device 130. Details of the processing by the information processing device 100 will be described later.

[0023] The virtual viewpoint image generating device 130 is connected to an input device 120 that receives instructions from a viewer regarding the position of the virtual viewpoint, etc. The virtual viewpoint image generating device 130 is also connected to an image display device 140 that displays a virtual viewpoint image to the viewer.

[0024] The virtual viewpoint image generating device 130 is a device having a function of generating a virtual viewpoint, and is an image processing device realized by a computer such as a PC, workstation, or server. Based on virtual viewpoint information input via the input device 120, the virtual viewpoint image generating device 130 generates a virtual viewpoint image representing the view from the virtual viewpoint by performing a rendering process in which a texture is projected onto a three-dimensional model based on texture information. The virtual viewpoint image generating device 130 outputs the generated virtual viewpoint image to the image display device 140.

[0025] The virtual viewpoint image generating device 130 may receive three-dimensional position information and object identification information of the object from the information processing device 100, and display information based on the object identification information generated by the information processing device 100. For example, information such as a player's name may be rendered for the object based on the object identification information and superimposed on the virtual viewpoint image.

[0026] The image display device 140 is a display device typified by a liquid crystal display, etc. The virtual viewpoint image generated by the virtual viewpoint image generation device 130 is displayed on the image display device 140 and viewed by a viewer.

[0027] The input device 120 is a device having controllers such as a joystick and switches, and is a device through which a user inputs viewpoint information of a virtual viewpoint. The viewpoint information input by the input device 120 is transmitted to the virtual viewpoint image generation device 130. A viewer can specify the position and direction of the virtual viewpoint using the input device 120 while viewing the virtual viewpoint image generated by the virtual viewpoint image generation device 130 via the image display device 140.

[0028] [Functional configuration of information processing device 100] Next, the functional configuration of the information processing device 100 of this embodiment will be described with reference to Fig. 1. The information processing device 100 has a three-dimensional information acquisition unit 101, an object coordinate acquisition unit 102, an object feature acquisition unit 103, an object identification unit 104, and an object identification information management unit 105.

[0029] The three-dimensional information acquisition unit 101 has a function of reading out the three-dimensional model and texture information of each object in a target frame for generating a virtual viewpoint image, and acquiring the read data, from the three-dimensional shape storage device 114. The three-dimensional information acquisition unit 101 outputs the read out three-dimensional model and texture information to the object coordinate acquisition unit 102, the object feature acquisition unit 103, and the object identification unit 104, which will be described later.

[0030] The object coordinate acquisition unit 102 identifies the coordinates of each object from the three-dimensional model of each object acquired by the three-dimensional information acquisition unit 101, and acquires the object's coordinate information as position information. The object's position feature identified by the position information is called a first type of feature. The object coordinate acquisition unit 102 notifies the object identification unit 104 of the object's position information.

[0031] The object feature acquisition unit 103 acquires information on multiple types of features, different from position features, for each object for which a three-dimensional model is to be generated. In this embodiment, three types of information corresponding to three types of features, namely, the volume, color, and text of the object, are acquired as information on multiple types of features of the object. The three types of features, namely, the volume, color, and text of the object, are referred to as second types of features. Furthermore, simply referring to features refers to the second types of features. Details of the method for acquiring information on object features will be described later.

[0032] The object identification unit 104 determines the type of feature that differs between target objects from the multiple types of object features acquired by the object feature acquisition unit 103. Then, the object identification unit 104 identifies the object based on at least one of the determined type of feature and the object position information acquired by the object coordinate acquisition unit 102. Identifying an object involves identifying which object in another frame corresponds to which object in the current frame; for example, an object can be identified if the distance between multiple objects is equal to or greater than a threshold.

[0033] Then, object identification information representing the result of identifying the object is generated. Object identification information will be described later. Furthermore, the object identification unit 104 may read the object identification information of the previous frame from the object identification information management unit 105 and use it to identify the object of the current frame in detail. For example, an object of the current frame that is identified as corresponding to an object that was identified as player A in the previous frame may be identified as player A. The object identification unit 104 outputs the object identification information to the object identification information management unit 105.

[0034] The object identification information management unit 105 stores and manages the object identification information in a storage unit such as a hard disk.

[0035] In this embodiment, the types of features that are different enough to identify the multiple objects are determined based on the position information of the multiple objects before the multiple objects intersect. This makes it possible to re-identify the multiple objects with a small amount of calculation after the intersection. Details will be described later.

[0036] [Hardware configuration] 2 is a diagram showing the hardware configuration of the information processing device 100. The hardware configurations of the silhouette image extraction device 112, the three-dimensional shape generation device 113, and the virtual viewpoint image generation device 130 are the same as the configuration of the information processing device 100 described below.

[0037] The information processing device 100 includes a CPU 211 , a ROM 212 , a RAM 213 , an auxiliary storage device 214 , a display unit 215 , an operation unit 216 , a communication I / F 217 , and a bus 218 .

[0038] The CPU 211 controls the entire information processing device 100 using computer programs and data stored in the ROM 212 and RAM 213, thereby realizing each functional unit included in the device. Note that the information processing device 100 may have one or more dedicated hardware devices different from the CPU 211, and at least a part of the processing by the CPU 211 may be executed by the dedicated hardware devices. Examples of the dedicated hardware devices include an ASIC (application specific integrated circuit), an FPGA (field programmable gate array), and a DSP (digital signal processor).

[0039] The ROM 212 stores programs that do not require modification. The RAM 213 temporarily stores programs and data supplied from the auxiliary storage device 214, and data supplied from the outside via the communication I / F 217. The auxiliary storage device 214 is configured, for example, with a hard disk drive or the like, and stores various data such as image data and audio data.

[0040] The display unit 215 is configured with, for example, a liquid crystal display, an LED, or the like, and displays a GUI (Graphical User Interface) or the like for the user to operate the information processing device 100. The operation unit 216 is configured with, for example, a keyboard, a mouse, a joystick, a touch panel, or the like, and receives operations by the user to input various instructions to the CPU 211. The CPU 211 operates as a display control unit that controls the display unit 215 and an operation control unit that controls the operation unit 216. In this embodiment, the display unit 215 and the operation unit 216 are described as being present inside the information processing device 100, but at least one of the display unit 215 and the operation unit 216 may be present as a separate device outside the information processing device 100.

[0041] The communication I / F 217 is used for communication with devices external to the information processing device 100. For example, when the information processing device 100 is connected to an external device via a wired connection, a communication cable is connected to the communication I / F 217. When the information processing device 100 has a function for wireless communication with an external device, the communication I / F 217 is equipped with an antenna. The bus 218 connects each unit of the information processing device 100 to transmit information.

[0042] 1 are realized by the CPU 211 of the information processing device 100 executing a predetermined program, but this is not limiting. Other hardware such as a GPU (Graphics Processing Unit) or an FPGA (Field Programmable Gate Array) may also be used to speed up calculations. Each functional unit may be realized by a combination of software and hardware such as a dedicated IC, or some or all of the functions may be realized solely by hardware.

[0043] [About generating 3D models] Figure 3 is a diagram illustrating a three-dimensional model of an object. In Figure 3, the object for which the three-dimensional model is to be generated is a soccer player and a soccer ball that are included in a soccer field, which is the imaging space. For ease of explanation, the three-dimensional model will be explained assuming that there are two players and a soccer ball on the field.

[0044] To generate a three-dimensional model, first, imaging device 111 captures images of a subject (object), such as a soccer player or a soccer ball, from multiple different directions. Imaging devices 111 installed around the soccer field capture images of the object at the same time. Next, silhouette image extraction device 112 separates the object region in the captured image from the background region, which is the region other than the object, and extracts a silhouette image representing the object region. Then, three-dimensional shape generation device 113 generates a three-dimensional model of the object from the silhouette images from multiple different viewpoints using a method such as volume intersection.

[0045] Three-dimensional space 300 shown in Fig. 3 shows a state in which a field, which is an imaging space, is viewed from above. Coordinates 301 in Fig. 3 are coordinates (0,0,0) indicating the origin. The three-dimensional models of objects 311 and 312, which are soccer players on the field, and object 313, which is a soccer ball, are expressed in three-dimensional shapes by, for example, a collection of voxels (voxel groups), which are tiny rectangular parallelepipeds. For example, in the three-dimensional models of objects 311 to 313, which are soccer players and a soccer ball, the three-dimensional shapes at the moment (one frame) captured by imaging device 111 are expressed by the voxel groups.

[0046] In this embodiment, the volume of one voxel is assumed to be one cubic millimeter. Therefore, the three-dimensional shape model of the soccer ball object 313 with a diameter of 22 centimeters in Fig. 3 is generated as a group of spherical voxels with a radius of 110 voxels enclosed by a rectangular parallelepiped of 220 x 220 x 220 mm. Similarly, the three-dimensional models of the soccer player objects 311 and 312 are also generated as groups of voxels.

[0047] The three-dimensional model, in which the three-dimensional shape is expressed by a group of voxels, and texture information (not shown) are stored in three-dimensional shape storage device 114. By repeating this process for each frame, the three-dimensional model and texture information corresponding to each frame of the video obtained by capturing a soccer match are stored. Three-dimensional information acquisition unit 101 of information processing device 100 reads the three-dimensional model and outputs it to object coordinate acquisition unit 102, object feature acquisition unit 103, and object identification unit 104.

[0048] [How to obtain object location information] The object coordinate acquisition unit 102 acquires coordinates as position information of the object by identifying the coordinates of the object for which the three-dimensional model was generated from the three-dimensional model. For example, the coordinates of each of the soccer ball and soccer player objects 311 to 313 shown in FIG. 3 are acquired.

[0049] For example, the coordinates of an object can be specified using a rectangular parallelepiped (called a bounding box) that circumscribes the group of voxels that represent the three-dimensional shape of the object. The coordinates of each of the eight vertices of the bounding box can be calculated from the maximum coordinate value (max) and minimum coordinate value (min) of each of the X, Y, and Z axes of the group of voxels that represent the three-dimensional shape of the object, as shown below. Vertex 1 (Xmin, Ymin, Zmin) Vertex 2 (Xmax, Ymin, Zmin) Vertex 3 (Xmin, Ymax, Zmin) Vertex 4 (Xmax, Ymax, Zmin) Vertex 5 (Xmin, Ymin, Zmax) Vertex 6 (Xmax, Ymin, Zmax) Vertex 7 (Xmin, Ymax, Zmax) Vertex 8 (Xmax, Ymax, Zmax)

[0050] The coordinates of the center of gravity of the object may be calculated from the coordinates of the eight vertices that make up the bounding box of the object, and the coordinates of the center of gravity may be acquired as the coordinates of the object. Alternatively, the coordinates of one of the eight vertices of the bounding box may be acquired as the coordinates of the object. In this embodiment, the description will be given assuming that the coordinates of the one vertex that is closest to the origin out of the eight vertices that make up the bounding box are acquired as the coordinates of the object.

[0051] In the soccer ball object 313 shown in FIG. 3, the coordinates of the vertex closest to the origin of the bounding box 323 are (X,Y,Z)=(50000,15000,0) In this way, the object coordinate acquisition unit 102 can identify the position of the object by acquiring the coordinates of the soccer ball object 313. Similarly, the object coordinate acquisition unit 102 can also acquire the coordinates of the soccer player objects 311 and 312 from the bounding boxes 321 and 322.

[0052] [About tracking based on coordinate information] 4 is a diagram illustrating a comparative example of a method for identifying multiple objects for which a three-dimensional model is generated. Here, a method for identifying objects based on the transition of the object coordinates will be described.

[0053] FIG. 4(a) is the same as FIG. 3, and assumes that one of two objects on the field is associated with player A and the other with player B. To identify which object is player A and which object is player B in a frame after time has passed, if the distance between the multiple objects is sufficiently large, the object is identified based on the transition of the object's coordinates in previous and subsequent frames. For example, the coordinates of each object are obtained, and the object that is the shortest distance from the object's position in the previous frame is identified. In this way, it is possible to identify which object is the object in the current frame, i.e., which object is player A or player B. For example, if the frame rate is 60 fps, the object identification unit 104 acquires coordinates every frame, i.e., every 16.6 milliseconds, to identify the object. Because multiple objects that are sufficiently far apart in the previous frame do not switch positions within such a short period of 16.6 milliseconds, the object can be identified based on the transition of the coordinates over a predetermined time span.

[0054] Figure 4(b) shows a 3D model generated from an image captured at a different time than Figure 4(a) and the positions of objects identified from the 3D model. As shown in Figure 4(b), when the distance between multiple objects falls below a threshold and they overlap (intersect), only one bounding box corresponding to the two objects is recognized. In this case, the positions of the two objects, Player A and Player B, are acquired as the same position.

[0055] FIG. 4(c) is a diagram showing the coordinates of the object and a 3D model generated from the captured image corresponding to the frame next to FIG. 4(b). When two intersecting objects separate, the bounding boxes of the two objects are again recognized as being contained in separate bounding boxes. However, the intersecting (overlapping) objects were acquired in the previous frame as being in the same position. Therefore, it becomes impossible to identify which object in the current frame, i.e., whether it is Player A or Player B, by comparing the transitions in coordinates.

[0056] Therefore, in this embodiment, a method for appropriately identifying a plurality of objects even after the objects intersect will be described.

[0057] [How to obtain information about volume as an object feature] In this embodiment, the object feature acquisition unit 103 acquires information on multiple types of features for each of multiple objects for which a three-dimensional model is to be generated. As described above, in this embodiment, three pieces of information corresponding to three types of features, namely, volume, color, and character, are acquired as the information on multiple types of features. A method for acquiring information on volume, which is a first type of feature of an object, will be described with reference to FIG. 3 .

[0058] To acquire information about the volume of each object, the object feature acquisition unit 103 derives the number of voxels constituting the three-dimensional shape from the three-dimensional model of each object. The reason for using the number of voxels as information about the volume is that, ideally, the number of voxels constituting the three-dimensional shape is proportional to the volume of the actual object.

[0059] For example, if the weight of Object 311, a soccer player, is 80 kg, and the specific gravity of a human is 0.97, the volume of the soccer player is approximately 82,000 cm. 3As mentioned above, the voxel size of one voxel is 1 x 1 x 1 mm. Therefore, the number of voxels required to represent the three-dimensional shape of object 311, a soccer player weighing 80 kg, is approximately 82000 x 10 3 That is, if the silhouette image extraction device 112 properly extracts the silhouette image of the player object 311 and the three-dimensional shape generation device 113 properly generates a three-dimensional model of the object 311, the number of voxel groups will be approximately 82000×10 3 It will be derived as individuals.

[0060] One method for deriving the number of voxel groups is to count the number of voxel groups that represent the three-dimensional shape within the bounding box of the target object. For example, in the case of soccer player object 311 in FIG. 3, it is sufficient to count the number of voxel groups within bounding box 321. In other words, object feature acquisition unit 103 can derive the number of voxel groups that make up the three-dimensional shape of the object by counting the number of voxel groups contained within a rectangular parallelepiped with eight vertices that makes up bounding box 321.

[0061] When the three-dimensional model of the soccer player object 311 is properly generated, the object feature acquisition unit 103 calculates the number of voxels constituting the three-dimensional shape of the object 311 shown in FIG. 3 as 82000×10 3 It will be measured as an individual.

[0062] Also, it is assumed that object 312 is a player who is smaller than object 311. For example, if object 312, a soccer player, weighs 70 kg, and if measured in the same way, the number of voxels that make up the three-dimensional shape of object 312 is approximately 72000×10 3The number of voxels is measured as 100. Comparing the number of voxels between soccer player object 311 and soccer player object 312, there is a difference of more than 10%. Furthermore, since the number of voxels is proportional to the volume of the object, it does not change suddenly depending on the player's posture, etc. For this reason, it is possible to identify multiple person objects with different physiques by comparing the number of voxels, which is information related to volume.

[0063] Furthermore, if the object 313 is a soccer ball, the number of voxels representing the three-dimensional shape is approximately 5500 × 10 3 By comparing the number of voxels, it is possible to identify whether an object is a player or a ball.

[0064] Alternatively, the volume of a bounding box circumscribing the voxels that make up the three-dimensional shape of the object may be acquired as information about the object's volume. In particular, when there is a difference in size between objects, such as a player and a soccer ball, the object can be identified by comparing the volume of the bounding box rather than comparing the number of voxels that make up the three-dimensional shape. The volume of the bounding box is proportional to the volume of the object and can be a volume-related feature for identifying the object.

[0065] The volume of the bounding box 321 of the player object 311 in FIG. 3 is calculated using the following formula: 800×400×1800=576000×10 3 mm 3 From 576000 x 10 3 mm 3 It can be calculated as follows.

[0066] On the other hand, the volume of the bounding box 323 of the soccer ball object 313 is calculated using the following formula: 220×220×220=10648×10 3 mm 3 From 10648 x 10 3 mm 3 It can be calculated as follows.

[0067] For a person such as a player, the volume of the bounding box may change depending on the player's posture. However, no matter what posture the player takes, there will be a difference between the volume of the ball's bounding box and the volume of the player's bounding box. When identifying a player or a ball, the volume of the bounding box may be obtained as information about the object volume, rather than the number of voxels.

[0068] [Method for acquiring color information as an object feature] Fig. 5 is a diagram showing an example of texture information and color histograms corresponding to each object. A method for acquiring information about the color of an object (color information) as information corresponding to a second type of feature of the object will be described using Fig. 5. In this embodiment, a method for generating a color histogram from texture information corresponding to the object and acquiring the representative color of the object as color information will be described.

[0069] FIG. 5(a) is a diagram showing the imaging directions of imaging devices 111 that capture an image of a soccer player, which is an object 311. A plurality of imaging devices 111 are installed to surround the periphery of the object, and each captured image obtained by each imaging device 111 contains texture information of the object. For ease of explanation, in this embodiment, it is assumed that the object 311, which is a soccer player, is captured from four imaging directions 1 to 4. In this case, four pieces of texture information are obtained from the captured images captured from the four imaging directions 1 to 4 shown in FIG. 5(a).

[0070] 5(b) is a diagram showing a captured image 520 obtained by capturing an image from imaging direction 1 out of imaging directions 1 to 4. Image data in an area 522 of the captured image 520 that includes an object is texture information 521 of object 311, which is a soccer player. By projecting the three-dimensional position of the object on the field onto the image coordinates of the imaging device 111 that captured an image from imaging direction 1, the area 522 that includes the object is derived from the captured image obtained by capturing an image from imaging direction 1. This texture information 521 is obtained by extracting image data from the derived area 522.

[0071] The object feature acquisition unit 103 generates a histogram for each RGB color from the texture information 521 shown in Fig. 5(b). The object feature acquisition unit 103 excludes the texture of the background area (black area in Fig. 5(b)) outside the object area of ​​the area 522 from the range of brightness values ​​to generate a color histogram. Using the silhouette image extracted by the silhouette image extraction device 112, it is possible to determine whether the area is an object area or a background area.

[0072] 5(c), (d), and (e) are graphs showing histograms of each RGB color generated by the object feature acquisition unit 103, with the horizontal axis of each graph representing the pixel brightness value and the vertical axis representing the number of pixels. In this embodiment, the brightness value of each color is 8 bits and ranges from 0 to 255. The brightness value that is the most frequent value for each color is determined from the histogram of each RGB color.

[0073] The red (R) histogram in Figure 5(c) shows that the mode was determined to be 120. The green (G) histogram in Figure 5(d) shows that the mode was determined to be 240. The blue (B) histogram in Figure 5(e) shows that the mode was determined to be 100.

[0074] The mode of each color histogram reveals characteristics such as the uniform worn by a player. Comparing the modes of the histograms in Figures 5(c), (d), and (e), the mode of the green (G) component is the highest, so it can be determined that the representative color of object 311 is green. For example, if a player, who is object 311, is wearing a green uniform, green will be determined as the representative color of object 311.

[0075] In a game like soccer, the representative colors of the uniforms of different teams are different, so if soccer players are used as objects, it is possible to identify multiple objects that are players from different teams by comparing the color histograms corresponding to the objects and using the information about the colors of each object.

[0076] Although the representative color is acquired as information about color in the above description, the information about the color of an object is not limited to the representative color.

[0077] Furthermore, in this embodiment, a method for obtaining color information (representative colors) using a histogram generated from texture information in a captured image corresponding to one imaging device has been described. Alternatively, a representative color may be determined based on histograms generated from texture information in multiple captured images corresponding to multiple imaging devices, and an object may be identified based on the representative color. When multiple captured images are used, the number of pixels in the area in which a player appears differs in each captured image. For this reason, a histogram normalized by the size of the texture information may be generated to determine a representative color, and then the object may be identified.

[0078] [Method for acquiring text information as an object feature] Fig. 6 is a diagram showing an example of a method for acquiring characters included in an object. A method for acquiring information (character information) related to characters included in an object as information corresponding to a third type of feature of the object will be described with reference to Fig. 6. In this embodiment, a method for acquiring character information from texture information corresponding to an object will be described.

[0079] Fig. 6(a), like Fig. 5(a), is a diagram showing the imaging directions of the imaging device 111 that images a soccer player, which is an object 311. As in Fig. 5, Fig. 6 will also be described assuming that the object 311 is imaged from four imaging directions 1 to 4.

[0080] 6(b) shows captured images 601 to 604 obtained by capturing images from imaging directions 1 to 4. Each of the captured images 601 to 604 contains texture information 611 to 614 corresponding to the object 311. As described above, the area containing the texture information 611 to 614 in the captured image is obtained by projecting the three-dimensional position of the object on the field onto coordinates in the captured image.

[0081] The object feature acquisition unit 103 performs character recognition processing on the texture information 611 to 614 using optical character recognition technology to acquire character strings contained in the texture information 611 to 614.

[0082] 6(b) includes the number "3" which is the number on the uniform worn by the player, who is the object 311. Therefore, by performing character recognition processing on the texture information 611, a character string representing "3" is obtained.

[0083] On the other hand, depending on the imaging direction, even if it is the same object, there are cases where the character string cannot be recognized from the texture information of the object. Because the captured image 602 is an image obtained by capturing an image of the object 311 facing sideways, the character string cannot be recognized from the texture information 612 of the captured image 602.

[0084] Furthermore, as in the captured image 603, there are cases where part of the character string is hidden by the hand of an object, making it difficult to recognize part of the character string included in the texture information. For this reason, information such as the probability that the character string recognized by the character recognition process was accurately recognized may be further acquired. In this way, the object feature acquisition unit 103 acquires the character string obtained by the character recognition process from the texture information in the captured images obtained by capturing images from various directions.

[0085] Furthermore, the object feature acquisition unit 103 derives a uniform number character string for identifying the object from information on the character string and the probability of the character string obtained from multiple pieces of texture information, and acquires the character string representing the uniform number as information on the character of the object. In Fig. 6(b), the character string "3" is acquired from multiple captured images, so character information indicating that the uniform number of this object is "3" is acquired.

[0086] To derive the character string of the uniform number from the character string obtained by character recognition processing from texture information, the character string of the uniform number can be derived by taking advantage of the fact that each character of the uniform number is displayed larger than the other characters on the uniform.

[0087] For example, in a sports game such as soccer, players' uniforms have numbers printed on them. Normally, players on the same team wear uniforms with different numbers. Therefore, by deriving the string of the uniform number from the string recognized from the texture information and comparing the strings of the uniform numbers of multiple objects, it is possible to identify each of the multiple objects.

[0088] In this embodiment, the character string recognized from the texture information is described as a character string of a uniform number, but other character strings may be recognized and the resulting character string may be acquired as character information of the object. For example, since the uniform also includes the player's name, the name of the player that can identify the object may be determined from the character string obtained by performing character recognition processing on the texture information and acquired as character information.

[0089] In this way, the object feature acquisition unit 103 has the function of acquiring information relating to the volume, color, and character as information representing the features of an object.

[0090] [Object identification using feature-related information] Fig. 7 is a diagram showing three-dimensional models of a plurality of objects in an imaging space. Fig. 7 is a diagram showing soccer players, who are objects 701 to 703 for which three-dimensional models are generated, as seen from above. The object specifying unit 104 will be described using Fig. 7. For simplicity of explanation, the description will be made assuming that there are three objects (players) for which three-dimensional models are generated.

[0091] In this embodiment, the range that is a distance D from an object is defined as the approach area. For example, in FIG. 7, the range that is a distance D from player A, which is object 701, is defined as the approach area 710. Distance D is set as the distance at which objects may intersect in the next frame and their bounding boxes may overlap to become one.

[0092] Conversely, if the distance between the objects is longer than distance D, it is determined that the objects are unlikely to intersect in the next frame. That is, it is determined that player B, which is object 703 outside the approach area 710, is unlikely to intersect with player A, which is object 701, in the next frame.

[0093] Furthermore, in this embodiment, the range where the bounding box of an object intersects with the bounding box of another object to form a single bounding box is defined as the overlap area 720. The overlap area 720 is an area whose radius is a threshold value set based on the distance at which the objects are recognized as a single bounding box. Therefore, if the distance between multiple objects is below the set threshold value, the multiple objects will be included in each other's overlap area 720.

[0094] For example, in the case of object 701 in Fig. 7, the area of ​​a circle that touches the bounding box of object 701 is defined as overlap area 720. As described above, if the bounding boxes of two objects overlap and are recognized as a single bounding box, the object cannot be identified in subsequent frames based on the transition of coordinates.

[0095] Therefore, in this embodiment, when objects approach (intersect) each other and enter each other's overlapping area, and then the distance between them increases and their coordinates can be acquired separately again, the objects are identified based on information about the type of identifiable feature rather than the coordinate information. For this reason, in this embodiment, the object identification unit 104 determines in advance the type of identifiable feature for objects in the approaching area from the multiple types described above.

[0096] For example, if object 702 (player C) is within the approach area 710 of object 701 (player A), it is possible that the objects 701 and 702 may intersect in the next frame. For this reason, it is possible that within a few frames, it may become impossible to identify whether objects 701 and 702 are player A or player C based on the change in coordinates. For this reason, when an object is included in the approach area, a type of feature that can identify each object in the approach area is determined from the multiple types of features described above.

[0097] 7, the object identification unit 104 causes the object feature acquisition unit 103 to acquire information on three types of features for each of the object 701 and the object 702. That is, in this embodiment, the object feature acquisition unit 103 acquires information on volume, information on color (color information), and information on characters (character information) as information on the object features.

[0098] Then, the object specifying unit 104 determines, from the acquired information on the three types of characteristics, the type of characteristic that differs among the plurality of objects in the approach area.

[0099] For example, if the objects 701 and 702 are players from different teams, they may have the same uniform number, and therefore there may be little or no difference in the text information of the objects 701 and 702. However, if the objects are players from different teams, they will be wearing different uniforms, and therefore there will be differences in the color information obtained from the texture information corresponding to the objects. Therefore, the object identification unit 104 can determine the color information as information on the type of different feature that can identify the objects 701 and 702.

[0100] On the other hand, if object 701 and object 702 are players on the same team, it is thought that there will be no difference in color information because they are wearing the same uniform. However, there are no players on the same team with the same uniform number, so there will be differences in the text information. In this case, the object identification unit 104 can determine that the type of information on the different characteristics that can identify object 701 and object 702 is text information. Alternatively, in cases where the physiques of players vary greatly for each position, such as in rugby, information on volume is determined as the type of information on the different characteristics.

[0101] Furthermore, when the object 701 and the object 702 in the approach area are a ball and a player, there is a difference in volume, and therefore information relating to the volume is determined.

[0102] In this way, when another object is included in the approach area, identifiable parameters (features) are selected in advance from multiple candidates. Therefore, even if the object enters an overlapping area and cannot be identified using coordinates alone, it is possible to re-identify the object using the predetermined information. Furthermore, because different information is determined from multiple pieces of information, it is possible to prevent the object from becoming unidentifiable.

[0103] Furthermore, in cases other than identifying an object after it has intersected, the object is identified based on the transition of its coordinates without using information representing its characteristics, as described above. For example, in FIG. 7, object 703 (player B) is outside the approach area 710. In this case, the object identification unit 104 assigns the object identification information identified in the previous frame based on the transition of the coordinates of object 703. For example, if object 703 was player B in the previous frame, object 703 is identified as player B in the current frame as well.

[0104] Acquiring color information and character information requires image processing based on texture information, and image processing generally requires a certain amount of computational load. In this embodiment, object identification using information representing features is limited to some cases, so that objects can be identified while suppressing the amount of computation.

[0105] [Object identification process flow] Fig. 8 is a flowchart illustrating the processing steps of the object identification process of this embodiment. The series of processes shown in the flowchart of Fig. 8 are performed by the CPU of the information processing device 100 by loading program code stored in ROM into RAM and executing it. In addition, some or all of the functions of the steps in Fig. 8 may be realized by hardware such as ASIC or electronic circuits. Note that the symbol "S" in the description of each process indicates a step in the flowchart, and this also applies to subsequent flowcharts.

[0106] In S801, the object identification unit 104 initializes the object identification information.

[0107] Fig. 9 is a diagram illustrating an example of object identification information. The object identification information of this embodiment holds information for each object, including the object ID, identification result, coordinate information, distance state, target object, and identification method. For simplicity's sake, the object identification information in Fig. 9 will be described as object identification information generated when four objects exist in the imaging space.

[0108] "ID" is a unique identifier assigned to an object in the image space. An identifier is assigned to each bounding box that contains the object.

[0109] The "identification result" is information indicating whether the object is a player or a ball, or if it is a player, which player it is.

[0110] The "coordinate information" is information on the position where the object exists, which is acquired by the object coordinate acquisition unit 102.

[0111] "Distance state" is information that indicates the distance between objects, as explained using Figure 7. If the object is outside the overlapping area but within the close area, "close" is retained; if the object is outside the close area, "separate" is retained; and if the object is within the overlapping area, "overlap" is retained. If the distance state changes from overlap to a state other than overlap, "no overlap" is retained.

[0112] A "target object" is an object that is included in the approach area or overlap area when the distance state is "approach" or "overlap," and the "target object" column holds the ID of the target object. For example, if an object with an ID of "1" and an object with an ID of "2" are included in each other's approach area, the column for the target object with an ID of "1" holds "2." Conversely, the column for the target object with an ID of "2" holds "1."

[0113] The "identification method" holds information that is determined as information that is different from the target object from among multiple types of feature information. As described above, when the distance state of a certain object becomes "close," information on the feature that is different between that object and the target object is determined from information representing multiple types of features, and the determined information is held.

[0114] At the time of initialization, the object identification unit 104 acquires information on the coordinates of objects from the object coordinate acquisition unit 102 and updates the "coordinate information" of each object in the object identification information. In this embodiment, for simplicity of explanation, the values ​​held in the coordinate information are the X-axis coordinate and the Y-axis coordinate values. Note that the Z-axis coordinate value may also be acquired as coordinate information.

[0115] The object identification unit 104 determines and updates the "distance state" of each object in the object identification information based on the coordinate information. The following explanation will be given assuming that, at the time of initialization, all objects are outside the approach area and are "independent."

[0116] At the time of initialization, the object specifying unit 104 acquires information on a plurality of types of features for each of all objects in the imaging space from the object feature acquiring unit 103 .

[0117] For example, the object feature acquisition unit 103 acquires information about the volume of the object and identifies whether the object is a player or a ball. Furthermore, the object feature acquisition unit 103 generates, for example, color histograms corresponding to all objects and acquires the representative colors of the uniforms as color information. The object feature acquisition unit 103 also performs character recognition processing on the texture information of all objects to acquire character information of the uniform numbers as character information. The object identification unit 104 then identifies the player names of each object by comparing the color information and character information of the players with a list of players for each team obtained in advance.

[0118] Object identification information 901 in FIG. 9 is an example of object identification information generated by object identification unit 104 during initialization. Through this initialization process, object identification information 901 identifies an object with ID "0" as an object of "player A," and this result is stored in the "identification result" of object identification information 901. Similarly, ID "1" is identified as "player B," and ID "3" is identified as "player C." Furthermore, ID "2" is identified as a ball based on its volume characteristics, and this result is stored in the "identification result." The generated object identification information is saved in the memory by object identification information management unit 105.

[0119] The timing for the object specifying unit 104 to perform initialization is preferably before kick-off in sports such as soccer, when the players, ball, referee, etc. are in an independent state.

[0120] The next steps S802 to S810 identify an object in the current frame to be processed. The object identification process is performed according to the cycle at which coordinate information in the imaging space is updated. For example, if coordinate information in the imaging space is updated at 60 fps, the object for which a 3D model is to be generated is identified every 16.6 milliseconds.

[0121] In S802, the object coordinate acquisition unit 102 acquires the coordinates of the object in the current frame, and the object identification unit 104 updates the "coordinate information" of the object. Based on the updated coordinate information, the object identification unit 104 updates the "distance state" of the object.

[0122] First, the following steps S803 to S810 will be explained using an example in which the current frame is the frame following initialization, and the coordinates of the object in the current frame acquired in S802 are the same as the coordinates held in the coordinate information in the object identification information 901 in Fig. 9. That is, the explanation will be given assuming that the "distance state" of IDs "1" to "4" is all "independent."

[0123] In S803, the object identification unit 104 determines whether there is an object included in the approaching area of ​​any object. If it is determined in S802 that the "distance state" of IDs "1" to "4" is all "independent," the object identification unit 104 determines that there is no object in the approaching state (NO in S803), and the flowchart proceeds to S805.

[0124] In S805, the object identification unit 104 determines whether there is an object included in the overlapping area of ​​any object. If it is determined in S802 that the "distance state" of IDs "1" to "4" is all "independent," the object identification unit 104 determines that there is no object in an overlapping state (NO in S805), and the flowchart proceeds to S807.

[0125] In S807, the object identification unit 104 determines whether any object included in the overlapping area of ​​any object in the previous frame has transitioned to a close state in the current frame. That is, it is determined whether there is any object whose "distance state" is "de-overlapped." If it is determined that the "distance state" of IDs "1" to "4" is all "independent," the object identification unit 104 determines that no object has transitioned from an overlapping state to a close state (NO in S807), and the flowchart transitions to S809.

[0126] In S809, the object identification unit 104 identifies objects by assigning the same ID as the ID assigned in the previous frame to each object based on the transition of coordinates, without using feature information. As shown in object identification information 901, the "identification result" of an object with an ID of "0" at the time of initialization (previous frame) is "Player A," and the "identification result" of an object with an ID of "1" is "Player B." By utilizing this correspondence between the "ID" and "identification result" in the previous frame, objects can be identified in more detail. In this way, when multiple objects are far apart, it is possible to identify objects using coordinate information and the object identification information in the previous frame.

[0127] In S810, the object identifying unit 104 updates the object identifying information using the identification result obtained in S809 to become the object identifying information of the current frame.

[0128] In S811, the object identification unit 104 checks whether an instruction to end the process has been received. If an instruction to end the process has not been received, that is, if there is a next frame, the process returns to S802, and the processes of S802 to S810 are repeated for the next frame.

[0129] [When the distance state includes "approach"] In the next frame, it is assumed that an object with an ID of "0" and an object with an ID of "1" have entered each other's approach area. Furthermore, it is assumed that an object with an ID of "2" and an object with an ID of "3" have entered each other's approach area, and steps S802 to S810 in the next frame will be explained.

[0130] In S802, the object coordinate acquisition unit 102 acquires the coordinates of the object in the next frame. Then, the object identification unit 104 updates the "distance state" of each object to "approaching." The object identification unit 104 further updates the "target object." The "target object" with ID "0" is updated to "1" because the object in the approaching area is an object with ID "1." Similarly, the "target object" with ID "1" is updated to "0."

[0131] In S803, the object identification unit 104 determines whether there is an object included in the approaching area of ​​any object. If it is determined in S802 that the "distance state" for all of the IDs "1" to "4" is "approaching," the object identification unit 104 determines that there is an object in an approaching state (YES in S803), and the flowchart transitions to S804.

[0132] In S804, the object identification unit 104 determines the type of feature to be used to identify the object. The object identification unit 104 compares information on multiple types of features for multiple objects in close proximity to each other, i.e., two objects with IDs "0" and "1." For example, suppose that the objects with IDs "0" and "1" are players on different teams. In this case, there will be at least a difference in the color information obtained based on the color histograms, as described above. Therefore, the object identification unit 104 determines that the different type of feature information for identifying the objects with IDs "0" and "1" is color information.

[0133] Similarly, the object identification unit 104 compares information on multiple types of features for the objects with IDs "2" and "3." Because the object with ID "2" is a ball and the object with ID "3" is a player, there is a difference in at least the information about volume. Therefore, the object feature acquisition unit 103 determines that the information on the type of feature that differs is information about volume.

[0134] Note that, when an object can be identified using multiple pieces of information due to differences in the multiple pieces of information, information with characteristics that requires a low processing load (amount of calculation) when identifying the object may be determined in consideration of the processing load. For example, when differences are found in both color information and character information, if the processing load for identifying the object using color information is low, the object identification unit 104 may determine color information in this step.

[0135] The determination of the difference characteristics may also be performed based on previous history. Although not shown, if there is a history of identifying whether a player is Player A or Player B based on color information, the color information may be determined based on that history.

[0136] Since color histograms are generated and character recognition processing is performed for all objects in the imaging space at the time of initialization, information on the types of different features may be determined based on the identification results at the time of initialization. However, for example, color information may differ from that at the time of initialization due to changes in imaging conditions such as stains on uniforms over the course of a game or changes in sunlight. In such cases where it is considered that the information corresponding to features is different from that at the time of initialization, it is preferable to newly acquire information on multiple types of features for objects in a close state and determine information on the different features.

[0137] The next steps S805 to S806 are the same as those in the previous frame, so the explanation will be omitted.

[0138] In S807, since there was no overlapping state in the previous frame, the object identification unit 104 determines that there is no object that has transitioned from an overlapping state to a close state (NO in S807), and the flow chart proceeds to S809.

[0139] In S809, the object identification unit identifies the object based on the coordinate transition and the "identification result" of the object identification information generated in the previous frame, as described above. Note that even if an object in a close state was not in an overlapping state in the previous frame, the object may be identified using the determined information.

[0140] In S810, the object identification unit 104 updates the object identification information. If information on different characteristics is determined in S804, the object identification unit 104 updates the object identification information so that the determined information is held in the "identification method." For example, the object identification information is updated so that the color information determined in S804 is held in the "identification method" of objects with IDs "0" and "1." Object identification information 902 in FIG. 9 shows an example of object identification information obtained as a result of this update. The updated object identification information is saved by the object identification information management unit 105.

[0141] In this way, by comparing the features of objects that are close to each other, the object identification unit 104 can determine in advance information on the features of the object to be used when the object cannot be identified by the transition of coordinates.

[0142] In S811, the object identification unit 104 checks whether an instruction to end the process has been received. If an instruction to end the process has not been received, that is, if there is a next frame, the process returns to S802, and the processes of S802 to S810 are repeated for the next frame.

[0143] [When distance status includes "overlap"] Furthermore, in the next frame, it is assumed that an object with an ID of "0" and an object with an ID of "1" are in each other's overlapping area, and steps S802 to S810 of the next frame will be explained.

[0144] In S802, the object coordinate acquisition unit 102 acquires the coordinates of the object in the next frame.

[0145] In S803, the object identification unit 104 determines whether there is an object within the approach area. The distance status of the objects with IDs "2" and "3" is "approaching", but since this is the same as in the previous frame, a description of S804 will be omitted.

[0146] In S805, the object identification unit 104 determines whether there is an object included in the overlapping area of ​​any of the objects. If it is determined in S802 that the "distance state" of IDs "1" and "2" is "overlap", the object identification unit 104 determines that there is an object in the "overlap" state (YES in S805), and the flowchart proceeds to S806.

[0147] In S806, the object identification unit 104 updates the object identification information of the object whose distance state is "overlapped."

[0148] Because the objects with ID "0" and ID "1" overlap, the bounding boxes of the two objects are formed as a single bounding box, as shown in FIG. 4(b). Therefore, the positions of the object with ID "1" and the object with ID "0" in the previous frame are acquired as a single object. Therefore, the object identification unit 104 can determine which objects overlap and are recognized as a single object based on the object identification information of the previous frame and the coordinate information of the current frame.

[0149] For example, if the object identification information 902 in FIG. 9 was the object identification information of the previous frame, the object with ID "1" would not be able to be identified using the coordinate transition. Suppose the distance state of the object with ID "1" in the previous frame was "approaching." In this case, it can be determined that the object with ID "1" overlapped with the object with ID "0" that was the target object in the previous frame. As a result, the object identification information of the current frame becomes the state of object identification information 903.

[0150] Therefore, the object identification unit 104 can determine that the object whose distance state is "overlapping" is the object whose ID is "0." Furthermore, from the object identification information 902 of the previous frame, it can be determined that the object whose ID is "0" in the current frame includes player A and player B.

[0151] Next, in S807, since there was no overlapping state in the previous frame, the object identification unit 104 determines that there is no object that has transitioned from an overlapping state to a close state (NO in S807), and the flow chart proceeds to S809.

[0152] In S809, the object identification unit identifies objects other than "overlapping" based on the coordinate transition and the "identification result" of the object identification information generated in the previous frame, as described above.

[0153] In S810, the object identification unit 104 updates the object identification information. The fact that the object with ID "0" is in an "overlapping" state is stored in the "distance state" of the object identification information. As described above, the two objects with IDs "0" and "1" in the previous frame are recognized as a single object with ID "0". However, the color information determined in the previous frame is stored as the identification method (information on different characteristics). Furthermore, the fact that the objects with ID "0" are player A and player B is stored in the "identification information".

[0154] In S811, the object identification unit 104 checks whether an instruction to end the process has been received. If an instruction to end the process has not been received, that is, if there is a next frame, the process returns to S802, and the processes of S802 to S810 are repeated for the next frame.

[0155] [When distance status includes "de-duplication"] Furthermore, in the next frame, the object with ID "0" and the object with ID "1" move out of each other's overlapping area and the overlapping state is resolved, and steps S802 to S810 of the next frame will be described.

[0156] In S802, the object coordinate acquisition unit 102 acquires the coordinates of the object in the next frame.

[0157] Since the overlapping state between the objects with ID "0" and ID "1" has been resolved, the bounding boxes of each object are recognized as separate bounding boxes, as shown in Figure 4(c). In this case, the object identification information will be in the state of object identification information 904 in Figure 9.

[0158] However, an object cannot be identified based on only the coordinate transition and the object identification information 903 of the previous frame. For this reason, an ID of "0" or "1" is provisionally assigned to an object that is close to the position information of an object whose ID in the previous frame was "0." In other words, it is not possible to identify which object among the objects with IDs "0" and "1" is player A and which object is player B based on the coordinate transition and the object identification information 903 of the previous frame.

[0159] Whether the distance state is de-overlapping or not can be determined from the coordinates and the previous frame's object identification information 903. For example, it can be determined that the overlap is de-overlapping by calculating the intersection of the bounding boxes from the coordinates of the eight vertices of the bounding boxes.

[0160] In S803, the object identification unit 104 determines whether there is an object within the approach area. The objects with IDs "2" and "3" have a distance state of "approaching," but since this is the same as in the previous frame, a description of S804 will be omitted.

[0161] In S805, the object identification unit 104 determines whether there is an object included in an overlapping area of ​​any object. In the current frame, the object identification unit 104 determines that there is no object in an overlapping state (NO in S805), and the flowchart proceeds to S807.

[0162] In S807, the object identification unit 104 determines whether any object included in the overlap area of ​​any object in the previous frame has transitioned to a close state in the current frame. In S802, the "distance state" of the objects with IDs "0" and "1" is "de-overlap." Therefore, the object identification unit 104 determines that there is an object that has transitioned from an overlap state to a close state (YES in S807), and the flowchart transitions to S808.

[0163] In S808, the object identification unit 104 identifies the "de-duplicated" object by using information determined in advance when the object is in the approaching state.

[0164] For example, for objects with ID "0" and ID "1", the object identification unit 104 identifies the objects using color information, which is the identification method (information on different characteristics) determined in S804 in the previous frame.

[0165] The object feature acquisition unit 103 generates color histograms for the objects with ID "0" and ID "1" and determines the representative color of each object. From the color information indicating the representative colors acquired by the object feature acquisition unit 103, the object identification unit 104 can identify that the object with ID "0" is player A and the object with ID "1" is player B.

[0166] For objects that are not "de-duplicated," the object may be identified based on the coordinate transition and the object identification information of the previous frame, similar to the process of S809.

[0167] Next, in S810, the object identification unit 104 updates the object identification information. The object identification unit 104 updates the object identification information so that "Player A" is held in the "Identification result" with ID "0" and "Player B" is held in the "Identification result" with ID "1", as in object identification information 905 in Fig. 9. The object identification information is saved by the object identification information management unit 105.

[0168] In S811, the object identification unit 104 checks whether an instruction to end the process has been received. If an instruction to end the process has not been received, that is, if there is a next frame, the process returns to S802, and the processes of S802 to S810 are repeated for the next frame. If an instruction to end the process has been received, the process ends.

[0169] As described above, according to this embodiment, when an overlapping state (a state in which objects are close to each other and intersect) is resolved, an identification process is performed using information on the different characteristics of the multiple objects whose overlapping state has been resolved. Therefore, according to this embodiment, it is possible to re-identify the objects whose overlapping state has been resolved. Furthermore, compared to a method of identifying objects that uses information on the characteristics of all objects, the method of this embodiment makes it possible to re-identify the objects whose overlapping state has been resolved while reducing the amount of calculation required for processing.

[0170] Furthermore, in this embodiment, since information effective for identification is determined in advance, when re-identifying an object after the overlap state is resolved, there is no need to identify the object using multiple types of features. Therefore, according to this embodiment, it is possible to re-identify the object at high speed while suppressing the amount of calculation required for processing.

[0171] In the above description, an object is identified based on a transition in coordinates before the object intersects, but the object may be recognized using information about its features regardless of whether the object intersects. For example, if the volume of an object in the captured space for which a three-dimensional model is generated varies from object to object, the object may be identified using information about its volume regardless of whether the object intersects.

[0172] <Other embodiments> In the above-described embodiment, it has been described that the silhouette image extraction device 112 generates a silhouette image, the three-dimensional shape generation device 113 generates a three-dimensional model, and the virtual viewpoint image generation device 130 generates a virtual viewpoint image. Alternatively, for example, the information processing device 100 may generate at least one of the silhouette image, the three-dimensional model, and the virtual viewpoint image.

[0173] The present disclosure can also be realized by supplying a program that realizes one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and having one or more processors in the computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that realizes one or more functions. [Explanation of symbols]

[0174] 100 Information processing device 103 Object feature acquisition unit 104 Object Identification Unit

Claims

1. an acquisition means for acquiring information for identifying a plurality of types of features for each of a plurality of objects included in an imaging space of the imaging device; an identification unit that identifies each of the plurality of objects based on at least one of the pieces of information for identifying the plurality of types of features, The identification means identifying each of the plurality of objects based on a first type of feature among the plurality of types of features until a distance between the plurality of objects falls below a threshold; When the distance between the plurality of objects falls below the threshold and the distance between the plurality of objects is no longer below the threshold, each of the plurality of objects is identified based on a second type of feature that is different from the first type of feature among the plurality of types of features.

1. An information processing device comprising:

2. When the distance between the plurality of objects falls below the threshold and is no longer below the threshold, and when the distance between the plurality of objects falls below another threshold that is greater than the threshold, the specifying means specifies each of the plurality of objects based on the second type of feature.

2. The information processing apparatus according to claim 1, wherein:

3. The first type of feature is the position of each of the plurality of objects in the imaging space.

3. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

4. The positions are obtained based on a bounding box including the three-dimensional shapes represented by the three-dimensional shape data of the plurality of objects.

4. The information processing apparatus according to claim 3,

5. The second type of feature is a feature related to at least one of the color, character, or volume of each of the plurality of objects.

5. The information processing apparatus according to claim 1, wherein the information processing apparatus is a computer.

6. The second type of feature is obtained based on at least one of three-dimensional shape data of the plurality of objects and the captured image of the imaging device.

6. The information processing apparatus according to claim 5,

7. the second type of feature is at least one of a feature related to a color of each of the plurality of objects or a feature related to a character; The color feature or the character feature is acquired based on the captured image.

7. The information processing apparatus according to claim 6,

8. The color-related characteristics are: A histogram for each color in the object region of the captured image is acquired, and the feature is obtained based on the most frequent value of the histogram.

8. The information processing apparatus according to claim 7,

9. The character features are: The features of the characters obtained by performing character recognition processing on the object region of the captured image are 9. The information processing device according to claim 7, wherein:

10. The character feature is a character representing the uniform number of the object.

10. The information processing device according to claim 7, wherein the information processing device is a computer.

11. the second type of features are volumetric features, The volumetric features are obtained based on three-dimensional shape data of the plurality of objects.

11. The information processing device according to claim 5, wherein the information processing device is a computer.

12. The identifying means identifies an object in a previous frame that corresponds to an object in a current frame.

12. The information processing device according to claim 1, wherein the information processing device is a computer.

13. When the distance between the plurality of objects is less than the threshold value, the specifying means specifies the plurality of objects as a single object.

13. The information processing device according to claim 1, wherein the information processing device is a computer.

14. The three-dimensional shape data of the object is used to generate a virtual viewpoint image.

14. The information processing device according to claim 1,

15. an acquisition step of acquiring information for identifying multiple types of features for each of multiple objects included in an imaging space of the imaging device; and an identifying step of identifying each of the plurality of objects based on at least one of the information for identifying the plurality of types of features, In the identifying step, identifying each of the plurality of objects based on a first type of feature among the plurality of types of features until a distance between the plurality of objects falls below a threshold; When the distance between the plurality of objects falls below the threshold and the distance between the plurality of objects is no longer below the threshold, each of the plurality of objects is identified based on a second type of feature that is different from the first type of feature among the plurality of types of features.

1. An information processing method comprising:

16. A program for causing a computer to function as each of the means of the information processing device according to any one of claims 1 to 14.

Images