Information processing device, method for controlling information processing device, and program

Abstract

To accurately create a three-dimensional posture model of a subject.SOLUTION: An information processing apparatus 10 acquires a foreground image that is an image indicating an area of a subject and information on the posture of the subject. The information processing apparatus creates a distance image from an imaging apparatus to the subject based on a three-dimensional shape model of the subject created based on the plurality of foreground images and a camera parameter of the imaging apparatus. The information processing apparatus creates a three-dimensional posture model of the subject based on the information on the posture of the subject, the distance image of the imaging apparatus, and the camera parameter of the imaging apparatus.SELECTED DRAWING: Figure 5

Description

[Technical field]

[0001] The present disclosure relates to an information processing device, Information processing device The present invention relates to a control method and a program, and in particular to a technique for estimating the posture of a subject (person). [Background technology]

[0002] Technology that estimates a posture model of a subject using images taken from multiple different camera positions is attracting attention. A posture model is information that represents the positions of the joints that make up the subject, the connections between the joints, and the angles between parts of the subject. By using posture estimation technology to estimate the posture model of a subject captured in camera images, it becomes possible to perform motion analysis such as an athlete's amount of exercise, fatigue level, and comparison of old and new forms.

[0003] In Patent Document 1, a 3D pose model of a single subject is estimated by repeatedly deforming a shape model using a pose model so that an area obtained by projecting a deformable shape model onto a captured image and an object area extracted from the captured image approximately match. The 3D pose model is a model in which joint positions and angles are expressed in an arbitrary coordinate system of a 3D space. [Prior art documents] [Patent documents]

[0004] [Patent Document 1] JP 2016-126425 A Summary of the Invention [Problem to be solved by the invention]

[0005] The technique of Patent Document 1 has a problem in that when there are multiple subjects, the subjects overlap each other and joints and the like are hidden, resulting in large errors in generating a posture model.

[0006] The present disclosure has been made in consideration of the above-mentioned problems, and has an object to provide a technique for generating a three-dimensional posture model of a subject with high accuracy. [Means for solving the problem]

[0007] The information processing device according to the present disclosure that achieves the above object includes: an acquisition means for acquiring a plurality of subject images showing an area of ​​a subject photographed by a plurality of photographing devices and posture information of the subject; a first generation means for generating a three-dimensional shape model of the subject based on a plurality of subject images acquired by the acquisition means and parameters for identifying positions and orientations of the plurality of image capturing devices; a second generating means for generating a plurality of distance images indicating distances from the plurality of photographing devices to the subject based on the three-dimensional shape model generated by the first generating means; a third generation means for generating a three-dimensional posture model of the subject based on posture information of the subject acquired by the acquisition means, the plurality of distance images generated by the second generation means, and the parameters of the plurality of image capturing devices; has. Effect of the Invention

[0008] According to the present disclosure, a three-dimensional posture model of a subject can be generated with high accuracy. [Brief description of the drawings]

[0009] [Figure 1] 1 is a diagram illustrating an example of the configuration of an information processing system according to a first embodiment. [Diagram 2] FIG. 2 is a schematic diagram showing an example of a coordinate system relating to camera parameters according to the first embodiment. [Diagram 3] 2A to 2C are schematic diagrams showing a three-dimensional shape model and a three-dimensional pose model according to the first embodiment. [Figure 4] 1 is a diagram illustrating an example of the hardware configuration of an imaging device and an information processing device according to a first embodiment. [Diagram 5] FIG. 2 is a diagram showing a processing sequence of the information processing system according to the first embodiment. [Figure 6]1A is a flowchart showing the procedure of processing performed by the photographing device according to the first embodiment, and FIG. 1B is a flowchart showing the procedure of processing performed by the information processing device according to the first embodiment. [Figure 7] FIG. 2 is a diagram showing an example of a silhouette image according to the first embodiment. [Figure 8] 5 is a flowchart showing details of a three-dimensional posture model estimation process performed by the information processing device according to the first embodiment. [Figure 9] FIG. 1 is a schematic diagram for explaining a method for calculating three-dimensional joint position candidates according to the first embodiment; [Figure 10] FIG. 11 is a diagram illustrating an example of the configuration of an information processing system according to a second embodiment. [Figure 11] FIG. 11 is a schematic diagram showing an imaging area according to the second embodiment. [Figure 12] 11 is a flowchart showing the procedure of a process executed by an information processing device according to the second embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0010] Hereinafter, the embodiments will be described in detail with reference to the accompanying drawings. Note that the following embodiments do not limit the present disclosure. Although the embodiments describe a plurality of features, not all of these features are necessarily essential to the present disclosure, and the plurality of features may be combined in any manner. Furthermore, in the accompanying drawings, the same reference numbers are used for the same or similar configurations, and duplicated descriptions are omitted.

[0011] (Embodiment 1) In this embodiment, an example is described in which an optimal 3D joint position is estimated (identified) for each subject from multiple 3D joint position candidates obtained from information from each imaging device, thereby estimating (generating) a 3D posture model of the subject.

[0012] <Configuration of information processing system> 1 is a configuration diagram of an information processing system in this embodiment. The information processing system includes n image capturing devices (image capturing device 1, image capturing device 2, ..., image capturing device n) and an information processing device 10. The information processing device 10 is a device for estimating (generating) a three-dimensional posture model of a subject, and is connected to the n image capturing devices.

[0013] The n camera devices are multi-view cameras that capture a subject from many different directions. The capture area of ​​the multi-view camera is, for example, an area surrounded by the plane of a stadium where sports are played and an arbitrary height. Each camera device is installed in a different position and direction so as to surround the capture area, and captures images in synchronization. The camera devices do not need to be installed all around the capture area, and may be installed only in a part of the capture area depending on restrictions on the installation location. The number of camera devices is not limited, and for example, when the capture area is a soccer or rugby stadium, several tens to several hundreds of camera devices may be installed around the stadium. Camera devices with different angles of view, such as telephoto cameras and wide-angle cameras, may also be installed. All camera devices are synchronized with one time information in the real world, and the captured image has the capture time information added to the image of each frame.

[0014] Furthermore, each image capture device has status information such as the position, attitude (direction, image capture direction), focal length, optical center, and distortion of the image capture device. The position and attitude (direction, image capture direction) of the image capture device may be controlled by the image capture device itself, or may be controlled by a camera platform that controls the position and attitude of the image capture device. In the following, the state of the image capture device will be described as the camera parameters of the image capture device, but the camera parameters may include parameters controlled by another device such as a camera platform. Moreover, the camera parameters related to the position and attitude (direction, image capture direction) of the image capture device are so-called external parameters, and the camera parameters related to the focal length, image center, and distortion of the image capture device are so-called internal parameters.

[0015] FIG. 2(a) shows an example in which there are four cameras (camera 110 to camera 113) capturing an image of a subject 205 (i.e., n=4). Information on the position and orientation of the capturing devices is expressed in a single world coordinate system represented by an origin 200, an Xw axis 201, a Yw axis 202, and a Zw axis 203. Also, as shown in FIG. 2(b), a captured image 250 of camera 110, an origin 260 of the camera image coordinate system, an Xi axis 261, and a Yi axis 262 are set. The camera image coordinate systems of the other cameras can be defined in a similar manner.

[0016] <Configuration of imaging device> The configuration of the photographing device 1 will be described below as an example, but the other photographing devices have the same configuration as the photographing device 1. The photographing device 1 includes a camera 110, a foreground region extraction section 120, and a posture information estimation section .

[0017] The foreground region extraction unit 120 extracts a foreground region (foreground image) indicating the region of a subject from a photographed image captured by the camera 110. Here, the subject generally refers to a dynamic subject (moving body) that moves (whose position or shape may change) when photographed from the same direction in chronological order. The subject may be, for example, a person such as a player or a referee on the field where a sport is being played, or a ball in a ball game in addition to a person. In addition, in concerts and entertainment, the subject may be a singer, a player, a performer, a presenter, etc. When multiple subjects are captured in a photographed image, multiple foreground images are obtained. The foreground image includes information on the position, width, and height of the foreground region relative to the photographed image.

[0018] The posture information estimation unit 130 estimates (specifies) posture information of the subject (joint position candidates of each joint) based on the foreground region (foreground image) extracted by the foreground region extraction unit 120. The posture information estimation unit 130 detects the joint positions of the defined posture model from the foreground image, which is a two-dimensional image. The estimation method may be an existing method using machine learning or the like.

[0019] FIG. 3 shows an example of a three-dimensional shape model and a three-dimensional posture model imitating a standard human shape. Here, the three-dimensional shape model 300 is represented by a set of points called voxels, and each point has three-dimensional coordinates in a world coordinate system. The three-dimensional shape model 300 may be a three-dimensional mesh model used in CG (Computer Graphics). The posture model 310 includes a plurality of joints 311 representing the positions of main parts and joints of the human body such as the head, neck, navel, shoulders, elbows, wrists, groin, knees, and ankles, information 312 indicating their connection relationships, and angle information between adjacent parts. For example, the joint angle of the left knee 321 is obtained by calculating the angle formed by a vector obtained by the three-dimensional coordinates of the joints 320 and 321 and a vector obtained by the three-dimensional coordinates of the joints 321 and 322.

[0020] The photographing device 1 outputs a foreground image and posture information of the subject to the information processing device 10.

[0021] Furthermore, the photographing device 1 has a detection unit (not shown) that detects an abnormality in the camera 110, and can send abnormality information notifying the presence or absence of an abnormality to the information processing device 10. An abnormality in the camera 110 can be detected, for example, by evaluating values ​​of general sensors such as heat and vibration.

[0022] <Configuration of information processing device> Next, the configuration of information processing device 10 will be described with reference to Fig. 1. Information processing device 10 includes shooting information acquisition section 140, shape model estimation section 150, distance image generation section 160, and posture model estimation section 170.

[0023] The photographing information acquisition unit 140 acquires a foreground image showing the area of ​​the subject and posture information of the subject from each of the photographing devices 1 to n. Furthermore, the photographing information acquisition unit 140 acquires camera abnormality information output from each of the photographing devices 1 to n when an abnormality occurs.

[0024] The shape model estimation unit 150 estimates (generates) a three-dimensional shape model of the subject based on the foreground image of each of the photographing devices 1 to n acquired by the photographing information acquisition unit 140 and the camera parameters of each photographing device. When there are multiple subjects, the shape model estimation unit 150 estimates (generates) a three-dimensional shape model for each subject. For example, a shape from silhouette method is used to estimate the three-dimensional shape model.

[0025] Here, the camera parameters may be stored in a memory as a file containing previously calculated parameters, and managed for each identification ID of each photographing device. Alternatively, the shape model estimation unit 150 may calculate these camera parameters. For example, the shape model estimation unit 150 calculates corresponding points from the photographed images of each photographing device, and calibrates each photographing device while optimizing the corresponding points so that the error when projecting them onto each photographing device is minimized, thereby calculating the camera parameters. The calibration method may be any existing method. The camera parameters may be acquired in synchronization with the photographed images, or may be acquired at the stage of advance preparation. They may be acquired asynchronously with the photographed images as necessary. When an abnormality in any of the cameras is detected based on the camera abnormality information, control is performed to exclude information acquired from the photographing device including the camera from use in the shape estimation process.

[0026] Distance image generation unit 160 generates a distance image for each imaging device using the 3D shape model of each subject estimated by shape model estimation unit 150 and the camera parameters of each imaging device. Specifically, all voxels constituting the 3D shape model of the subject estimated by shape model estimation unit 150 are projected onto each imaging device to generate a distance image for each imaging device.

[0027] Pose model estimation unit 170 estimates (generates) posture model 310 of the subject using posture information of the subject appearing in the images captured by each image capture device (joint position candidates for each joint), distance images of each image capture device, and camera parameters of each image capture device. A specific method will be described in detail in the operation flow.

[0028] Here, the hardware configuration of the photographing device 1 excluding the camera and the hardware configuration of the information processing device 10 will be described with reference to FIG. 4. Each device has a CPU 411, a ROM 412, a RAM 413, an auxiliary storage device 414, a display unit 415, an operation unit 416, a communication I / F 417, and a bus 418. CPU is an abbreviation for Central Processing Unit, ROM is an abbreviation for Read Only Memory, RAM is an abbreviation for Random Access Memory, and I / F is an abbreviation for Interface. The CPU 411 realizes the function of each device by controlling the entire device using computer programs and data stored in the ROM 412 and the RAM 413. Each device may have one or more dedicated hardware different from the CPU 411, and at least a part of the processing by the CPU 411 may be executed by the dedicated hardware. Examples of the dedicated hardware include an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), a DSP (Digital Signal Processor), and a GPU (Graphics Processing Unit). The ROM 412 stores programs that do not require modification. The RAM 413 temporarily stores programs and data supplied from the auxiliary storage device 414, and data supplied from the outside via the communication I / F 417. The auxiliary storage device 414 is constituted by, for example, a hard disk drive, and stores various data such as image data and audio data.

[0029] The display unit 415 is composed of, for example, a liquid crystal display, an LED (Light Emitting Diode), etc., and displays a GUI (Graphical User Interface) etc. for the user to operate the information processing device 10. The operation unit 416 is composed of, for example, a keyboard, a mouse, a joystick, a touch panel, etc., and inputs various instructions to the CPU 411 in response to operations by the user. The CPU 411 operates as a display control unit that controls the display unit 415 and an operation control unit that controls the operation unit 416.

[0030] The communication I / F 417 is used for communication between each device and an external device. For example, when each device is connected to an external device by wire, a communication cable is connected to the communication I / F 417. When each device has a function for wireless communication with an external device, the communication I / F 417 is equipped with an antenna. The bus 418 connects each part of each device to transmit information.

[0031] In this embodiment, the display unit 415 and the operation unit 416 are assumed to exist inside each device, but at least one of the display unit 415 and the operation unit 416 may exist outside the device as a separate device.

[0032] <Processing sequence of information processing system> Next, the processing procedure of the information processing system according to this embodiment will be described with reference to the sequence of FIG.

[0033] In F501, foreground region extraction unit 120 of each of photographing devices 1 to n extracts a silhouette image of a subject as a foreground image showing the region of the subject from the photographed image captured by camera 110. In F502, each of photographing devices 1 to n outputs the extracted foreground image to information processing device 10.

[0034] In F503, the posture information estimation unit 130 of each of the photographing devices 1 to n estimates (generates) information on the joint position candidates of the subject from the foreground image extracted in F501 as posture information of the subject. In F504, each of the photographing devices 1 to n outputs the estimated posture information to the information processing device 10.

[0035] In F505, the photographing information acquisition unit 140 of the information processing device 10 acquires the camera parameters of each photographing device. For example, the camera parameters of each photographing device can be acquired by reading a file in which the camera parameters of each photographing device are described.

[0036] In F506, the shape model estimation unit 150 of the information processing device 10 estimates (generates) a three-dimensional shape model of the subject based on the foreground images acquired from each image capturing device and the camera parameters of each image capturing device.

[0037] In F507, the distance image generation unit 160 of the information processing device 10 generates a distance image of the subject viewed from each of the photographing devices by projecting all voxels that constitute the three-dimensional shape model of the subject estimated by the shape model estimation unit 150 onto each of the photographing devices, photographing device 1 to photographing device n.

[0038] In F508, the posture model estimation unit 170 of the information processing device 10 estimates (generates) a three-dimensional posture model of the subject based on the foreground image and posture information (information on candidate joint positions of each joint of the subject) acquired from each photographing device, the distance images of each photographing device generated by the distance image generation unit 160, and the camera parameters of each photographing device.

[0039] This completes the sequence in Fig. 5. If there are multiple subjects, processing is performed for each subject. Details of each process will be described in detail with reference to the flowcharts described later.

[0040] <Processing of imaging device> First, the procedure of the process performed by each of the photographing devices 1 to n according to this embodiment will be described with reference to the flowchart in Fig. 6(a). Since the process performed by each photographing device is the same, the following description will be given taking the photographing device 1 as an example.

[0041] In S601, the camera 110 of the image capturing device 1 captures an image. If the subject of the image capturing is a sports event, the image capturing device 1 captures an image of the competition field.

[0042] In S602, the foreground region extraction section 120 of the image capturing device 1 extracts, from the captured image captured by the camera 110, a silhouette image of a subject (person) as a foreground image indicating the region of the subject.

[0043] FIG. 7 shows an example of a silhouette image generated when four cameras (i.e., n=4), cameras 110 to 113 shown in FIG. 2(a), are installed around the subject at 90 degree intervals and photograph the subject. Reference numeral 701 denotes a silhouette image of the left side of the subject, 702 denotes a silhouette image of the back of the subject, 703 denotes a silhouette image of the right side of the subject, and 704 denotes a silhouette image of the front of the subject. The pixels of each silhouette image express the foreground region with a value other than 0 (white) and the background region with 0 (black). The silhouette image can be generated using a general method such as a background difference method that calculates the difference between a photographed image of the subject and a background image photographed in advance when the subject does not exist, such as before the start of a match. However, the method of generating the silhouette image is not limited to this. For example, the subject region may be extracted using a method such as recognizing the subject (human body). The silhouette image may also be generated by erasing texture information from the photographed image. Specifically, the pixel value of the region where the subject exists may be set to a value other than 0, and the pixel value of the other region may be set to 0.

[0044] In S603, the foreground region extraction section 120 of the photographing device 1 outputs the foreground image extracted in S602 to the information processing device 10.

[0045] In S604, the posture information estimation section 130 of the image capturing device 1 estimates (specifies) posture information of the subject (information on candidate joint positions of each joint) from the foreground image extracted in S602.

[0046] Any method may be used to detect joint position candidates from an image, but here, the posture information estimation unit 130 uses machine learning to perform inference. For this purpose, learning data is created by specifying the joint positions of the subject (human) in the captured image in advance, and a file describing the results of learning the joint positions is input to the posture information estimation unit 130. The joint positions to be specified may be representative joint positions such as the multiple joints 311 in FIG. 3, for example. During inference, a foreground image is input, and a joint position candidate image showing the joint position candidate with a probability is obtained for each joint position as an inference result. If the pixel value of this image is likely to be a joint position, the probability is expressed as high, and if it is unlikely to be a joint position, the probability is expressed as low. The joint position candidate is set to the camera image coordinate of the pixel with the maximum probability in each joint position candidate image. For example, when obtaining the joint position (image coordinate system) of the left knee, the joint position coordinates (Xi, Yi) on the image can be obtained by calculating the pixel with the maximum probability from the joint position candidate image of the left knee. In this way, the joint position candidate information of each joint (joint position candidate image, coordinates of the joint position candidate) can be estimated as posture information of the subject.

[0047] In S605, the posture information estimation section 130 of the image capturing apparatus 1 outputs to the information processing apparatus 10 the posture information of the subject acquired in S604 (information on the joint position candidates of each joint).

[0048] In S606, the photographing device 1 determines whether or not photographing is continuing. If photographing is continuing, the process returns to S601 and repeats the series of processes. On the other hand, if photographing is ended by an instruction to end photographing from the information processing device 10, the process ends.

[0049] <Processing of information processing device> Next, the procedure of the process performed by the information processing device 10 according to the present embodiment will be described with reference to the flowchart in FIG.

[0050] In S611, the photographing information acquisition unit 140 acquires the camera parameters of each photographing device. For example, the camera parameters of each photographing device can be acquired by reading a file in which the camera parameters of each photographing device are described.

[0051] In S612, shooting information acquisition unit 140 acquires the foreground image and posture information of the subject (information on candidate joint positions of each joint) output from each shooting device. Then, the shooting information acquisition unit 140 outputs the foreground image to shape model estimation unit 150 and the posture information to posture model estimation unit 170.

[0052] In S613, shape model estimation unit 150 estimates (generates) a three-dimensional shape model of the subject based on the foreground images of each image capturing device acquired in S612 and the camera parameters of each image capturing device acquired in S611. Then, information on the estimated three-dimensional shape model is output to distance image generation unit 160.

[0053] Here, the estimation of the 3D shape model uses, for example, a volume intersection method (shape from silhouette). The estimation result is expressed, for example, as a voxel, which is a set of 3D coordinates. Other general methods can also be used for shape estimation. Furthermore, the obtained voxel set is divided into voxel sets for each object by performing a general 3D labeling process based on the presence or absence of adjacent voxels. Each voxel is assigned a subject ID, which is the labeling result, and the voxels for each object can be obtained by specifying the subject ID.

[0054] In S614, distance image generation unit 160 projects all voxels constituting the 3D shape model of the subject estimated in S613 onto each imaging device to generate distance images of each imaging device, and then outputs information on the 3D shape model of the subject and the distance images to posture model estimation unit 170.

[0055] Here, to generate a distance image of one imaging device, first, the central coordinates (Xw, Yw, Zw) of a voxel expressed in the world coordinate system are transformed into the camera image coordinate system (Xi, Yi, Zi) using the camera parameters. Then, Zi is stored as a temporary distance in the pixel at image coordinate (Xi, Yi). When different voxels are transformed into the same image coordinates, the one with the closer distance from the camera (e.g., the smaller Zi value) is stored as the distance value of that pixel. By repeating these processes for all voxels, a distance image can be generated in which the pixel value is the shortest distance from the imaging device to the surface of the subject. Furthermore, by performing the same process for all imaging devices, distance images for all imaging devices can be generated.

[0056] In S615, posture model estimation unit 170 estimates (generates) a three-dimensional posture model of the subject based on the foreground images of each imaging device and posture information of the subject (information on candidate joint positions of each joint) acquired in S612, the distance images of each imaging device generated in S614, and the camera parameters of each imaging device acquired in S611.

[0057] The process of S615 will now be described in detail with reference to the flowchart of FIG.

[0058] In S6151, the posture model estimation unit 170 selects the foreground image of each image capture device corresponding to the three-dimensional shape model of the first subject estimated in S613 from the foreground image acquired in S612. This is because, when there are multiple subjects, multiple foreground images corresponding to the multiple subjects are acquired from each image capture device in S612. For this purpose, first, the center of gravity coordinates 810 (world coordinate system) of the three-dimensional shape model 800 are calculated as shown in FIG. 9. The center of gravity coordinates 810 can be calculated by calculating the average of the center coordinates of all voxels constituting the three-dimensional shape model 800. Then, the center of gravity coordinates 810 are transformed into the camera image coordinate system, and it is determined whether the foreground image includes the camera image coordinates of the center of gravity in the foreground area. If it is determined that the camera image coordinates of the center of gravity are included in the foreground area, the foreground image is selected as the foreground image corresponding to the three-dimensional shape model 800.

[0059] In S6152, posture model estimation unit 170 acquires distance d of the coordinates of the joint position candidate of each joint from the distance image based on the posture information acquired in S612 (information of the joint position candidate of each joint) and the distance image generated in S614, and calculates coordinates (Xi, Yi, d). Then, by transforming the coordinates (Xi, Yi, d) into the world coordinate system, it calculates and acquires coordinates (Xw, Yw, Zw) of the 3D joint position candidate 811.

[0060] By repeating these processes on the acquired foreground images, a set of 3D joint position candidates 811, 812, 813, 814 for, for example, the left knee of the first subject in each imaging device (when the number of imaging devices, n, is 4) is obtained. Furthermore, by performing similar processes on other joints, a set of 3D coordinates indicating 3D joint position candidates is obtained for all joints of each subject.

[0061] In S6153, posture model estimation unit 170 estimates an optimal three-dimensional joint position (joint position 820 in the example of FIG. 9) based on the set of three-dimensional joint position candidates acquired in S6152. For example, the optimal three-dimensional joint position is estimated by calculating the average of the set of three-dimensional coordinates. Alternatively, an equation of a three-dimensional ellipse may be fitted to the set of three-dimensional coordinates, and the coordinates of the center of the fitted ellipse (center position) may be estimated as the optimal joint position, or any other method may be used.

[0062] In S6154, posture model estimation unit 170 calculates the angle between the joints from the information of the optimal three-dimensional joint positions estimated in S6153. Connection information of each three-dimensional joint position is input in advance as known information, such as information 312 shown in FIG. 3. For example, in the example of FIG. 3, the joint angle of left knee 321 is obtained by calculating the angle between a vector obtained in the three-dimensional coordinates of joints 320 and 321 and a vector obtained in the three-dimensional coordinates of joints 321 and 322. By repeating this process for all joints, a three-dimensional posture model of the subject can be estimated.

[0063] In S6155, posture model estimation section 170 determines whether or not 3D posture models have been estimated for all objects. If they have been estimated for all objects, the process in FIG. 8 ends. On the other hand, if there are still objects that have not been estimated, the process returns to S6151 and is repeated. When the process in FIG. 8 ends, the process in FIG. 6 also ends.

[0064] As described above, in this embodiment, a plurality of subject images showing the area of ​​the subject photographed by a plurality of image capturing devices and posture information of the subject are acquired, and a three-dimensional shape model of the subject is generated based on the acquired plurality of subject images and parameters for identifying the positions and postures of the plurality of image capturing devices. Then, a plurality of distance images showing the distances from the plurality of image capturing devices to the subject are generated based on the generated three-dimensional shape model. Furthermore, a three-dimensional posture model of the subject is generated based on the acquired posture information of the subject, the generated plurality of distance images, and parameters of the plurality of image capturing devices.

[0065] In this manner, in this embodiment, an optimal 3D joint position is estimated (specified) for each subject from multiple 3D joint position candidates obtained from information of each image capture device. Therefore, even if multiple subjects are present and a joint of a subject is hidden by another subject as viewed from any image capture device, a 3D posture model can be estimated with high accuracy for each subject. In addition, by distributing the process of generating posture information (information of joint position candidates of each joint) to each image capture device, a posture model can be acquired in a short time.

[0066] (Embodiment 2) In this embodiment, an example is described in which the process of calculating foreground images and joint position candidates is distributed to each photographing device, and a 3D posture model of the subject is calculated in a distributed manner for each photographing area, and finally these are integrated as necessary.

[0067] <Configuration of information processing system> 1 is a configuration diagram of an information processing system in this embodiment. The information processing system includes a first camera group (first camera group) including n camera devices, camera device 1, ..., camera device n, a second camera group (second camera group) including m camera devices, camera device n+1, ..., camera device n+m, and an information processing device 90. The information processing device 90 is a device for estimating a three-dimensional posture model of a subject.

[0068] The hardware configurations of each photographing device and the information processing device 90 are the same as those in the first embodiment, and therefore will not be described.

[0069] <Configuration of imaging device> The configuration of each of the photographing devices 1 to m is similar to the configuration of the photographing device 1 described with reference to FIG.

[0070] Each of the photographing devices of the first camera group and each of the photographing devices of the second camera group are directed toward their respective photographing areas. In this embodiment, since each of the camera groups photographs a different photographing area, each subject is not necessarily captured by the photographing devices of both camera groups. The area photographed by each of the photographing devices of the first camera group in this embodiment is a photographing area 1001 surrounding half of the competition area 1000 shown in FIG. 10. The area photographed by each of the photographing devices of the second camera group is a photographing area 1002. The area photographed by both of the photographing devices of the first camera group and the photographing devices of the second camera group is a photographing area 1003.

[0071] <Configuration of information processing device> The information processing device 90 estimates (generates) 3D shape models of all subjects from foreground images of all the photographing devices. Then, it estimates (generates) 3D pose models of subjects for each camera group in the same manner as in embodiment 1. Furthermore, it integrates 3D pose models for subjects captured by all the photographing devices (subjects present in the photographing area 1003 in the example of FIG. 11) to obtain a final 3D pose model.

[0072] The information processing device 90 includes a first shooting information acquisition unit 910, a first distance image generation unit 920, a first posture model estimation unit 930, a shape model estimation unit 940, a second shooting information acquisition unit 950, a second distance image generation unit 960, a second posture model estimation unit 970, and a posture model integration unit 980.

[0073] The functions of the first photographing information acquisition unit 910 and the second photographing information acquisition unit 950 are similar to those of the photographing information acquisition unit 140 described in the first embodiment. The first photographing information acquisition unit 910 acquires a foreground image indicating the area of ​​the subject and posture information of the subject from each of the photographing devices 1 to n of the first camera group. Furthermore, the first photographing information acquisition unit 910 acquires camera abnormality information from each of the photographing devices 1 to n. The second photographing information acquisition unit 950 acquires a foreground image indicating the area of ​​the subject and posture information of the subject from each of the photographing devices n+1 to m of the second camera group. Furthermore, the second photographing information acquisition unit 950 acquires camera abnormality information from each of the photographing devices n+1 to m.

[0074] The function of the shape model estimation unit 940 is similar to that of the shape model estimation unit 150 described in embodiment 1. The shape model estimation unit 940 estimates (generates) a three-dimensional shape model of the subject based on the foreground images of each of the photographing devices, photographing device 1 to photographing device n and photographing device n+1 to photographing device n+m (i.e., all photographing devices), acquired by the first photographing information acquisition unit 910 and the second photographing information acquisition unit 950, and the camera parameters of each photographing device (all photographing devices).

[0075] The functions of the first distance image generation unit 920 and the second distance image generation unit 960 are similar to those of the distance image generation unit 160 described in the first embodiment. The first distance image generation unit 920 generates a distance image for each imaging device using the 3D shape model of each subject estimated by the shape model estimation unit 940 and the camera parameters of each imaging device of the first camera group, imaging device 1 to imaging device n. The second distance image generation unit 960 generates a distance image for each imaging device using the 3D shape model of each subject estimated by the shape model estimation unit 940 and the camera parameters of each imaging device of the second camera group, imaging device n+1 to imaging device m.

[0076] The functions of first posture model estimation unit 930 and second posture model estimation unit 970 are similar to those of posture model estimation unit 170 described in embodiment 1. First posture model estimation unit 930 estimates (generates) a posture model of a subject included in imaging area 1001 using posture information of a subject captured in an image captured by each imaging device of the first camera group (information on joint position candidates of each joint), a distance image of each imaging device of the first camera group, and camera parameters of each imaging device of the first camera group. Second posture model estimation unit 970 estimates (generates) a posture model of a subject included in imaging area 1002 using posture information of a subject captured in an image captured by each imaging device of the second camera group (information on joint position candidates of each joint), a distance image of each imaging device of the second camera group, and camera parameters of each imaging device of the second camera group.

[0077] Pose model integration section 980 integrates the pose models of each subject estimated by first pose model estimation section 930 and second pose model estimation section 970, and outputs an optimal pose model.

[0078] <Processing of information processing device> Next, the procedure of the process performed by the information processing device 90 according to this embodiment will be described with reference to the flowchart in Fig. 12. Note that the process of each photographing device is the same as in the first embodiment, and therefore the description will be omitted.

[0079] In S1211, the first photographing information acquisition unit 910 acquires the camera parameters of each of the photographing devices 1 to n corresponding to the first camera group. The second photographing information acquisition unit 950 acquires the camera parameters of each of the photographing devices n+1 to m corresponding to the second camera group. The details of the process of S1211 are the same as S611.

[0080] In S1212, the first photographing information acquisition unit 910 acquires, from each of the photographing devices 1 to n of the first camera group photographing the photographing area 1001, a foreground image showing the area of ​​the subject and posture information of the subject.

[0081] In S1213, the second photographing information acquisition unit 950 acquires a foreground image showing the area of ​​the subject and posture information of the subject from each of the photographing devices n+1 to m of the second camera group. The details of the processes of S1212 and S1213 are the same as those of S612.

[0082] In S1214, the shape model estimation unit 940 estimates (generates) a three-dimensional shape model of the subject based on the foreground images of each of the photographing devices (all photographing devices) of the photographing devices 1 to m acquired in S1212 and S1213 and the camera parameters of each of the photographing devices (all photographing devices) of the photographing devices 1 to m acquired in S1211. Details of the generation process in S1214 are the same as those in S613.

[0083] In S1215, the first distance image generation unit 920 projects all voxels that constitute the three-dimensional shape model of the subject estimated in S1214 onto each of the imaging devices 1 to n of the first camera group, thereby generating distance images of each imaging device.

[0084] The first distance image generation unit 920 first selects and acquires the camera parameters of the imaging devices in the first camera group by using an identification ID or the like from the acquired camera parameters of each of the imaging devices, imaging device 1 to imaging device m. The first distance image generation unit 920 also acquires the 3D shape models of all subjects estimated by the shape model estimation unit 940 in S1214. Then, the acquired 3D shape models are projected using the camera parameters of the imaging devices in the first camera group to generate distance images of each imaging device. Details of the method of generating the distance images are the same as those of S614 in the first embodiment.

[0085] In S1216, first posture model estimation unit 930 estimates (generates) a three-dimensional posture model of the subject in imaging area 1001 based on the foreground image of each of the imaging devices 1 to n of the first camera group acquired in S1212 and posture information of the subject (information of candidate joint positions of each joint), the distance image of each of the imaging devices 1 to n of the first camera group generated in S1215, and the camera parameters of each of the imaging devices 1 to n of the first camera group acquired in S1211. Details of the method of estimating the three-dimensional posture model are similar to S615 in the first embodiment.

[0086] In S1217, the second distance image generation unit 960 projects all voxels that constitute the three-dimensional shape model of the subject estimated in S1214 onto each of the imaging devices, n+1 to m, of the second camera group, to generate distance images of each imaging device.

[0087] In S1218, the second posture model estimation unit 970 estimates (generates) a three-dimensional posture model of the subject in the shooting area 1002 based on the foreground images and posture information of the subject (information on candidate joint positions of each joint) of each of the shooting devices of the second camera group, n+1 to m, acquired in S1213, the distance images of each of the shooting devices of the second camera group, n+1 to m, generated in S1217, and the camera parameters of each of the shooting devices of the second camera group, n+1 to m, acquired in S1211.

[0088] In S1219, pose model integration section 980 integrates the 3D pose model of the subject estimated by first pose model estimation section 970 and the 3D pose model of the subject estimated by second pose model estimation section 971 into a single 3D pose model.

[0089] The subject to be integrated is a subject captured by both the photographing devices 1 to n of the first camera group and the photographing devices n+1 to m of the second camera group, that is, a subject existing in the photographing area 1003, which is a photographing area (common area) common to the photographing areas 1001 and 1002 shown in FIG. 11. In order to determine which photographing device the subject is captured in, for example, the barycentric coordinates 810 of the three-dimensional shape model 800 of the subject shown in FIG. 9 is used. The barycentric coordinates 810 expressed in the world coordinate system are converted into the camera image coordinate system using the camera parameters, and if the obtained coordinates fit within the horizontal and vertical sizes of the image, it is determined that the subject is captured in that photographing device. If the subject is captured in both the photographing device of the first camera group and the photographing device of the second camera group, it can be determined that the subject exists in the photographing area 1003.

[0090] If it is determined that the subject exists within shooting area 1003 (i.e., the subject to be subjected to integration processing), the three-dimensional distance between center of gravity coordinates 810 of 3D shape model 800 of the subject and center of gravity coordinates of each of the 3D pose models estimated by first posture model estimation unit 930 and second posture model estimation unit 970 is calculated. Then, the 3D pose model having center of gravity coordinates closest to center of gravity coordinates 810 of 3D shape model 800 of the subject to be subjected to integration processing is selected. That is, one 3D pose model estimated by first posture model estimation unit 930 and one 3D pose model estimated by second posture model estimation unit 970 are selected.

[0091] The new joint position obtained by integrating the two selected three-dimensional posture models is, for example, the average position of the three-dimensional joint position coordinates defined in each three-dimensional posture model. Then, the joint angle is calculated based on the three-dimensional coordinates of the new joint position to create a final one three-dimensional posture model. If there are multiple subjects in the shooting area 1003, the three-dimensional posture models are integrated for each subject. This completes the series of processes in FIG. 12.

[0092] As described above, in this embodiment, the process of calculating the foreground image and joint position candidates is distributed to each image capture device, and the 3D posture model of the subject is calculated separately for each image capture area, and finally these are integrated. This makes it possible to calculate a highly accurate 3D posture model in a short time.

[0093] In this embodiment, an example has been explained using two camera groups, a first camera group and a second camera group; however, the processing of this embodiment can also be similarly applied to a case where there are three or more camera groups, making it possible to integrate three 3D pose models.

[0094] In addition, in this embodiment, a shooting information acquisition unit, a distance image generation unit, and a posture model estimation unit are provided corresponding to each camera group, but as in embodiment 1, it may be configured so that one shooting information acquisition unit, one distance image generation unit, and one posture model estimation unit execute the processing of embodiment 2.

[0095] (Other embodiments) The present disclosure can also be realized by a process in which a program for implementing one or more functions of the above-described embodiments is supplied to a system or device via a network or a storage medium, and one or more processors in a computer of the system or device read and execute the program. It can also be realized by a circuit (e.g., ASIC) that implements one or more functions.

[0096] The present disclosure is not limited to the above-described embodiments, and various modifications and variations are possible without departing from the spirit and scope of the present disclosure. [Explanation of symbols]

[0097] 1,...,n: photographing device, 10: information processing device, 140: photographing information acquisition unit, 150: shape model estimation unit, 160: distance image generation unit, 170: posture model estimation unit

Claims

1. an acquisition means for acquiring a plurality of subject images showing an area of ​​a subject photographed by a plurality of photographing devices and posture information of the subject; a first generation means for generating a three-dimensional shape model of the subject based on a plurality of subject images acquired by the acquisition means and parameters for identifying positions and orientations of the plurality of image capturing devices; a second generating means for generating a plurality of distance images indicating distances from the plurality of image capturing devices to the subject based on the three-dimensional shape model generated by the first generating means; a third generation means for generating a three-dimensional posture model of the subject based on posture information of the subject acquired by the acquisition means, the plurality of distance images generated by the second generation means, and the parameters of the plurality of image capturing devices; 13. An information processing device comprising:

2. The information processing apparatus according to claim 1 , wherein the posture information is information on candidate positions of the joints of the subject in the subject image.

3. The third generating means is acquiring three-dimensional joint position candidates for a plurality of image capture devices using distance values ​​of the distance image corresponding to the joint position candidates; Identifying a three-dimensional joint position of the subject based on a plurality of three-dimensional joint position candidates acquired for a plurality of image capture devices; The information processing apparatus according to claim 2 , wherein the three-dimensional posture model is generated based on identified three-dimensional joint positions.

4. 4 . The information processing apparatus according to claim 3 , wherein the third generating means specifies an average position of the plurality of three-dimensional joint position candidates as the three-dimensional joint position of the subject.

5. 4 . The information processing apparatus according to claim 3 , wherein the third generating means fits an ellipse to the plurality of three-dimensional joint position candidates, and specifies a center position of the ellipse as the three-dimensional joint position of the subject.

6. The apparatus further includes a storage unit for storing the parameters of a plurality of imaging devices, 6. The information processing apparatus according to claim 1, wherein the first generating means uses the parameters of the plurality of image capturing devices that are stored in advance in the storage means.

7. The acquiring means further acquires anomaly information, 7. The information processing apparatus according to claim 1, wherein the first generating means excludes a photographing device identified by the abnormality information in the process of generating the three-dimensional shape model.

8. the plurality of photographing devices include a plurality of photographing devices included in a first photographing device group that photographs a first photographing area and a plurality of photographing devices included in a second photographing device group that photographs a second photographing area, the first photographing area and the second photographing area have a common area that is partially common to each other, the acquiring means acquires a subject image and posture information of the subject from a plurality of camera devices of the first camera device group, and acquires a subject image and posture information of the subject from a plurality of camera devices of the second camera device group; the first generation means generates a three-dimensional shape model of the subject based on a plurality of subject images acquired from all of the camera devices of the first camera device group and the second camera device group and the parameters of the plurality of camera devices; the second generation means generates a distance image from a camera of the first camera group to the subject, and generates a distance image from a camera of the second camera group to the subject, based on the three-dimensional shape model; the third generation means generates a first three-dimensional posture model of the subject based on a plurality of pieces of posture information acquired from the camera devices of the first camera device group, a distance image of the camera devices of the first camera device group generated by the second generation means, and the parameters of the camera devices of the first camera device group, and generates a second three-dimensional posture model of the subject based on a plurality of pieces of posture information acquired from the camera devices of the second camera device group, a distance image of the camera devices of the second camera device group generated by the second generation means, and the parameters of the camera devices of the second camera device group; The information processing device includes: a determination means for determining whether the subject is present in the common area; an integration means for integrating the first 3D pose model and the second 3D pose model when it is determined that the subject is present in the common area; 7. The information processing apparatus according to claim 1, further comprising:

9. A method for controlling an information processing device, comprising: an acquisition step of acquiring a plurality of subject images showing an area of ​​the subject photographed by a plurality of photographing devices and posture information of the subject; a first generation step of generating a three-dimensional shape model of the subject based on a plurality of subject images acquired in the acquisition step and parameters for identifying positions and orientations of the plurality of image capturing devices; a second generation step of generating a plurality of distance images indicating distances from the plurality of image capturing devices to the subject based on the three-dimensional shape model generated by the first generation step; a third generation step of generating a three-dimensional posture model of the subject based on posture information of the subject acquired in the acquisition step, the plurality of distance images generated in the second generation step, and the parameters of the plurality of image capturing devices; 13. A method for controlling an information processing apparatus comprising the steps of:

10. A program for causing a computer to execute the control method according to claim 9.

Images