Information processing device, information processing method, and information processing program

Abstract

To enable generation of three-dimensional skeleton models.SOLUTION: A server device 10 corresponding to an example of an information processing device of the present invention comprises an estimation unit 15B configured to estimate a three-dimensional skeleton based on images captured from multiple viewpoints, and a generation unit 15C for generating a skeleton model obtained by modeling the three-dimensional skeleton.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an information processing device, an information processing method, and an information processing program. [Background technology]

[0002] Motion capture is used in a variety of situations, such as the production, analysis, and editing of computer graphics for sports, games, and other fields.

[0003] For example, motion capture that uses inertial navigation for 3D skeletal estimation is known. In motion capture using inertial navigation, the position of each joint is measured from sensor data acquired by motion sensors attached to the joints. When inertial navigation is used for motion capture, it is unavoidable to wear a motion sensor. For this reason, wearing a motion sensor can interfere with movements in sports, games, and other competitive activities, and can reduce the visual value, such as the sense of realism, of live-action footage.

[0004] In addition, a method called OpenPose has been proposed as a markerless motion capture method that detects two-dimensional pose from images taken with a monocular camera. [Prior art documents] [Patent documents]

[0005] [Patent Document 1] International Publication No. 2019 / 016152 [Non-patent literature]

[0006] [Non-Patent Document 1] Zhe Cao, Tomas Simon, Shih-EnWei, Yaser Sheikh “Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields” Summary of the Invention [Problem to be solved by the invention]

[0007] However, since OpenPose only detects two-dimensional postures, it is difficult to accurately detect skeletal parts hidden by occlusion. Therefore, it can only generate a skeletal model that lacks a sense of three-dimensionality.

[0008] Therefore, an object of the present disclosure is to provide an information processing device, an information processing method, and an information processing program capable of generating a three-dimensional skeletal model. [Means for solving the problem]

[0009] In order to solve the above problems, an information processing device of one embodiment according to the present disclosure includes an estimation unit that estimates a three-dimensional skeleton based on captured images from multiple viewpoints, and a generation unit that generates a skeleton model in which the three-dimensional skeleton is modeled. [Brief description of the drawings]

[0010] [Figure 1] 1 is a diagram illustrating an example of the configuration of a system according to a first embodiment. [Diagram 2] FIG. 13 is a diagram showing an example of a skeleton image. [Diagram 3] FIG. 2 is a diagram showing an example of a three-dimensional skeletal model. [Figure 4] FIG. 11 is a diagram showing an example of a method for associating a skeleton image with a captured image. [Diagram 5] 5 is a flowchart showing the procedure of an image generation process according to the first embodiment. [Figure 6] FIG. 11 is a diagram illustrating an example of the configuration of a system according to a second embodiment. [Figure 7] FIG. 13 is a diagram illustrating an example of a heat map effect. [Figure 8] FIG. 13 is a diagram showing an example of a highlighting effect. [Figure 9] 10 is a flowchart showing the procedure of an image generating process according to a second embodiment. [Figure 10] FIG. 2 is a diagram showing an example of a CG character model. [Figure 11] FIG. 13 is a diagram showing an example of a trajectory of positions of skeletal parts. [Figure 12] FIG. 13 is a diagram showing an example of a trajectory of positions of skeletal parts. [Figure 13] FIG. 2 is a hardware configuration diagram illustrating an example of a computer. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, embodiments of the present disclosure will be described in detail with reference to the drawings. In the following embodiments, the same components are designated by the same reference numerals, and duplicated descriptions will be omitted.

[0012] The present disclosure will be described in the following order. 1. First embodiment 1-1. System configuration example 1-1-1. Server equipment 1-1-2.Camera 1-1-3. Client terminal 1-2. One aspect of the problem-solving approach 1-3. Server device functional configuration example 1-3-1.Communication interface 1-3-2. Storage section 1-3-2-1.Camera parameters 1-3-2-2. 3D skeletal data 1-3-3.Control unit 1-3-3-1. Acquisition part 1-3-3-2.Estimation part 1-3-3-3.Generation part 1-4. Server device processing procedure 1-5. One aspect of effectiveness 2. Second embodiment 2-1. Server device functional configuration example 2-1-1.Settings section 2-2. Server device processing procedure 2-3. One aspect of effectiveness 3. Application Examples 3-1. CG (Computer Graphics) character model 3-2. Linkage with personal identification 3-3. Visualization of bone structure estimation accuracy 3-4. Skeletal Part Trajectory 4. Variations 5. Hardware Configuration

[0013] <<1. First embodiment>> <1-1. System configuration example> Fig. 1 is a diagram showing an example of the configuration of a system 1 according to a first embodiment. The system 1 shown in Fig. 1 provides an image generation service that generates, as computer graphics, a three-dimensional skeletal model in which a three-dimensional skeleton obtained by markerless motion capture is modeled.

[0014] One aspect of such image generation services is that they can be used in a variety of use cases, such as the production, analysis, and editing of computer graphics for sports, games, and other fields.

[0015] Hereinafter, the image captured by the camera 30 may be referred to as the "captured image" in order to distinguish the image captured by the camera 30, i.e., the label of the live-action footage, from the skeleton image including the three-dimensional skeletal model, which is computer graphics.

[0016] As shown in Fig. 1, the system 1 may include a server device 10, cameras 30A to 30N, and a client terminal 50. Hereinafter, when there is no need to distinguish between the individual cameras 30A to 30N, they may be referred to as "camera 30." Note that, although Fig. 1 shows an example in which the system 1 includes one client terminal 50, it is also possible for the system 1 to include multiple client terminals 50.

[0017] The server device 10, the camera 30, and the client terminal 50 may be connected to each other via any network NW. For example, the network NW may be any type of communication network, such as the Internet or a LAN (Local Area Network), regardless of whether it is wired or wireless.

[0018] 1 illustrates an example in which the server device 10 provides a skeleton image including a three-dimensional skeleton model to the client terminal 50 via the network NW, but communication does not necessarily have to be bidirectional between the server device 10 and the client terminal 50. For example, the skeleton image does not necessarily have to pass through the network NW, and may be provided from the server device 10 to the client terminal 50 via broadcast waves.

[0019] <1-1-1. Server device> The server device 10 is a computer that provides the image generation service described above. The server device 10 may correspond to an example of an information processing device.

[0020] In one embodiment, the server device 10 can be implemented by installing an image generation program that realizes a function corresponding to the above image generation service on a desired computer as package software or online software. For example, the server device 10 can be implemented as a server that provides functions related to the above image generation service on-premise, such as a Web server. Without being limited thereto, the server device 10 may be implemented as a SaaS (Software as a Service) type application to provide the above image generation service as a cloud service.

[0021] <1-1-2. Camera> The camera 30 is an example of an imaging device equipped with an imaging element such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

[0022] For example, the cameras 30 are arranged so that the imaging ranges of the cameras 30 are combined to cover the entire three-dimensional space in which a sport or game to be generated by computer graphics is played. Furthermore, from the viewpoint of estimating the three-dimensional skeleton of a subject existing in a three-dimensional space from images captured by two or more cameras 30, each camera 30 can be arranged in a state where the imaging ranges of the cameras 30 partially overlap with each other. Under such an arrangement, the cameras 30 capture images in synchronization with each other for each frame, thereby obtaining a plurality of images captured at the same timing for each different viewpoint, that is, a multi-view captured image, on a frame-by-frame basis. It should be noted that the frame rate at which each camera 30 captures an image is also unified to the same frame rate.

[0023] <1-1-3. Client terminal> The client terminal 50 corresponds to an example of a computer that receives a function corresponding to the image generation service described above. For example, the client terminal 50 may be a desktop computer such as a personal computer. This is merely an example, and the client terminal 50 may be any computer such as a laptop computer, a mobile terminal device, or a wearable terminal.

[0024] 1 shows an example in which the system 1 is constructed as a client-server system including the server device 10 and the client terminal 50, but the configuration example of the system 1 is not limited to this. For example, the function corresponding to the image generation service described above may be realized by a computer that operates as a stand-alone computer.

[0025] <1-2. One aspect of the problem-solving approach> As explained in the Background section above, in motion capture using inertial navigation, it is unavoidable to wear a motion sensor. For this reason, wearing a motion sensor can interfere with movements in sports, games, and other competitive activities, and can reduce the visual value of live-action footage, such as the sense of realism. As such, in motion capture using inertial navigation, markerless motion capture cannot be achieved.

[0026] However, OpenPose only detects 2D postures, so it cannot accurately detect the postures of skeletal parts hidden by occlusion. Therefore, OpenPose can only generate skeletal models that lack a sense of three-dimensionality.

[0027] Even if a depth sensor such as Kinect is used, the above-mentioned occlusion problem cannot be solved. Furthermore, although a technology called monocular 3D pose estimation has emerged that estimates the depth information of joints using machine learning, it is difficult to accurately estimate the depth information of joints because it is a prediction based on machine learning for an indefinite problem.

[0028] Therefore, as part of the image generation service, the server device 10 of the present disclosure estimates a three-dimensional skeleton from images captured from multiple viewpoints. This allows estimation of skeleton parts hidden by occlusion, and allows three-dimensional skeleton estimation to be achieved by markerless motion capture. Then, the server device 10 of the present disclosure generates a three-dimensional skeleton model in which the three-dimensional skeleton estimated in this manner is modeled. Therefore, the server device 10 of the present disclosure makes it possible to generate a skeleton model with a three-dimensional feel.

[0029] <1-3. Example of server device function configuration> Next, a functional configuration example of the server device 10 of the present disclosure will be described. In Fig. 1, blocks corresponding to functions of the server device 10 among the devices included in the system 1 are diagrammed. As shown in Fig. 1, the server device 10 has a communication interface 11, a storage unit 13, and a control unit 15.

[0030] Note that Figure 1 merely shows a selection of functional units related to the above-mentioned image generation service, and does not prevent the server device 10 from being equipped with functional units other than those shown, such as functional units that existing computers are equipped with as default or options.

[0031] <1-3-1. Communication Interface> The communication interface 11 corresponds to an example of a communication control unit that controls communication with other devices, such as the camera 30 and the client terminal 50.

[0032] As just one example, a network interface card such as a LAN card can be adopted as the communication interface 11. For example, the communication interface 11 notifies each camera 30 of an instruction to start capturing an image or an instruction to stop capturing an image, and receives captured images from the camera 30. The communication interface 11 also accepts various setting operations related to viewing of a skeleton image from the client terminal 50, such as setting operations for the viewpoint of a camera including a virtual viewpoint, and outputs a skeleton image.

[0033] <1-3-2. Storage section> The storage unit 13 may correspond to hardware that stores data used by various programs such as the OS (Operating System) executed by the control unit 15 and the image generation program described above.

[0034] As an embodiment, the storage unit 13 may correspond to an auxiliary storage device in the server device 10. For example, an HDD (Hard Disk Drive), an optical disk, an SSD (Solid State Drive), etc. may correspond to the auxiliary storage device. In addition, a flash memory such as an EPROM (Erasable Programmable Read Only Memory) may also correspond to the auxiliary storage device.

[0035] The storage unit 13 stores camera parameters 13A and three-dimensional skeletal data 13B as examples of data used in the program executed by the control unit 15. In addition to the camera parameters 13A and the three-dimensional skeletal data 13B, the storage unit 13 can store various data such as captured images from each viewpoint and account information of the user of the client terminal 50.

[0036] <1-3-2-1.Camera parameters> The camera parameters 13A are data related to camera parameters. As just one example, the camera parameters 13A may include external parameters such as the position and orientation of each camera 30, and internal parameters such as the angle of view and lens distortion of each camera 30.

[0037] <1-3-2-2. 3D skeletal data> The three-dimensional skeleton data 13B is data related to the estimation result of the three-dimensional skeleton. As an example, the three-dimensional skeleton data 13B is obtained by estimating the positions of the joints of the subject in three-dimensional space for each frame of the captured image captured by the camera 30 by an estimation unit 15B described later. In this way, the three-dimensional skeleton data 13B may include, as an example, time-series data of the positions of each joint in three-dimensional space. Furthermore, the three-dimensional skeleton data 13B may include, in addition to the positions in three-dimensional space, the orientation of each joint in three-dimensional space, for example, a joint angle calculated based on inverse kinematics.

[0038] <1-3-3. Control section> The control unit 15 is a processing unit that performs overall control of the server device 10 .

[0039] In one embodiment, the control unit 15 can be implemented by a hardware processor such as a CPU (Central Processing Unit) or an MPU (Micro Processing Unit). Here, a CPU or an MPU is given as an example of a processor, but the control unit 15 can be implemented by any processor, whether general-purpose or specialized. Alternatively, the control unit 15 can be realized by hardwired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

[0040] The control unit 15 virtually realizes the following processing units by expanding the above image generation program on a work area of ​​a RAM (Random Access Memory) implemented as a main storage device (not shown). Note that while Fig. 1 illustrates functional units corresponding to the above image generation program, it is also possible to include functional units corresponding to package software in which a program module corresponding to the above image generation program is packaged with other functions.

[0041] As shown in FIG. 1, the control unit 15 includes an acquisition unit 15A, an estimation unit 15B, and a generation unit 15C.

[0042] <1-3-3-1. Acquisition part> The acquisition unit 15A is a processing unit that acquires captured images from each viewpoint.

[0043] As an embodiment, the acquisition unit 15A can acquire the captured images of each viewpoint transmitted from the cameras 30A to 30N in units of frames. Here, the information source from which the acquisition unit 15A acquires the captured images may be any information source and is not limited to the camera 30. For example, the acquisition unit 15A can acquire the captured images of multiple viewpoints from an auxiliary storage device such as a hard disk or optical disk that stores the captured images of each viewpoint, or a removable medium such as a memory card or a USB (Universal Serial Bus) memory. In addition, the acquisition unit 15A can acquire the captured images of each viewpoint from an external device other than the camera 5 via the network NW.

[0044] <1-3-3-2. Estimation part> The estimation unit 15B is a processing unit that estimates a three-dimensional skeleton based on the captured images from each viewpoint.

[0045] In one embodiment, the estimation unit 15B inputs the captured image into a model obtained by machine learning such as deep learning for each viewpoint of the camera 30, and obtains an output such as a map of the confidence of the position in the captured image for each joint. Then, the estimation unit 15B estimates the position of each joint in three-dimensional space by performing triangulation between the captured images based on the position of the joint obtained for each captured image. Furthermore, the estimation unit 15B can also calculate the posture of each joint in three-dimensional space by calculating the joint angle based on inverse kinematics using the position of each joint in three-dimensional space. The position and posture of each joint in three-dimensional space obtained for each frame of the captured image in this way are stored in the storage unit 13 as three-dimensional skeleton data.

[0046] Note that, although the case where the position and orientation of each joint in a three-dimensional space is estimated using a person as just one example of a subject has been described above, it is also possible to generate a three-dimensional model of an environment other than a person.

[0047] <1-3-3-3. Generation part> The generation unit 15C is a processing unit that generates a skeleton model in which a three-dimensional skeleton is modeled.

[0048] As one aspect, the generating unit 15C renders a skeleton image at the viewpoints of the cameras 30A to 30N or at any virtual viewpoint in the three-dimensional space. For example, the generating unit 15C renders a three-dimensional skeleton model of each subject based on the position of the viewpoint set by the user or the system via the client terminal 50 or the like and the positions of the joints included in the three-dimensional skeleton data 13B.

[0049] FIG. 2 is a diagram showing an example of a skeleton image. As an example, FIG. 2 shows a skeleton image 200 generated from a captured image of a table tennis match. Furthermore, FIG. 2 shows a skeleton image 200 in a case where the position of the center front in the longitudinal direction of the table tennis table is set as the virtual viewpoint Vc. As shown in FIG. 2, three-dimensional skeleton models M1 and M2 corresponding to two players playing a table tennis match are rendered in the skeleton image 200. Furthermore, a three-dimensional model M3 corresponding to a table tennis table is rendered in the skeleton image 200 as an example of a three-dimensional model of an environment other than people. Here, the three-dimensional model M3 of the table tennis table does not necessarily have to be modeling data generated from a captured image. For example, an environmental object whose position and posture do not change in three-dimensional space, such as the three-dimensional model M3, may be modeling data created in advance. Such modeling data can be added to the skeleton image 200. In this way, since a three-dimensional skeletal model is rendered in the skeleton image 200, the ability to express skeletal parts hidden by occlusion is also improved.

[0050] Fig. 3 is a diagram showing an example of a three-dimensional skeletal model. Fig. 3 shows an enlarged view of the three-dimensional skeletal model M2 shown in Fig. 2. As shown in Fig. 3, the three-dimensional skeletal model M2 includes joints J1 to J20 that can correspond to parts such as the head including characteristic points such as eyes and nose, collarbone, left and right shoulders, elbows, wrists, hips, knees, ankles, toes, etc., and bones B1 to B17 connecting the respective joints. When the three-dimensional skeletal model M2 is generated in this way, the joints J1 to J20 are modeled as spherical, while the bones B1 to B17 are modeled as linear or cylindrical.

[0051] Here, the size of the ball of the joints J1 to J20 is set according to the distance between the virtual viewpoint Vc and the joints J1 to J20. For example, the radius of the ball of the joint is set to be larger as the distance from the virtual viewpoint Vc becomes smaller, in other words, as the joint is located closer to the front. On the other hand, the radius of the ball of the joint is set to be smaller as the distance from the virtual viewpoint Vc becomes larger, in other words, as the joint is located further back. As an example of only one aspect, if the joints J4, J7, and J8 are taken as examples, the order of distance from the virtual viewpoint Vc is as follows: joint J8, joint J7, and joint J4. In this case, as shown in FIG. 3, the radius of the ball of the joints J4, J7, and J8 are set so that J8>J7>J4.

[0052] Also, the thickness of the cylinders of bones B1 to B17 is set according to the distance between the virtual viewpoint Vc and bones B1 to B17. For example, the radius of the cylinders of the bones is set to be larger as the distance from the virtual viewpoint Vc becomes smaller, in other words, as the bones are positioned closer to the front. On the other hand, the radius of the cylinders of the bones is set to be smaller as the distance from the virtual viewpoint Vc becomes larger, in other words, as the bones are positioned further back. As an example of only one aspect, if bones B5 and B6 are taken as examples, the bones are arranged in ascending order of distance from the virtual viewpoint Vc, namely, bones B6 and B5. In this case, as shown in FIG. 3, the radius of the cylinders of bones B5 and B6 is set so that B6>B5.

[0053] By modeling the three-dimensional skeletal model M2 including such joints J1 to J20 and bones B1 to B17, the ability to express the joints and bones in the front-back direction, that is, in the depth direction, is improved.

[0054] The viewpoint of the skeleton image is not necessarily limited to the virtual viewpoint given to the virtual camera, but may be any of the viewpoints of the cameras 30A to 30N. Furthermore, the viewpoint of the skeleton image does not necessarily have to be fixed, and can be moved according to a trajectory set by the user via the client terminal 50 or the like or a trajectory of a GUI operation.

[0055] As another aspect, the generation unit 15C can display the three-dimensional skeleton model in association with the captured image of a virtual viewpoint generated using a plurality of captured images, in addition to the captured images of each viewpoint. For example, the generation unit 15C can display the skeleton image and the captured image corresponding to the same frame side by side or vertically, or can project the three-dimensional skeleton model onto the captured image to superimpose the skeleton image on the captured image. In this way, when projecting the three-dimensional skeleton model onto the captured image, the joint angle in addition to the position of each joint is used, thereby improving the ability to express the twist of the joints.

[0056] Fig. 4 is a diagram showing an example of a method for associating a skeleton image with a captured image. As shown in Fig. 4, captured images 40A-40C of three viewpoints of cameras 30A-30C are displayed. On the right side of these captured images 40A-40C, superimposed images 40α-40γ are displayed in which the skeleton images of each viewpoint are superimposed on the captured images 40A-40C by projecting three-dimensional skeleton models M1 and M2 corresponding to each viewpoint onto the captured images 40A-40C. When the skeleton image is superimposed on the captured images in this way, the superimposed images 40α-40γ can be generated after performing image processing for reducing the contrast ratio of the captured images, for example, reducing brightness or saturation, increasing transparency, etc., in order to increase the contrast of the skeleton image compared to the captured images.

[0057] Such display of the superimposed images 40α to 40γ, and therefore the aligned display of the captured images 40A to 40C and the superimposed images 40α to 40γ, makes it possible to easily compare the actual video image with the skeletal model.

[0058] When the 3D skeletal model is occluded by the 3D model of the environment, it is also possible to control whether or not to draw the part of the 3D skeletal model that is hidden by the 3D model of the environment. For example, when the legs of the 3D skeletal model M2 are hidden by the ping-pong table as in the superimposed image 40γ, it is possible to provide a perspective effect by drawing the legs of the 3D skeletal model M2, or to suppress the feeling of incongruity by suppressing the drawing of the legs of the 3D skeletal model M2.

[0059] <1-4. Server device processing procedure> 5 is a flowchart showing the procedure of the image generation process according to the first embodiment. This process is merely an example, and can be performed continuously until acquisition of captured images from the camera 30 is completed.

[0060] 5, the process of step S102 is awaited until the acquisition unit 15A acquires a captured image of each viewpoint (step S101 No). Then, when the acquisition unit 15A acquires a captured image of each viewpoint (step S101 Yes), the estimation unit 15B estimates a three-dimensional skeleton based on the captured image of each viewpoint (step S102).

[0061] Furthermore, the generation unit 15C generates a three-dimensional skeleton model in which the three-dimensional skeleton estimated in step S102 is modeled (step S103). After the process of step S103 is thus performed, the process proceeds to step S101.

[0062] <1-5. One aspect of the effect> As described above, the server device 10 of the present disclosure estimates a three-dimensional skeleton from images captured from multiple viewpoints as part of the image generation service. This allows estimation of skeleton parts hidden by occlusion, and allows three-dimensional skeleton estimation to be achieved by markerless motion capture. Then, the server device 10 of the present disclosure generates a three-dimensional skeleton model in which the three-dimensional skeleton estimated in this manner is modeled. Therefore, the server device 10 of the present disclosure can generate a skeleton model with a three-dimensional feel.

[0063] <<2. Second embodiment>> Various effects can be added to the three-dimensional skeleton model described in the first embodiment. Below, as the second embodiment, the contents of various effects and a method for setting them will be illustrated.

[0064] <2-1. Example of server device function configuration> Fig. 6 is a diagram showing an example of the configuration of a system according to the second embodiment. As shown in Fig. 6, the system 2 differs from the system 1 shown in Fig. 1 in that it includes a server device 20 to which the above-mentioned effect-related functions have been added. Furthermore, the server device 20 differs from the server device 10 shown in Fig. 1 in that the control unit 25 further includes a setting unit 25A.

[0065] <2-1-1.Settings section> The setting unit 25A is a processing unit that sets an effect on the three-dimensional skeletal model.

[0066] As one aspect, the setting unit 25A can set a heat map effect to skeletal parts such as joints and bones of the three-dimensional skeletal model. FIG. 7 is a diagram showing an example of a heat map effect. FIG. 7 shows an example in which a heat map effect is added to the three-dimensional skeletal model M2 shown in FIG. 3. As shown in FIG. 7, heat maps of different colors are added to the joints included in the three-dimensional skeletal model M2 according to the skeletal parts. More specifically, the colors of the heat maps covering the joints are set separately according to the difference in the skeletal parts of the right arm, left arm, right leg, and left leg. For example, a heat map corresponding to a first color, for example, the lattice pattern hatching in the figure, is added to the joints J3, J5, and J6 included in the right arm. In addition, a heat map corresponding to a second color, i.e., the dot hatching in the figure, is added to the joints J4, J7, and J8 included in the left arm. In addition, a heat map corresponding to a third color, i.e., the diagonal hatching in the figure, is added to the joints J13 to J16 included in the right leg. Furthermore, a fourth color, that is, a heat map corresponding to the vertical hatching in the figure, is added to the joints J17 to J20 included in the left leg.

[0067] In this way, by setting heat maps of different colors according to the position of the skeletal part, it is possible to easily identify the parts of the joints. Although Fig. 7 shows an example in which a heat map effect is added to a joint, it goes without saying that a similar effect can be added to a bone. Furthermore, Fig. 7 shows an example in which a heat map of different colors is set according to the position of the skeletal part, it is also possible to set a heat map of different areas according to the position of the skeletal part.

[0068] As another aspect, the setting unit 25A can set an effect for highlighting the display color of skeletal parts such as joints and bones of the three-dimensional skeletal model. As just one example, the setting unit 25A can set an effect for changing color according to the movement of skeletal parts such as joints and bones.

[0069] As an example of an index of the movement of the skeletal part used as a criterion for applying the effect, the setting unit 25A can use the movement amount of the skeletal part. For example, the setting unit 25A can calculate the movement amount of the skeletal part obtained from the position of the skeletal part in the frame in which the skeleton image is being displayed and the position of the skeletal part in the frame before the frame. For example, when the skeleton image of the Nth frame is being displayed, the movement amount of the skeletal part from the N-1th frame to the Nth frame can be calculated. In addition, the setting unit 25A sets the saturation of the skeletal part higher as the movement amount increases, while setting the saturation of the skeletal part lower as the movement amount decreases. In addition, the setting unit 25A can set the color of the range corresponding to the movement amount of the skeletal part among the colors assigned to each range of the movement amount as the display color of the skeletal part. This makes it possible to realize an effect in which the display color is distinguished depending on the magnitude of the movement amount. Note that, although an example of calculating the movement amount of the skeletal part between adjacent frames has been given here, the interval between the frames used for calculating the movement amount does not necessarily have to be adjacent. For example, the amount of movement of a skeletal part between a predetermined number of frames can be calculated.

[0070] In addition, the setting unit 25A can set an effect for highlighting skeletal parts included in the three-dimensional skeletal model whose magnitude of movement amount is equal to or greater than a predetermined threshold value. At this time, in order to highlight skeletal parts whose relative difference in movement is large among the skeletal parts, the setting unit 25A can set an effect for highlighting skeletal parts whose subtracted value of the movement amount is equal to or greater than a predetermined threshold value after subtracting the movement amount of the center of gravity of the entire three-dimensional skeletal model from the movement amount of each skeletal part, such as changing the display color or blinking.

[0071] Fig. 8 is a diagram showing an example of a highlighting effect. In Fig. 8, a highlighting effect is applied to skeletal parts for which the subtraction value obtained by subtracting the movement amount of the center of gravity of the entire three-dimensional skeletal model M2 from the movement amount of the skeletal parts of the joints J1 to J20 and the bones B1 to B17 is equal to or greater than a threshold value. As shown in Fig. 8, among the joints J1 to J20, the display colors of the joints J5, J6, J18, J19, and J20 for which the subtraction value of the movement amount is equal to or greater than the threshold value are set to a different display color from the display colors of the other joints. Furthermore, among the bones B1 to B17, the display colors of the bones B3, B4, B16, and B17 for which the subtraction value of the movement amount is equal to or greater than a threshold value are set to a different display color from the display colors of the other bones.

[0072] In this way, by setting a highlighting effect for skeletal parts where the subtraction value obtained by subtracting the movement amount of the center of gravity of the entire 3D skeletal model M2 from the movement amount of the skeletal part is equal to or greater than a threshold value, it is possible to easily identify skeletal parts that have a relatively large movement amount compared to other parts. For example, in the case of table tennis, by identifying skeletal parts that move vigorously when swinging a racket, it is possible to support the evaluation and analysis of the form.

[0073] In this example, the movement amount of a skeletal part is taken as an example of a movement index, but is not limited thereto. For example, the speed of each skeletal part can be calculated by calculating the change in the movement amount per unit time, for example, per frame, from the movement amount in a predetermined number of frames. Also, the acceleration of each skeletal part can be calculated by calculating the change in the speed per unit time, for example, per frame, from the speed in a predetermined number of frames. These speeds and accelerations can be used to set the above-mentioned highlighting effect.

[0074] Although the example given here shows how highlighting effects are set according to changes in the movement of a skeletal part, it is also possible to set heat map effects of different colors or different areas according to changes in the movement of a skeletal part. Conversely, it is also possible to set highlighting effects according to the position of a skeletal part.

[0075] Furthermore, although an example of changing the display color of the joints and bones has been given here, an effect of changing the size of the joint ball, the thickness of the bone cylinder, or the area of ​​the heat map may be set. For example, the size of the joint ball, the thickness of the bone cylinder, or the area of ​​the heat map corresponding to the skeletal part may be set to be larger as the amount of movement increases, and the size of the joint ball, the thickness of the bone cylinder, or the area of ​​the heat map corresponding to the skeletal part may be set to be smaller as the amount of movement decreases.

[0076] <2-2. Server device processing procedure> 9 is a flowchart showing the procedure of the image generation process according to the second embodiment. This process is merely an example, and can be performed continuously until acquisition of captured images from the camera 30 is completed.

[0077] 9, the process of step S102 is awaited until the acquisition unit 15A acquires a captured image of each viewpoint (step S101 No). Then, when the acquisition unit 15A acquires a captured image of each viewpoint (step S101 Yes), the estimation unit 15B estimates a three-dimensional skeleton based on the captured image of each viewpoint (step S102).

[0078] Then, the setting unit 25A sets the heat map effect and the highlighting effect based on the parts and the movement amount of the three-dimensional skeleton estimated in step S102 (step S201).

[0079] Thereafter, the generation unit 15C generates a three-dimensional skeleton model in which the three-dimensional skeleton estimated in step S102 is modeled (step S103). Then, the generation unit 15C adds the effect set in step S201 to the three-dimensional skeleton model generated in step S103 (step S202). After the process of step S202 is thus performed, the process proceeds to step S101.

[0080] <2-3. One aspect of the effect> As described above, the server device 20 of the present disclosure sets heat map effects and highlighting effects based on the parts and movement amounts of the three-dimensional skeleton as part of the image generation service. Therefore, the server device 20 of the present disclosure can improve the visibility of the skeleton parts depending on the purpose of the video evaluation, analysis, etc.

[0081] <<3. Application Examples>> Application examples of the first and second embodiments will be illustrated below.

[0082] <3-1.CG character model> In the first and second embodiments, an example was given in which a three-dimensional skeletal model was generated and displayed on the client terminal 50, but instead of a three-dimensional skeletal model, a CG character model can also be generated. Fig. 10 is a diagram showing an example of a CG character model. As shown in Fig. 10, a CG character model CG1 corresponding to the three-dimensional skeletal model M1 and a CG character model CG2 corresponding to the three-dimensional skeletal model M2 are rendered in a CG image 210. By displaying these CG character models CG1 and CG character models CG2, it is possible to improve understanding of the situation and the viewing effect.

[0083] It is also possible to display the 3D skeleton model and the CG character model side by side, and to move the viewpoint in tandem. This makes it possible to simultaneously understand the situation (the skeleton model is easy to understand) and check the finished product (the CG character). In addition, by drawing the CG character model from the viewpoint of a real-life camera, for example, the viewpoint of one of the cameras 30A to 30N, it becomes easier to compare it with the real-life model, thereby improving the effectiveness of understanding the situation.

[0084] <3-2. Linkage with personal identification> For example, the server device 10 or 20 identifies an individual by performing authentication processing such as face authentication or iris authentication on a person included in a captured image. Then, the server device 10 or 20 can refer to personal settings of a CG character model or effects and select a CG character model or effects according to the personal settings corresponding to the individual identified as a result of the authentication processing.

[0085] <3-3. Visualization of skeletal estimation accuracy> In the second embodiment, an example is given in which a heat map display is used to identify a skeletal part, but the color or size of the heat map can be changed according to the confidence of the estimation result of the three-dimensional skeleton at the skeletal part. For example, the higher the confidence of the estimation result, the higher the saturation of the color of the heat map of the skeletal part can be, or the area of ​​the heat map of the skeletal part can be, or the lower the confidence of the estimation result, the lower the saturation of the color of the heat map of the skeletal part can be, or the area of ​​the heat map of the skeletal part can be, or the lower the confidence of the estimation result. This makes it possible to visualize the estimation accuracy of the three-dimensional skeleton on the three-dimensional skeleton model.

[0086] <3-4. Trajectory of skeletal parts> In the first and second embodiments, an example was given in which the skeletal parts were modeled by focusing on the frame in which the latest captured image was acquired by the acquisition unit 15A, but it is also possible to model the positions of the skeletal parts across a predetermined number of past frames.

[0087] 11 and 12 are diagrams showing an example of the trajectory of the position of a skeletal part. In FIG. 11, superimposed images 60A-60C are shown in which the positions of skeletal parts in the tNth frame are superimposed on three captured images of the cameras 30A-30C in the tNth frame, which is N frames before the tth frame in which the latest captured image is acquired by the acquisition unit 15A. On the right side of these superimposed images 60A-60C, superimposed images 80A-80C are shown in which the trajectory of the positions of skeletal parts in N frames from the tNth frame to the tth frame are superimposed on the three captured images of the cameras 30A-30C in the tth frame. In addition, FIG. 12 shows a superimposed image 80V in which the trajectory of the positions of skeletal parts in N frames from the tNth frame to the tth frame is superimposed on the captured image of the bird's-eye view (virtual view) in the tth frame. Furthermore, in superimposed images 80A to 80C shown in FIG. 11 and superimposed image 80V shown in FIG. 12, the trajectories of the positions of joints in N frames are superimposed on joints whose subtracted values, obtained by subtracting the amount of movement of the center of gravity of the entire three-dimensional skeletal model in the t-th frame, are equal to or greater than a threshold value, and for other skeletal parts, only the positions of the skeletal parts in the t-th frame are superimposed.

[0088] Here, when the trajectory of the joint position in N frames is superimposed on the superimposed images 80A to 80C or the superimposed image 80V, the radius of the joint ball in past frames other than the t-th frame is set to be smaller than the radius of the joint ball in the t-th frame. That is, for the same joint, the radius of the joint ball in the t-th frame is set to the maximum value, and the radius of the joint ball in the t-th frame is monotonically decreased from the t-th frame to past frames, thereby setting the radius of the joint ball in N frames. In addition, the luminance or saturation of the joint in the t-th frame can be monotonically decreased or the transparency can be monotonically increased from the t-th frame to past frames.

[0089] The superimposed images 80A to 80C and superimposed image 80V generated in this manner can visualize the trajectory of the position of the skeletal parts. Furthermore, since the radius of the ball of the joint in the t-th frame is monotonically decreased going back from the t-th frame to the previous frames, it becomes easier to identify the time series change of the position of the skeletal parts. Note that, although an example of generating the trajectory of the joint position is given here, it goes without saying that the trajectory of the bone position or the trajectory of the heat map position can be generated in a similar manner.

[0090] <<4. Modifications>> Furthermore, among the processes described in the above embodiments, all or part of the processes described as being performed automatically can be performed manually, or all or part of the processes described as being performed manually can be performed automatically by a known method. In addition, the information including the processing procedures, specific names, various data and parameters shown in the above documents and drawings can be changed arbitrarily unless otherwise specified. For example, the various information shown in each drawing is not limited to the illustrated information.

[0091] In addition, each component of each device shown in the figure is a functional concept, and does not necessarily have to be physically configured as shown in the figure. In other words, the specific form of distribution and integration of each device is not limited to that shown in the figure, and all or part of them can be functionally or physically distributed and integrated in any unit according to various loads, usage conditions, etc.

[0092] Furthermore, the effects of each embodiment described in this specification are merely examples and are not limiting, and other effects may also be provided.

[0093] <<5. Hardware Configuration>> The server device 10 according to each of the above-described embodiments is realized by a computer 1000 having a configuration as shown in FIG. 13, for example. The following describes the server device 10 or the server device 20 according to the above-described embodiments as an example. FIG. 13 is a hardware configuration diagram showing an example of the computer 1000. The computer 1000 has a CPU 1100, a RAM 1200, a ROM (Read Only Memory) 1300, a HDD (Hard Disk Drive) 1400, a communication interface 1500, and an input / output interface 1600. The various components of the computer 1000 are connected by a bus 1050.

[0094] The CPU 1100 operates and controls each unit based on programs stored in the ROM 1300 or the HDD 1400. For example, the CPU 1100 loads the programs stored in the ROM 1300 or the HDD 1400 into the RAM 1200 and executes processes corresponding to the various programs.

[0095] The ROM 1300 stores boot programs such as a basic input output system (BIOS) executed by the CPU 1100 when the computer 1000 is started, and programs that depend on the hardware of the computer 1000.

[0096] HDD 1400 is a computer-readable recording medium that non-temporarily records programs executed by CPU 1100 and data used by such programs. Specifically, HDD 1400 is a recording medium that records a development support program according to the present disclosure, which is an example of program data 1450.

[0097] The communication interface 1500 is an interface for connecting the computer 1000 to an external network 1550 (e.g., the Internet). For example, the CPU 1100 receives data from other devices and transmits data generated by the CPU 1100 to other devices via the communication interface 1500.

[0098] The input / output interface 1600 is an interface for connecting the input / output device 1650 and the computer 1000. For example, the CPU 1100 receives data from an input device such as a keyboard or a mouse via the input / output interface 1600. The CPU 1100 also transmits data to an output device such as a display, a speaker, or a printer via the input / output interface 1600. The input / output interface 1600 may also function as a media interface for reading a program or the like recorded on a predetermined recording medium. The medium may be, for example, an optical recording medium such as a DVD (Digital Versatile Disc) or a PD (Phase change rewritable Disk), a magneto-optical recording medium such as an MO (Magneto-Optical disk), a tape medium, a magnetic recording medium, or a semiconductor memory.

[0099] For example, when the computer 1000 functions as the server device 10 or the server device 20 according to the above-described embodiment, the CPU 1100 of the computer 1000 executes an image generation program loaded onto the RAM 1200 to realize each functional unit included in the control unit 15. The image generation program according to the present disclosure and data in the content storage unit 121 are stored in the HDD 1400. The CPU 1100 reads and executes the program data 1450 from the HDD 1400, but as another example, the CPU 1100 may obtain these programs from other devices via an external network 1550.

[0100] The present technology can also be configured as follows. (1) an estimation unit that estimates a three-dimensional skeleton based on images captured from a plurality of viewpoints; A generator for generating a skeleton model in which the three-dimensional skeleton is modeled; An information processing device comprising: (2) The generation unit models the joints of the three-dimensional skeleton as spherical shapes and the bones as cylindrical shapes. The information processing device according to (1). (3) the generation unit sets a radius of the sphere of the joint or a radius of the cylinder of the bone based on a distance between any one of the plurality of viewpoints or a virtual viewpoint and the joint or the bone; The information processing device according to (2). (4) The generation unit sets the radius of the ball of the joint or the radius of the cylinder of the bone to be larger as the distance becomes smaller, or sets the radius of the ball of the joint or the radius of the cylinder of the bone to be smaller as the distance becomes larger. The information processing device according to (3). (5) The skeleton model is displayed in association with the captured images from the multiple viewpoints or virtual viewpoints. The information processing device according to (1). (6) The skeletal model is displayed side by side with a captured image of the same frame as the skeletal model, either vertically or horizontally. The information processing device according to (5). (7) the skeletal model is displayed superimposed on a captured image of the same frame as the skeletal model; The information processing device according to (5). (8) The captured image is displayed with reduced brightness or saturation or increased transparency. The information processing device according to (7). (9) A setting unit that sets an effect on the skeleton model, The information processing device according to (2). (10) The setting unit sets a heat map effect covering the joint or the bone. The information processing device according to (9). (11) The setting unit sets a heat map having a different color or a different area depending on a position of the joint or the bone. The information processing device according to (10). (12) The setting unit sets a heat map having a different color or a different area depending on a movement amount, a speed, or an acceleration of the joint or a skeletal part of the bone. The information processing device according to (10). (13) The setting unit sets a heat map of a different color or a different area depending on a degree of confidence of an estimation result of a three-dimensional skeleton at a skeletal portion of the joint or the bone. The information processing device according to (10). (14) the setting unit sets an effect for highlighting the joint or a skeletal part of the bone. The information processing device according to (9). (15) the setting unit sets different display colors depending on the movement amount, speed, or acceleration of the joint or the skeletal part of the bone. The information processing device according to (14). (16) the setting unit sets an effect for highlighting a skeletal part of the joint or the bone whose movement amount, speed, or acceleration is equal to or greater than a predetermined threshold value; The information processing device according to (14). (17) the setting unit sets an effect for highlighting a skeletal part for which a subtraction value obtained by subtracting a movement amount of a center of gravity of the skeletal model from a movement amount of the skeletal part of the joint or the bone is equal to or greater than a predetermined threshold value. The information processing device according to (16). (18) the generation unit models a trajectory of a position of the joint or a skeletal part of the bone across a predetermined number of past frames from a latest frame in which the captured images are acquired; The information processing device according to (2). (19) The generation unit sets a radius of a joint sphere or a radius of a bone cylinder in a past frame other than the latest frame to a radius smaller than a radius of a joint sphere or a radius of a bone cylinder in the latest frame. The information processing device according to (18). (20) the generation unit sets the radius of the joint sphere or the radius of the bone cylinder in the latest frame as a maximum value, and monotonically decreases the radius of the joint sphere or the radius of the bone cylinder in the latest frame going back from the latest frame to previous frames, thereby setting the radius of the joint sphere or the radius of the bone cylinder in the previous frames; The information processing device according to (19). (twenty one) A 3D skeleton is estimated based on images captured from multiple viewpoints. generating a skeleton model in which the three-dimensional skeleton is modeled; An information processing method in which processing is performed by a computer. (twenty two) A 3D skeleton is estimated based on images captured from multiple viewpoints. generating a skeleton model in which the three-dimensional skeleton is modeled; An information processing program that causes a computer to execute processing. [Explanation of symbols]

[0101] 1 System 10. Server device 11 Communication Interface 13 Storage section 13A Camera parameters 13B 3D skeletal data 15 Control section 15A Acquisition Department 15B Estimation part 15C generation part 30A, 30B, 30N Camera 50 client terminals

Claims

1. an estimation unit that estimates a three-dimensional skeleton based on captured images from multiple viewpoints; a generation unit that generates a skeleton model in which the three-dimensional skeleton is modeled; a setting unit that sets an effect that covers a joint or a bone on the skeletal model; An information processing device comprising:

2. the effect is a heat map covering the joint or the bone; The information processing device according to claim 1 .

3. the setting unit sets a heat map of a different color or a different area depending on the position of the joint or the bone. The information processing device according to claim 2 .

4. the setting unit sets a heat map of a different color or a different area depending on the movement amount, speed, or acceleration of the joint or the skeletal part of the bone. The information processing device according to claim 2 .

5. the setting unit sets a heat map of a different color or a different area depending on a degree of certainty of an estimation result of the three-dimensional skeleton at the skeletal portion of the joint or the bone. The information processing device according to claim 2 .

6. the setting unit sets an effect for highlighting the joint or the skeletal part of the bone. The information processing device according to claim 1 .

7. the setting unit sets different display colors depending on the movement amount, speed, or acceleration of the joint or the skeletal part of the bone. The information processing device according to claim 6 .

8. the setting unit sets an effect for highlighting a skeletal part of the joint or the bone whose movement amount, velocity, or acceleration is equal to or greater than a predetermined threshold. The information processing device according to claim 6 .

9. the setting unit sets an effect for highlighting a skeletal part for which a subtraction value obtained by subtracting a movement amount of the center of gravity of the skeletal model from a movement amount of the skeletal part of the joint or the bone is equal to or greater than a predetermined threshold value. The information processing device according to claim 8 .

10. Estimating a three-dimensional skeleton based on images captured from multiple viewpoints; generating a skeleton model in which the three-dimensional skeleton is modeled; Set effects that cover joints or bones on the skeletal model An information processing method in which processing is performed by a computer.

11. Estimating a three-dimensional skeleton based on images captured from multiple viewpoints; generating a skeleton model in which the three-dimensional skeleton is modeled; Set effects that cover joints or bones on the skeletal model An information processing program that causes a computer to execute a process.

Images