Information processing apparatus, information processing method, and information processing program product

Abstract

According to the present application, a server device (10) corresponding to an example of an information processing device is provided with an estimation unit (15B) that estimates a three-dimensional skeleton based on captured images at a plurality of viewpoints, and a generation unit (15C) that generates a skeleton model obtained by modeling the three-dimensional skeleton.

Description

Technical Field

[0001] This disclosure relates to information processing apparatus, information processing method, and information processing procedure. Background Technology

[0002] Motion capture has been used in a variety of scenarios, such as creating, analyzing, and editing computer graphics commonly used in sports and games.

[0003] For example, motion capture using inertial navigation for 3D skeleton estimation is known. In motion capture using inertial navigation, the position of each joint is measured based on sensor data acquired through motion sensors connected to the joints. In this way, the connection of motion sensors is unavoidable when inertial navigation is used for motion capture. Therefore, the connection of motion sensors hinders operation in sports or gaming competitions, or degrades visual value such as the realism of live-action videos.

[0004] In addition, a label-free motion capture technique called OpenPose proposes to detect two-dimensional pose from monocular camera images.

[0005] Reference List

[0006] Patent documents

[0007] Patent Document 1: WO2019 / 016152A

[0008] Non-patent literature

[0009] Non-Patent Literature 1: Zhe Cao, Tomas Simon, Shih-EnWei, Yaser Sheikh, "Real-time Multi-person 2D Pose Estimation Using Part Affinity Fields" Summary of the Invention

[0010] Technical issues

[0011] However, in the OpenPose example above, because only two-dimensional pose is detected, it is difficult to accurately detect the skeletal parts hidden by occluded objects. Therefore, only a skeletal model lacking a three-dimensional effect can be generated.

[0012] Therefore, the purpose of this disclosure is to provide an information processing apparatus, information processing method, and information processing program capable of generating three-dimensional skeleton models.

[0013] Solution to the problem

[0014] To address the aforementioned problems, an information processing apparatus according to an embodiment of the present disclosure includes: an estimation unit that estimates a three-dimensional skeleton based on images captured from multiple viewpoints; and a generation unit that generates a skeleton model by modeling the three-dimensional skeleton. Attached Figure Description

[0015] Figure 1 This is a diagram illustrating a configuration example of the system according to the first embodiment.

[0016] Figure 2 This is a diagram showing an example of a skeleton image.

[0017] Figure 3 This is a diagram showing an example of a three-dimensional skeleton model.

[0018] Figure 4 This is a diagram illustrating an example of a method for associating a skeleton image with a captured image.

[0019] Figure 5 This is a flowchart illustrating the image generation process according to the first embodiment.

[0020] Figure 6 This is a diagram illustrating an example of a system configuration according to the second embodiment.

[0021] Figure 7 This is an example diagram showing the effect of a heatmap.

[0022] Figure 8 This is a diagram showing an example of the highlighting effect.

[0023] Figure 9 This is a flowchart illustrating the steps of the image generation process according to the second embodiment.

[0024] Figure 10 This is an image showing an example of a CG character model.

[0025] Figure 11 This is a diagram illustrating an example of the trajectory showing the location of a skeletal section.

[0026] Figure 12 This is a diagram illustrating an example of the trajectory showing the location of a skeletal section.

[0027] Figure 13 This is a diagram illustrating an example of a computer's hardware configuration. Detailed Implementation

[0028] In the following description, embodiments of the present disclosure will be illustrated with reference to the accompanying drawings. In each of the following embodiments, the same parts are given the same reference numerals to omit redundant descriptions.

[0029] This disclosure will be described in the following order of items.

[0030] 1. First Embodiment

[0031] 1-1. System Configuration Example

[0032] 1-1-1. Server Equipment

[0033] 1-1-2. Camera

[0034] 1-1-3. Client Terminal

[0035] 1-2. One aspect of problem-solving methods

[0036] 1-3. Example of server device function configuration

[0037] 1-3-1. Communication Interface

[0038] 1-3-2. Storage Unit

[0039] 1-3-2-1. Camera Parameters

[0040] 1-3-2-2. Three-dimensional skeleton data

[0041] 1-3-3. Control Unit

[0042] 1-3-3-1. Acquisition Unit

[0043] 1-3-3-2. Estimating Unit

[0044] 1-3-3-3. Generation Unit

[0045] 1-4. Server equipment processing procedures

[0046] 1-5. One aspect of the effect

[0047] 2. Second Embodiment

[0048] 2-1. Example of server device function configuration

[0049] 2-1-1. Setting Unit

[0050] 2-2. Server Equipment Processing Procedures

[0051] 2-3. One aspect of the effect

[0052] 3. Application Examples

[0053] 3-1. Computer Graphics (CG) Character Models

[0054] 3-2. Integration with Personal Identification

[0055] 3-3. Visualization of skeleton estimation accuracy

[0056] 3-4. Trajectory of the skeleton

[0057] 4. Variation Example

[0058] 5. Hardware Configuration

[0059] <<1. First Embodiment>>

[0060] <1-1. System Configuration Example>

[0061] Figure 1 This is a diagram illustrating a configuration example of system 1 according to the first embodiment. Figure 1 System 1 shown provides an image generation service for generating a 3D skeleton model as computer graphics, in which a 3D skeleton obtained through markerless motion capture is modeled.

[0062] As one aspect, image generation services can be used for a variety of use cases, such as creating, analyzing, and editing computer graphics commonly used in sports and games.

[0063] In the following text, from the point of view of distinguishing the image captured by camera 30 (i.e., the real-world video tag) from the skeleton image (i.e., computer graphics) which includes a three-dimensional skeleton model, the image captured by camera 30 can be referred to as the "captured image".

[0064] like Figure 1 As shown, system 1 may include server device 10, cameras 30A to 30N, and client terminal 50. In the following text, when it is not necessary to distinguish each of cameras 30A to 30N, cameras 30A to 30N may be referred to as "camera 30". Although Figure 1 An example of a system 1 including one client terminal 50 is shown, but it may include multiple client terminals 50.

[0065] Server device 10, camera 30, and client terminal 50 can be connected via any network NW. For example, the network NW can be any type of communication network, such as the Internet or a local area network (LAN), regardless of whether it is wired or wireless.

[0066] Notice, Figure 1 This illustration shows a scenario where server device 10 provides a skeleton image, including a 3D skeleton model, to client terminal 50 via network NW. However, server device 10 and client terminal 50 do not necessarily communicate bidirectionally with each other. For example, the skeleton image may not necessarily be provided via network NW, but could be provided from server device 10 to client terminal 50 via broadcast.

[0067] <1-1-1. Server Equipment>

[0068] Server device 10 is a computer that provides the aforementioned image generation service. Server device 10 may correspond to an example of an information processing device.

[0069] As an example, server device 10 can be implemented by installing an image generation program on a desired computer, which implements the functions corresponding to the image generation service as packaged software or online software. For example, server device 10 can be implemented as a web server that provides the functions related to the image generation service on-premises. This disclosure is not limited thereto, and server device 10 can provide the image generation service as a cloud service by being implemented as a Software as a Service (SaaS) type application.

[0070] <1-1-2. Camera>

[0071] Camera 30 is an example of an imaging device equipped with imaging elements such as charge-coupled devices (CCD) or complementary metal-oxide-semiconductor (CMOS).

[0072] For example, by combining the shooting ranges of multiple cameras 30, each camera 30 is arranged such that the entire area of ​​the three-dimensional space of the generated object (e.g., sports or game competition) falls within the shooting range of the multiple cameras 30. Furthermore, from the perspective of estimating the three-dimensional skeleton of the subject present in three-dimensional space based on the captured images captured by two or more cameras 30, each camera 30 can be arranged such that a portion of its shooting range overlaps with the shooting ranges of other cameras 30. Using this arrangement, the multiple cameras 30 capture images synchronously for each frame, allowing multiple images captured from different viewpoints at the same timing to be obtained frame by frame—i.e., multi-viewpoint image capture. Note, of course, that the frame rate at which the cameras 30 capture images is also uniformly matched to the same frame rate.

[0073] <1-1-3. Client Terminal>

[0074] Client terminal 50 corresponds to an example of a computer that receives functions corresponding to the image generation service described above. For example, a desktop computer, such as a personal computer, can correspond to client terminal 50. This is merely an example, and client terminal 50 can be any computer, such as a laptop computer, mobile terminal device, or wearable terminal.

[0075] Notice, Figure 1An example of a client-server system, comprising server device 10 and client terminal 50, is shown. However, the configuration example of System 1 is not limited to this. For example, the functions corresponding to the image generation service described above can be implemented by a computer operating independently.

[0076] <1-2. One aspect of problem-solving methods>

[0077] As described in the background section above, the connection of motion sensors is unavoidable in motion capture using inertial navigation. Therefore, the connection of motion sensors hinders operations such as in sports or gaming competitions, or degrades visual value such as the realism of live-action videos. Consequently, markerless motion capture cannot be achieved in motion capture using inertial navigation.

[0078] However, OpenPose only detects 2D pose, making it impossible to accurately detect the pose of skeletal parts hidden by occlusion. Therefore, OpenPose can only generate skeletal models lacking 3D depth.

[0079] Furthermore, even combining depth sensors such as Kinect cannot solve the aforementioned occlusion problem. In addition, techniques for estimating joint depth information using machine learning have been developed, known as monocular 3D pose estimation. However, because the estimation is performed on uncertainties through machine learning, it is difficult to accurately estimate joint depth information.

[0080] Therefore, as part of the aforementioned image generation service, the server device 10 of this disclosure estimates a three-dimensional skeleton using images captured from multiple viewpoints. Thus, skeleton portions hidden by occlusions can be estimated, and three-dimensional skeleton estimation can also be achieved through markerless motion capture. The server device 10 of this disclosure then generates a three-dimensional skeleton model in which the three-dimensional skeleton estimated in this manner is modeled. Therefore, according to the server device 10 of this disclosure, a stereo skeleton model can be generated.

[0081] <1-3. Example of Server Device Functional Configuration>

[0082] Next, a functional configuration example of the server device 10 disclosed herein will be described. Figure 1 The diagram schematically illustrates blocks corresponding to the functions of server device 10 included in system 1. For example... Figure 1 As shown, the server device 10 includes a communication interface 11, a storage unit 13, and a control unit 15.

[0083] Notice, Figure 1Only the functional units related to the above-described image generation service are shown, and this does not preclude the provision of functional units other than those shown in the server device 10, such as functional units that are provided by default or optionally in existing computers.

[0084] <1-3-1. Communication Interface>

[0085] Communication interface 11 corresponds to an example of a communication control unit that performs communication control with other devices such as camera 30 or client terminal 50.

[0086] As an example, a network interface card, such as a LAN card, can be used as the communication interface 11. For example, the communication interface 11 notifies each camera 30 of the imaging start and imaging end commands, or receives captured images from the camera 30. In addition, the communication interface 11 receives various setting operations (e.g., camera viewpoint setting operations such as virtual viewpoints) from the client terminal 50 regarding viewing the skeleton image, and outputs the skeleton image.

[0087] <1-3-2. Storage Unit>

[0088] Storage unit 13 may correspond to hardware that stores data for various programs, such as the operating system (OS) executed by control unit 15 and the aforementioned image generation program.

[0089] As one embodiment, storage unit 13 may correspond to an auxiliary storage device in server device 10. For example, hard disk drives (HDDs), optical disks, solid-state drives (SSDs) and the like correspond to auxiliary storage devices. In addition, flash memory such as erasable programmable read-only memory (EPROM) may also correspond to auxiliary storage devices.

[0090] Storage unit 13 stores camera parameters 13A and 3D skeleton data 13B as examples of data for a program executed by control unit 15. In addition to camera parameters 13A and 3D skeleton data 13B, storage unit 13 can also store various types of data, such as captured images from each viewpoint and user account information of client terminal 50.

[0091] <1-3-2-1. Camera Parameters>

[0092] Camera parameters 13A are data related to camera parameters. As an example, camera parameters 13A may include external parameters (such as the position and orientation of each camera 30) and internal parameters (such as the angle of view and lens distortion of each camera 30).

[0093] <1-3-2-2. Three-dimensional skeleton data>

[0094] The 3D skeleton data 13B is data related to the estimation results of the 3D skeleton. As an example, in the 3D skeleton data 13B, the estimation unit 15B, described later, estimates the positions of the joints of the subject in 3D space for each frame of the captured image captured by the camera 30. As described above, as an example, the 3D skeleton data 13B may include time-series data of the position of each joint in 3D space. Furthermore, in addition to the position in 3D space, the 3D skeleton data 13B may also include the pose of each joint in 3D space, such as joint angles calculated based on inverse kinematics.

[0095] <1-3-3. Control Unit>

[0096] The control unit 15 is the processor that controls the entire server device 10.

[0097] As one embodiment, the control unit 15 can be implemented using a hardware processor such as a central processing unit (CPU) or a microprocessor unit (MPU). Here, CPU and MPU are given as examples of processors, but a processor can be implemented using any processor, whether general-purpose or special-purpose. Furthermore, the control unit 15 can be implemented using hard-wired logic, such as application-specific integrated circuits (ASICs) or field-programmable gate arrays (FPGAs).

[0098] The control unit 15 virtually implements the following processing unit by expanding the aforementioned image generation program on the working area of ​​a random access memory (RAM) installed as a main storage device (not shown). Note that, although Figure 1 The diagram shows functional units corresponding to the image generation program described above, but may include functional units corresponding to packaging software, in which the program module corresponding to the image generation program described above is packaged together with other functions.

[0099] like Figure 1 As shown, the control unit 15 includes an acquisition unit 15A, an estimation unit 15B, and a generation unit 15C.

[0100] <1-3-3-1. Acquiring Units>

[0101] The acquisition unit 15A is a processing unit that acquires captured images from each viewpoint.

[0102] As one embodiment, the acquisition unit 15A can acquire captured images from various viewpoints transmitted from cameras 30A to 30N in frames. Here, the information source from which the acquisition unit 15A acquires the captured images can be any information source, and is not limited to camera 30. For example, the acquisition unit 15A can also acquire multi-view captured images from an auxiliary storage device (e.g., a hard disk or optical disc) or a removable medium (e.g., a memory card or Universal Serial Bus (USB) memory) that accumulates captured images from various viewpoints. Furthermore, the acquisition unit 15A can also acquire captured images from each viewpoint from an external device other than camera 5 via network NW.

[0103] <1-3-3-2. Estimation Unit>

[0104] The estimation unit 15B is a processing unit that estimates the 3D skeleton based on the captured images from each viewpoint.

[0105] As one embodiment, estimation unit 15B inputs the captured images into a model obtained through machine learning, such as deep learning, for each viewpoint of camera 30, thereby obtaining an output such as a deterministic map of the position of each joint in the captured images. Then, estimation unit 15B estimates the position of each joint in three-dimensional space by performing triangulation between captured images based on the joint positions obtained for each captured image. Furthermore, estimation unit 15B can calculate the pose of each joint in three-dimensional space by calculating joint angles based on inverse kinematics using the position of each joint in three-dimensional space. As described above, the position and pose of each joint in three-dimensional space obtained for each frame of the captured images are stored as three-dimensional skeleton data in storage unit 13.

[0106] Note that this example uses only a human as the subject to estimate the position and orientation of each joint in 3D space. It is also possible to generate 3D models of the environment other than the human.

[0107] <1-3-3-3. Generating Units>

[0108] Generation unit 15C is the processing unit for generating the skeleton model, in which the three-dimensional skeleton is modeled.

[0109] As one aspect, the generation unit 15C renders skeleton images from the viewpoints of cameras 30A to 30N or from any virtual viewpoint in three-dimensional space. For example, the generation unit 15C renders a three-dimensional skeleton model of a subject for each subject based on the viewpoint position set by the user or system through the client terminal 50 and the positions of the joints included in the three-dimensional skeleton data 13B.

[0110] Figure 2This is an example diagram showing a skeleton image. As an example, Figure 2 A skeleton image 200 generated from captured images of a table tennis match is shown. Furthermore, Figure 2 A skeleton image 200 is shown with the central frontal position in the longitudinal direction of the ping-pong table set as a virtual viewpoint Vc. (See image 200.) Figure 2 As shown, 3D skeleton models M1 and M2 corresponding to two athletes playing a table tennis match are presented in skeleton image 200. Furthermore, as an example of a 3D model of the environment other than the athletes, a 3D model M3 corresponding to the table tennis table is presented in skeleton image 200. Here, the 3D model M3 of the table tennis table is not necessarily modeling data generated from the captured image. For example, environmental objects whose position or pose does not change in 3D space (e.g., 3D model M3) can be pre-created modeling data. Such modeling data can be added to skeleton image 200. As described above, since 3D skeleton models are presented in skeleton image 200, the ability to express skeletal parts hidden by occlusion is also improved.

[0111] Figure 3 This is a diagram showing an example of a three-dimensional skeleton model. Figure 3 Shown in an enlarged manner Figure 2 The three-dimensional skeleton model M2 is shown in the image. Figure 3 As shown, the 3D skeleton model M2 includes joints J1 to J20 and bones B1 to B17 connecting the joints. Joints J1 to J20 correspond to the head, which includes feature points such as eyes and nose, and body parts such as collarbone, left and right shoulders, elbows, wrists, waist, knees, ankles, and toes. When the 3D skeleton model M2 is generated in this way, joints J1 to J20 are modeled as spheres, while bones B1 to B17 are modeled as linear or cylindrical shapes.

[0112] Here, the size of the spheres representing joints J1 to J20 is set based on the distance between the virtual viewpoint Vc and joints J1 to J20. For example, the radius of the joint spheres is set to increase as the distance from the virtual viewpoint Vc decreases. In other words, the joint spheres are set to be larger as the joint is closer to the virtual viewpoint Vc. On the other hand, the radius of the joint spheres is set to be smaller as the distance from the virtual viewpoint Vc increases, i.e., as the joint is further away from the virtual viewpoint Vc. As an example, considering joints J4, J7, and J8, the order of shorter distances from the virtual viewpoint Vc is joint J8, joint J7, and joint J4. In this case, as... Figure 3 As shown, the radius of each sphere in joints J4, J7, and J8 is set to satisfy J8>J7>J4.

[0113] Furthermore, the size of the cylinders representing bones B1 to B17 is set based on the distance between the virtual viewpoint Vc and bones B1 to B17. For example, as the distance to the virtual viewpoint Vc decreases, i.e., on the closer side, the larger the radius of the bone cylinder is set. Conversely, as the distance to the virtual viewpoint Vc increases, i.e., on the farther side, the smaller the radius of the bone cylinder is set. As an example, taking bones B5 and B6 as examples, the order of shorter distances from the virtual viewpoint Vc is bone B6 followed by bone B5. In this case, as... Figure 3 As shown, the radius of each cylinder of bones B5 and B6 is set to satisfy B6>B5.

[0114] By modeling a 3D skeletal model M2 that includes joints J1 to J20 and bones B1 to B17, the ability to represent joints and bones in the anterior-posterior direction (i.e., the depth direction) is improved.

[0115] Note that the viewpoint of the skeleton image is not necessarily limited to the virtual viewpoint provided to the virtual camera, and can be any viewpoint from camera 30A to camera 30N. Furthermore, the viewpoint of the skeleton image is not necessarily fixed, and can move according to user-defined trajectories or GUI operations on the client terminal 50, etc.

[0116] On the other hand, in addition to the captured images from each viewpoint, the generation unit 15C can also display a 3D skeleton model associated with the captured images from the virtual viewpoint generated using multiple captured images. For example, the generation unit 15C can display the skeleton image and the captured image corresponding to the same frame side by side, either horizontally or vertically, or it can display a skeleton image superimposed on the captured image by projecting the 3D skeleton model onto the captured image. When the 3D skeleton model is projected onto the captured image in this way, joint angles are used in addition to the position of each joint, thereby improving the ability to express twisted joints.

[0117] Figure 4 This is a diagram illustrating an example of a method for associating a skeleton image with a captured image. (Example:) Figure 4 As shown, captured images 40A to 40C from three viewpoints of cameras 30A to 30C are displayed. An overlay image 40, showing the skeleton images from each viewpoint superimposed on the captured images 40A to 40C, is displayed by projecting 3D skeleton models M1 and M2 corresponding to each viewpoint onto the captured images 40A to 40C on the right side of the captured images 40A to 40C. up to 40 In this case, when the skeleton image is superimposed on the captured image, from the perspective of increasing the contrast of the skeleton image compared to the captured image, the superimposed image can be generated after performing image processing to reduce the contrast of the captured image (e.g., reducing brightness and chroma, increasing transparency, etc.). up to 40 .

[0118] Display the overlay image 40 as described above up to 40 Alternatively, display captured images 40A to 40C and overlaid image 40 side-by-side. up to 40 It can facilitate comparisons between live-action videos and skeleton models.

[0119] Note that when a 3D skeleton model is occluded by a 3D model of the environment, it's possible to control whether the parts hidden by the environment's 3D model are drawn within the 3D skeleton model. For example, when the legs in the 3D skeleton model M2 are overlaid with image 40... When the ping-pong table is hidden, perspective can be provided by drawing the legs in the 3D skeleton model M2, or discomfort can be suppressed by inhibiting the drawing of the legs in the 3D skeleton model M2.

[0120] <1-4. Server Equipment Processing Procedures>

[0121] Figure 5 This is a flowchart illustrating the image generation process according to the first embodiment. As an example, this process can be performed continuously until the captured image from camera 30 is completed.

[0122] like Figure 5 As shown, before the acquisition unit 15A acquires the captured image of each viewpoint (step S101, no), the processing in step S102 is paused. Then, when the acquisition unit 15A acquires the captured image from each viewpoint (step S101, yes), the estimation unit 15B estimates the 3D skeleton based on the captured image from each viewpoint (step S102).

[0123] Furthermore, generation unit 15C generates a three-dimensional skeleton model, in which the three-dimensional skeleton estimated in step S102 is modeled (step S103). After performing the processing in step S103 in this manner, the processing proceeds to step S101.

[0124] <1-5. One aspect of the effect>

[0125] As described above, as part of the image generation service, the server device 10 of this disclosure estimates a 3D skeleton from multi-view captured images. Therefore, skeletal portions hidden by occlusions can be estimated, and 3D skeleton estimation can also be achieved through markerless motion capture. The server device 10 then generates a 3D skeleton model in which the 3D skeleton estimated in this manner is modeled. Therefore, a stereo skeleton model can be generated according to the server device 10 of this disclosure.

[0126] <<2. Second Embodiment>>

[0127] Various effects can be added to the 3D skeleton model described in the first embodiment. In the following, as a second embodiment, the content of these various effects and their setting methods will be described.

[0128] <2-1. Example of Server Device Functional Configuration>

[0129] Figure 6 This is a diagram illustrating an example of a system configuration according to the second embodiment. (See diagram for example.) Figure 6 As shown, System 2 and Figure 1 The difference between System 1 and System 2 shown is that System 2 includes a server device 20 with added functionality related to the aforementioned effects. Furthermore, server device 20 and... Figure 1 The difference in the server device 10 shown is that the control unit 25 also includes a setting unit 25A.

[0130] <2-1-1. Setting Unit>

[0131] Setting unit 25A is a processing unit for setting effects on the 3D skeleton model.

[0132] As one aspect, the setting unit 25A can set heat map effects for the skeletal parts (such as joints or bones) of a 3D skeleton model. Figure 7 This is an example diagram showing the effect of a heatmap. Figure 7 This demonstrates adding heatmap effects. Figure 3 The example shown is a 3D skeleton model M2. Figure 7As shown, heatmaps of different colors, depending on the skeletal portion, are added to the joints included in the 3D skeletal model M2. More specifically, the colors of the heatmaps covering the joints are set to differentiate based on the differences in the skeletal portions of the right arm, left arm, right leg, and left leg. For example, a heatmap of the first color, corresponding to the shading of the grid pattern in the figure, is added to joints J3, J5, and J6 included in the right arm. Furthermore, a heatmap of the second color, corresponding to the shading of the dotted line in the figure, is added to joints J4, J7, and J8 included in the left arm. Furthermore, a heatmap of the third color, corresponding to the shading of the diagonal line in the figure, is added to joints J13 to J16 included in the right leg. Furthermore, a heatmap of the fourth color, corresponding to the shading of the vertical line in the figure, is added to joints J17 to J20 included in the left leg.

[0133] By using heatmaps with different colors based on the location of skeletal parts in this way, joints can be easily identified. Note that, although... Figure 7 An example of adding a heatmap effect to a joint is shown, but similar effects can certainly be added to bones. Furthermore, Figure 7 An example is shown where heatmaps with different colors are set based on the location of the skeleton parts. However, heatmaps with different sizes can be set based on the location of the skeleton parts.

[0134] On the other hand, the setting unit 25A can set a highlighting effect for the skeletal parts (such as joints or bones) of a 3D skeleton model. As an example, the setting unit 25A can set a color change effect corresponding to the movement of skeletal parts such as joints or bones.

[0135] As an example of an indicator of the movement of the skeleton portion as a reference for adding effects, the setting unit 25A can use the amount of movement of the skeleton portion. For example, the setting unit 25A can calculate the amount of movement of the skeleton portion based on its position in the frame where the skeleton image is being displayed and its position in the frames preceding the displayed frame. For example, assuming the skeleton image of frame N is being displayed, the amount of movement of the skeleton portion from frame (N-1) to frame N can be calculated. Then, the setting unit 25A sets a higher chromaticity to the skeleton portion as the amount of movement increases and a lower chromaticity as the amount of movement decreases. Furthermore, the setting unit 25A can also set the color of the range corresponding to the amount of movement of the skeleton portion as the display color of the applicable skeleton portion, according to the color assigned to each range of movement. As a result, the effect of distinguishing the magnitude of the movement by display color can be achieved. Note that an example of calculating the amount of movement of the skeleton portion between adjacent frames has been described here. However, the frame interval used to calculate the amount of movement is not necessarily adjacent frames. For example, the movement of the skeleton portion can also be calculated between a predetermined number of frames.

[0136] Furthermore, the setting unit 25A can also set a highlighting effect for skeleton parts in the 3D skeleton model whose movement is equal to or greater than a predetermined threshold. In this case, from the perspective of emphasizing skeleton parts with large relative movement differences, the setting unit 25A can set a highlighting effect (such as different display colors and flashing display) for skeleton parts whose subtraction value of movement is equal to or greater than the predetermined threshold. The subtraction value is calculated by subtracting the movement of the center of gravity of the entire 3D skeleton model from the movement of each skeleton part.

[0137] Figure 8 This is an example diagram showing the highlighting effect. In Figure 8 In this context, a highlighting effect is added to skeletal portions where the subtraction value is equal to or greater than a threshold. This subtraction value is obtained by subtracting the shift of the center of gravity of the entire 3D skeleton model M2 from the shift of the skeletal portions from joints J1 to J20 and bones B1 to B17. For example... Figure 8 As shown, among joints J1 to J20, joints J5, J6, J18, J19, and J20, whose subtraction value of movement is equal to or greater than a threshold, are displayed in a color different from the display colors of the other joints. Furthermore, among bones B1 to B17, bones B3, B4, B16, and B17, whose subtraction value of movement is equal to or greater than a threshold, are displayed in a color different from the display colors of the other bones.

[0138] In this way, by highlighting skeleton parts whose subtraction value (obtained by subtracting the movement of the center of gravity of the entire 3D skeleton model M2 from the movement of the skeleton parts) is equal to or greater than a threshold, skeleton parts with relatively large movements can be easily identified compared to other parts. For example, in the table tennis example, the evaluation and analysis of playing posture can be supported by identifying skeleton parts that move violently when swinging the racket.

[0139] Note that the amount of movement of the skeleton parts here is only described as an example of a motion metric. However, motion metrics are not limited to this. For example, the velocity of each skeleton part can be calculated by measuring the change in the amount of movement per unit time (such as per frame) based on the amount of movement in a predetermined number of frames. Furthermore, the acceleration of each skeleton part can be calculated by measuring the change in velocity per unit time (such as per frame) based on the velocity in a predetermined number of frames. These velocities and accelerations can be used to set the aforementioned highlighting effect.

[0140] Furthermore, an example is given here of setting a highlighting effect based on changes in the movement of the skeleton parts. However, it is also possible to set heatmap effects with different colors or sizes based on changes in the movement of the skeleton parts. Conversely, a highlighting effect can also be set based on the position of the skeleton parts.

[0141] Furthermore, although examples of changing the display color of joints and bones have been described, the effects of changing the size of joint spheres, the thickness of bone cylinders, or the size of heatmaps can be configured. For example, the size of joint spheres corresponding to skeletal parts, the thickness of bone cylinders, or the size of heatmaps can be set to increase with increasing movement. Alternatively, the size of joint spheres corresponding to skeletal parts, the thickness of bone cylinders, or the area of ​​heatmaps can be set to decrease with decreasing movement.

[0142] <2-2. Server Equipment Processing Procedures>

[0143] Figure 9 This is a flowchart illustrating the image generation process according to the second embodiment. As an example, this process can be performed continuously until the captured image from camera 30 is completed.

[0144] like Figure 9 As shown, before the acquisition unit 15A acquires the captured image of each viewpoint (step S101, no), the processing in step S102 is paused. Then, when the acquisition unit 15A acquires the captured image from each viewpoint (step S101, yes), the estimation unit 15B estimates the 3D skeleton based on the captured image from each viewpoint (step S102).

[0145] Then, the setting unit 25A sets the heat map effect or highlighting effect based on the three-dimensional skeleton part and the amount of movement estimated in step S102 (step S201).

[0146] Subsequently, generation unit 15C generates a three-dimensional skeleton model, in which the three-dimensional skeleton estimated in step S102 is modeled (step S103). Then, generation unit 15C adds the effect set in step S201 to the three-dimensional skeleton model generated in step S103 (step S202). After performing the processing in step S202 in this manner, the processing proceeds to the processing in step S101.

[0147] <2-3. One aspect of the effect>

[0148] As described above, as part of the image generation service, the server device 20 of this disclosure sets heatmap effects and highlighting effects based on the three-dimensional skeleton and the amount of movement. Therefore, according to the server device 20 of this disclosure, the visibility of the skeleton can be enhanced for purposes such as video evaluation and analysis.

[0149] <<3. Application Examples>>

[0150] The following will describe application examples of the first and second embodiments.

[0151] <3-1. CG Character Models>

[0152] The first and second embodiments provide examples of generating a three-dimensional skeleton model and displaying the model on a client terminal 50. However, instead of a three-dimensional skeleton model, a CG character model can be generated. Figure 10 This is an image showing an example of a CG character model. (For example...) Figure 10 As shown, CG character model CG1, corresponding to the 3D skeleton model M1, and CG character model CG2, corresponding to the 3D skeleton model M2, are presented in CG image 210. The display of CG character model CG1 and CG character model CG2 enhances the understanding and viewing experience of the situation.

[0153] Furthermore, 3D skeleton models and CG character models can be displayed side-by-side, and viewpoint movement can be synchronized. This allows for both understanding the situation (the skeleton model is easy to understand) and confirming completion (the CG character). Additionally, drawing CG character models from the viewpoint of a live-action camera (e.g., any viewpoint from camera 30A to 30N) facilitates comparison with the live-action footage, thereby enhancing the understanding of the situation.

[0154] <3-2. Integration with Personal Identification>

[0155] For example, server device 10 or server device 20 identifies an individual by performing authentication processes such as facial authentication and iris authentication on the people included in the captured image. Then, server device 10 or server device 20 can refer to the individual settings of the CG character model or effect, and select a CG character model or effect based on the individual settings corresponding to the individual identified as a result of the authentication process.

[0156] <3-3. Visualization of skeleton estimation accuracy>

[0157] The second embodiment provides an example of using heatmap display to identify skeleton parts. The color or size of the heatmap can be changed based on the certainty factor of the estimation result of the 3D skeleton in the skeleton part. For example, as the certainty of the estimation result increases, the chromaticity of the heatmap color of the skeleton part can be increased, or the size of the heatmap of the skeleton part can be increased. Conversely, as the certainty of the estimation result decreases, the chromaticity of the heatmap color of the skeleton part can be decreased, or the area of ​​the heatmap of the skeleton part can be reduced. Therefore, the estimation accuracy of the 3D skeleton can be visualized on the 3D skeleton model.

[0158] <3-4. Trajectory of the Skeleton>

[0159] The first and second embodiments provide examples of modeling the skeleton portion only in the frame where the acquisition unit 15A acquires the latest captured image. However, the position of the skeleton portion can be modeled across a predetermined number of past frames.

[0160] Figure 11 and Figure 12 This is a diagram showing an example of the positional trajectory of a skeletal section. Figure 11 Overlay images 60A to 60C are shown, in which the position of the skeleton portion in frame tN is overlaid on three captured images from cameras 30A to 30C in frame tN. Frame tN consists of N frames preceding frame t, which is the frame before the latest captured image acquired by acquisition unit 15A. To the right of these overlay images 60A to 60C, overlay images 80A to 80C are shown, in which the trajectory of the skeleton portion's position in the N frames from frame tN to frame t is overlaid on the three captured images from cameras 30A to 30C. Furthermore, Figure 12 An overlay image 80V is shown, in which the trajectory of the skeleton portion's position in N frames from frame tN to frame t is overlaid on the captured image of the bird's-eye view (virtual viewpoint) in frame t. Furthermore, in Figure 11 The overlay images 80A to 80C shown are Figure 12 In the overlay image 80V shown, the trajectories of the joint positions in N frames are overlaid, thus focusing on joints whose subtraction value obtained by subtracting the amount of movement of the center of gravity of the entire 3D skeleton model in frame t is equal to or greater than a threshold, and for other skeleton parts, only the positions of the skeleton parts in frame t are overlaid.

[0161] Here, when the trajectories of the joint positions in N frames are superimposed on superimposed images 80A to 80C and superimposed image 80V, the radius of the joint sphere in past frames (excluding frame t) is set to be smaller than the radius of the joint sphere in frame t. In other words, for the same joint, the radius of the joint sphere in N frames is set by setting the radius of the joint sphere in frame t to its maximum value and monotonically decreasing the radius of the joint sphere in frame t as we trace back from frame t to past frames. Furthermore, compared to the joint in frame t, from frame t to past frames, the brightness and chroma can monotonically decrease, or the transparency can monotonically increase.

[0162] Based on the overlay images 80A to 80C and overlay image 80V generated in this way, the positional trajectory of the skeletal portion can be visualized. Furthermore, since the radius of the joint sphere in frame t decreases monotonically as it moves back from frame t to previous frames, it is easier to identify the temporal changes in the position of the skeletal portion. Note that an example of generating joint positional trajectories has been described here, but of course, the same method can be used to generate the positional trajectories of bones and the positional trajectories of heatmaps.

[0163] <<4. Variation Example>>

[0164] In the processes described in the above embodiments, all or part of the processes described as automatically executed can be performed manually, or all or part of the processes described as manually executed can be performed automatically by known methods. Furthermore, unless otherwise specified, the processing procedures, specific names, and information including various data and parameters shown in the above documents and figures can be arbitrarily changed. For example, the various types of information shown in each figure are not limited to the information shown.

[0165] Furthermore, each component of each device shown in the accompanying drawings is conceptual in function and is not necessarily physically configured as shown in the drawings. In other words, the specific form of distribution and integration of each device is not limited to the form shown, and all or part of it can be functionally or physically distributed and integrated into arbitrary units according to various loads, usage conditions, etc.

[0166] Note that the effects described in this manual are merely illustrative and not limited to these, and other effects may be provided.

[0167] <<5. Hardware Configuration>>

[0168] The server device 10 according to each of the above embodiments is, for example, having, as shown in the example, Figure 13 The computer 1000 with the configuration shown is implemented. Hereinafter, server device 10 or server device 20 according to the above embodiments will be described as examples. Figure 13This is a hardware configuration diagram illustrating an example of a computer 1000. The computer 1000 includes a CPU 1100, RAM 1200, read-only memory (ROM) 1300, hard disk drive (HDD) 1400, communication interface 1500, and input / output interface 1600. Each unit of the computer 1000 is connected via a bus 1050.

[0169] The CPU 1100 operates based on programs stored in ROM 1300 or HDD 1400 and controls each unit. For example, the CPU 1100 expands programs stored in ROM 1300 or HDD 1400 into RAM 1200 and executes processing corresponding to various programs.

[0170] ROM 1300 stores boot programs, such as the Basic Input / Output System (BIOS) executed by CPU 1100 when computer 1000 is activated, and programs that depend on the hardware of computer 1000.

[0171] HDD 1400 is a computer-readable recording medium that non-transitorily records a program executed by CPU 1100, data used by the program, etc. Specifically, HDD 1400 is a recording medium that records a development support program according to the present disclosure, which is an example of program data 1450.

[0172] Communication interface 1500 is an interface for computer 1000 to connect to external network 1550 (e.g., the Internet). For example, CPU 1100 receives data from other devices or sends data generated by CPU 1100 to other devices via communication interface 1500.

[0173] Input / output interface 1600 is an interface used to connect input / output device 1650 and computer 1000. For example, CPU 1100 receives data from input devices such as keyboards or mice via input / output interface 1600. Furthermore, CPU 1100 sends data to output devices such as displays, speakers, or printers via input / output interface 1600. Additionally, input / output interface 1600 can be used as a media interface for reading programs recorded on a predetermined recording medium (medium). This medium may be, for example, optical recording media such as digital multifunction discs (DVDs) or phase-change rewritable discs (PDs), magneto-optical recording media such as magneto-optical discs (MOs), magnetic tape media, magnetic recording media, semiconductor memory, etc.

[0174] For example, when computer 1000 functions as server device 10 or server device 20 according to the above embodiments, the CPU 1100 of computer 1000 implements each functional unit included in control unit 15 by executing an image generation program loaded on RAM 1200. Furthermore, HDD 1400 stores the image generation program and data according to this disclosure in content storage unit 121. Note that CPU 1100 reads program data 1450 from HDD 1400 and executes the program data; however, as another example, these programs can be obtained from other devices via external network 1550.

[0175] This technology can also have the following configurations. (1)

[0177] An information processing device, comprising: An estimation unit that estimates a 3D skeleton based on images captured from multiple viewpoints; and A generation unit generates a skeleton model by modeling the three-dimensional skeleton. (2)

[0179] According to the information processing apparatus described in (1), wherein, The generation unit models the joints as spheres and the bones as cylinders in the three-dimensional skeleton. (3)

[0181] According to the information processing device described in (2), wherein, The generation unit sets the radius of the sphere of the joint or the radius of the cylinder of the bone based on the distance between any one of the plurality of viewpoints or virtual viewpoints and the joint or the bone. (4)

[0183] According to the information processing device described in (3), wherein, The generating unit increases the radius of the sphere of the joint or the radius of the cylinder of the bone as the distance decreases, or decreases the radius of the sphere of the joint or the radius of the cylinder of the bone as the distance increases. (5)

[0185] According to the information processing apparatus described in (1), wherein, The skeleton model is displayed in association with images captured from the plurality of viewpoints or virtual viewpoints. (6)

[0187] According to the information processing apparatus described in (5), wherein, The skeleton model and captured images of frames identical to the skeleton model are displayed side by side, either horizontally or vertically. (7)

[0189] According to the information processing apparatus described in (5), wherein, The skeleton model is displayed by being overlaid on a captured image of the same frame as the skeleton model. (8)

[0191] According to the information processing apparatus described in (7), wherein, The captured image is displayed with reduced brightness or chroma, or with increased transparency. (9)

[0193] The information processing device according to (2) further includes: a setting unit, which sets the effect on the skeleton model. (10)

[0195] According to the information processing apparatus described in (9), wherein, The setting unit sets the effect of the heat map covering the joint or the bone. (11)

[0197] According to the information processing apparatus described in (10), wherein, The setting unit sets heat maps of different colors or sizes according to the position of the joint or the bone. (12)

[0199] According to the information processing apparatus described in (10), wherein, The setting unit sets heat maps of different colors or sizes based on the amount of movement, speed, or acceleration of the joint or the skeletal part of the bone. (13)

[0201] According to the information processing apparatus described in (10), wherein, The setting unit sets heat maps of different colors or sizes based on the certainty of the estimation results of the three-dimensional skeleton in the skeletal part of the joint or the bone. (14)

[0203] According to the information processing apparatus described in (9), wherein, The setting unit is configured to highlight the skeletal portion of the joint or bone. (15)

[0205] According to the information processing apparatus described in (14), wherein, The setting unit sets different display colors based on the amount of movement, speed, or acceleration of the joint or the skeletal part of the bone. (16)

[0207] According to the information processing apparatus described in (14), wherein, The setting unit sets the effect of highlighting skeletal portions of the joint or the skeleton where the amount of movement, velocity, or acceleration is equal to or greater than a predetermined threshold. (17)

[0209] According to the information processing apparatus described in (16), wherein, The setting unit highlights the effect of the skeleton portion whose subtraction value obtained by subtracting the movement of the skeleton model's center of gravity from the movement of the joint or the skeleton portion of the bone is equal to or greater than a predetermined threshold. (18)

[0211] According to the information processing device described in (2), wherein, The generation unit models the trajectory of the position of the joint or the skeletal portion of the bone across a predetermined number of past frames since the latest frame from which the captured image was acquired. (19)

[0213] According to the information processing apparatus described in (18), the generating unit sets the radius of the joint ball or the radius of the skeletal cylinder in past frames other than the latest frame to be smaller than the radius of the joint ball or the radius of the skeletal cylinder in the latest frame. (20)

[0215] According to the information processing apparatus described in (19), the generating unit sets the radius of the joint ball or the radius of the skeletal cylinder in the past frame by setting the radius of the joint ball or the radius of the skeletal cylinder in the latest frame to a maximum value and monotonically decreasing the radius of the joint ball or the radius of the skeletal cylinder in the latest frame as it goes back from the latest frame to the past frame. (twenty one)

[0217] An information processing method performed by a computer: Estimate a 3D skeleton based on images captured from multiple viewpoints; and A skeleton model is generated by modeling the three-dimensional skeleton. (twenty two)

[0219] An information processing program that causes a computer to execute: Estimate a 3D skeleton based on images captured from multiple viewpoints; and A skeleton model is generated by modeling the three-dimensional skeleton.

[0220] Reference Symbol List

[0221] 1 System

[0222] 10 Server Equipment

[0223] 11 Communication Interface

[0224] 13 storage units

[0225] 13A Camera Specifications

[0226] 13B 3D skeleton data

[0227] 15 Control Unit

[0228] 15A Acquisition Unit

[0229] 15B Estimation Unit

[0230] 15C generation unit

[0231] 30A, 30B, 30N cameras

[0232] 50 client terminals

Claims

1. An information processing apparatus, comprising: An estimation unit that estimates a 3D skeleton based on images captured from multiple viewpoints; as well as The generation unit generates a skeleton model by modeling the three-dimensional skeleton. The generation unit models the joints as spheres and the bones as cylinders in the three-dimensional skeleton. The generation unit sets the radius of the sphere of the joint or the radius of the cylinder of the bone based on the distance between the virtual viewpoint and the joint or the bone. The generating unit either increases the radius of the sphere of the joint or the radius of the cylinder of the bone as the distance decreases, or decreases the radius of the sphere of the joint or the radius of the cylinder of the bone as the distance increases; and The generation unit presents the skeleton model from the virtual viewpoint.

2. The information processing apparatus according to claim 1, wherein, The skeleton model is displayed in association with images captured from the plurality of viewpoints or virtual viewpoints.

3. The information processing apparatus according to claim 2, wherein, The skeleton model and captured images of frames identical to the skeleton model are displayed side by side, either horizontally or vertically.

4. The information processing apparatus according to claim 2, wherein, The skeleton model is displayed by being overlaid on a captured image of the same frame as the skeleton model.

5. The information processing apparatus according to claim 4, wherein, The captured image is displayed with reduced brightness or chroma, or with increased transparency.

6. The information processing apparatus according to claim 1, further comprising: A setting unit sets the effect on the skeleton model.

7. The information processing apparatus according to claim 6, wherein, The setting unit sets the effect of the heat map covering the joint or the bone.

8. The information processing apparatus according to claim 7, wherein, The setting unit sets heat maps of different colors or sizes according to the position of the joint or the bone.

9. The information processing apparatus according to claim 7, wherein, The setting unit sets heat maps of different colors or sizes based on the amount of movement, speed, or acceleration of the joint or the skeletal part of the bone.

10. The information processing apparatus according to claim 7, wherein, The setting unit sets heat maps of different colors or sizes based on the certainty of the estimation results of the three-dimensional skeleton in the skeletal part of the joint or the bone.

11. The information processing apparatus according to claim 6, wherein, The setting unit is configured to highlight the skeletal portion of the joint or bone.

12. The information processing apparatus according to claim 11, wherein, The setting unit sets different display colors based on the amount of movement, speed, or acceleration of the joint or the skeletal part of the bone.

13. The information processing apparatus according to claim 11, wherein, The setting unit sets the effect of highlighting skeletal portions of the joint or the skeleton where the amount of movement, velocity, or acceleration is equal to or greater than a predetermined threshold.

14. The information processing apparatus according to claim 13, wherein, The setting unit highlights the effect of the skeleton portion whose subtraction value obtained by subtracting the movement of the skeleton model's center of gravity from the movement of the joint or the skeleton portion of the bone is equal to or greater than a predetermined threshold.

15. The information processing apparatus according to claim 1, wherein, The generation unit models the trajectory of the position of the joint or the skeletal portion of the bone across a predetermined number of past frames since the latest frame from which the captured image was acquired.

16. An information processing method executed by a computer: Estimate a 3D skeleton based on images captured from multiple viewpoints; and A skeleton model is generated by modeling the three-dimensional skeleton. in, The computer models the joints as spheres and the bones as cylinders in the three-dimensional skeleton; The computer sets the radius of the sphere of the joint or the radius of the cylinder of the bone based on the distance between the virtual viewpoint and the joint or the bone. The computer either increases the radius of the sphere of the joint or the radius of the cylinder of the bone as the distance decreases, or decreases the radius of the sphere of the joint or the radius of the cylinder of the bone as the distance increases; and The computer presents the skeleton model from the virtual viewpoint.

17. An information processing program product that causes a computer to execute: Estimate a 3D skeleton based on images captured from multiple viewpoints; and A skeleton model is generated by modeling the three-dimensional skeleton. in, The computer models the joints as spheres and the bones as cylinders in the three-dimensional skeleton; The computer sets the radius of the sphere of the joint or the radius of the cylinder of the bone based on the distance between the virtual viewpoint and the joint or the bone. The computer either increases the radius of the sphere of the joint or the radius of the cylinder of the bone as the distance decreases, or decreases the radius of the sphere of the joint or the radius of the cylinder of the bone as the distance increases; and The computer presents the skeleton model from the virtual viewpoint.

Images