Avatar display control device, program, and avatar display control method

Abstract

To draw an avatar by suppressing deviation from an imaging range of a measurement camera without depending on physique differences of users.SOLUTION: An avatar display control device 2 comprises: bone extraction means 21 which extracts a bone structure of a user from input video in which the user is imaged; joint point scaling means 25 which scales and reflects information of joint points of the user on an avatar; absolute scale calculation means 27 which performs homography conversion from coordinates of both leg points of the user to coordinates in an actual space to calculate an absolute scale; reference point correction means 31 which corrects a position of a reference point of bones of the avatar; and rendering means 33 which renders the avatar at the position of the reference point of the bones of the avatar that the reference point correction means 31 has corrected.SELECTED DRAWING: Figure 11

Description

[Technical Field]

[0001] The present invention relates to an avatar display control device, a program, and an avatar display control method. [Background technology]

[0002] In recent years, with the development of virtual reality (hereinafter referred to as VR) technology and augmented reality (hereinafter referred to as AR) technology, it has become possible to provide services using VR technology and AR technology more easily and at lower cost than before. For example, Non-Patent Documents 1 to 3 describe extracting bone structures from images of a user captured by a camera.

[0003] The applicant, Japan Broadcasting Corporation (NHK), is currently working on various initiatives that apply VR and AR technologies to improve services for viewers. One of these is a system in which a viewer (hereinafter, "user") stands in front of a camera (hereinafter, "measurement camera"), and a three-dimensional CG (Computer Graphics) avatar, Domo-kun (registered trademark), mimics the user's movements as captured by the measurement camera. This system uses image recognition technology to extract bone structures from the user's image captured by the measurement camera, and reflects the bone structures in a model of the Domo-kun (registered trademark) avatar on a CG renderer, thereby providing an experiential user service in which the avatar follows the user's movements. [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] H. Fang, et al., “RMPE: Regional Multi-person Pose Estimation”, IEEE International Conference on Computer Vision, 2017 [Non-patent document 2] RA Guler, et al. ,“DensePose: Dense Human Pose Estimation In The Wild”, IEEE Conference on Computer Vision and Pattern Recognition, 2018 [Non-patent document 3] Z. Cao, et al. ,“OpenPose: Realtime Multi-Person 2D Pose Estimation using Part Affinity Fields”, IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019 Summary of the Invention [Problem to be solved by the invention]

[0005] Services that use image recognition technology to analyze user movements and reflect the results as visual effects require a function to keep the user's position in real space near the center of the measurement camera's imaging range.This is because if the user's body is cut off by the edge of the measurement camera, the bone structure cannot be extracted accurately, and the avatar's movements will be strange.

[0006] One solution is to use a wide-angle lens for the measurement camera. However, using a wide-angle lens makes the user area on the screen relatively small, which leads to a decrease in bone estimation accuracy. This decrease in bone estimation accuracy is particularly noticeable for users with small builds.

[0007] Therefore, we invent a device that has the function of naturally and appropriately limiting the movement range of the measurement camera so that the user does not move close to the edge of the screen, even when taking pictures with a normal lens. One way to prevent users from approaching the edges of the measurement camera screen is to display some kind of warning message on the screen. However, displaying messages, icons, etc. disrupts the user's sense of immersion. To avoid disrupting the user's sense of immersion and more naturally prevent users from approaching the edges of the screen, it is desirable to change the expression of the avatar itself.

[0008] First, let's review the assumptions behind the output screen. In AR, CG is synthesized onto a live-action background to generate the output screen and present it to the user. Therefore, in AR, the measurement camera that captures the background and user and the camera on the CG renderer (hereafter referred to as the CG camera) basically maintain the same angle of view and position and orientation.

[0009] Generally, when a user's movements on the measurement camera screen are reflected on an avatar on the CG camera screen, the user's bone structure is appropriately scaled and fitted to an avatar of a given size. This ensures the identity of the avatar. Even when a large user waves their hand, the hand wave of the avatar on the CG camera screen does not become large, and the amount of the avatar's hand wave is in accordance with the avatar's physique.

[0010] However, if a large user moves their entire body widely in real space, the user will move widely on the measurement camera screen, but the movement of the avatar on the CG camera screen will be smaller. Furthermore, if the user moves outside the measurement range of the measurement camera in real space, the user will be cut off on the measurement camera screen, as mentioned above. This will prevent the system from obtaining the user's bone structure, significantly affecting measurement accuracy.

[0011] If the user's physical characteristics, such as height and width, are similar to those of the avatar, the amount of movement of the user on the measurement camera screen after calibration will be approximately equal to the amount of movement of the avatar on the CG camera screen. In this case, the user can ensure that the avatar does not deviate from the measurement range of the measurement camera in real space by preventing it from deviating from the CG camera screen.

[0012] However, in many cases, the user's physique differs from that of the avatar, so the amount of movement of the user on the measurement camera screen differs from the amount of movement of the avatar on the CG camera screen. If the avatar's physique is smaller than the user's physique, even if the user moves so that the avatar does not deviate from the CG camera screen, the user may move outside the measurable range of the measurement camera in real space, requiring adjustment.

[0013] One of the simplest ways to match the amount of movement of the user on the measurement camera screen with the amount of movement of the avatar on the CG camera screen is to match the scale of the bone structure of the user captured by the measurement camera with the scale of the bone structure of the avatar on the CG camera. However, in this case, the size of the avatar will change depending on the user's physique, compromising the identity of the avatar.

[0014] Therefore, an object of the present invention is to provide an avatar display control device, program, and avatar display control method that can render an avatar while preventing it from deviating from the imaging range of a measurement camera regardless of differences in the user's physique. [Means for solving the problem]

[0015] In order to solve the above problems, the avatar display control device of the present invention includes bone extraction means for extracting a bone structure of a user from an input image of the user, joint point scaling means for scaling information of the user's joint points and reflecting the scaling information in an avatar, and absolute scale calculation means for calculating an absolute scale by performing homography transformation on the coordinates of both feet of the user to coordinates in real space. Correct the position of the reference point of the avatar's bone according to the absolute scale. The apparatus includes a reference point correcting means, and a rendering means for rendering the avatar at the position of the reference point of the avatar's bone corrected by the reference point correcting means.

[0016] The avatar display control method according to the present invention includes the steps of: extracting a bone structure of a user from an input image of the user; scaling information on the user's joint points and reflecting the scaling information on the avatar; and calculating an absolute scale by performing homography transformation on the coordinates of both feet of the user to coordinates in real space. Correcting the position of the reference point of the avatar's bone according to the absolute scale and a procedure for rendering the avatar at the position of the reference point of the corrected bone of the avatar.

[0017] The program according to the present invention includes the steps of: extracting a bone structure of a user from an input image of the user; scaling information on the user's joint points and reflecting the scaling information on the avatar; and calculating an absolute scale by performing homography transformation on the coordinates of both feet of the user to coordinates in real space. Correcting the position of the reference point of the avatar's bone according to the absolute scale and rendering the avatar at the positions of the reference points of the bones of the avatar that have been corrected. [Effects of the Invention]

[0018] According to the present invention, it is possible to render an avatar while preventing it from deviating from the imaging range of the measurement camera, regardless of differences in the physique of the user. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 2 is a diagram illustrating an example of a device configuration according to the present embodiment. [Figure 2A] FIG. 10 is a diagram showing coordinates of an arbitrary user in this embodiment. [Figure 2B] FIG. 10 is a diagram showing the coordinates of a reference person in this embodiment. [Figure 2C] FIG. 2 is a diagram showing the coordinates of an avatar in this embodiment. [Figure 3] 3 is a diagram showing an example of a bone structure of a humanoid model in this embodiment. [Figure 4A] FIG. 10 is a schematic diagram of coordinates of an arbitrary user in this embodiment. [Figure 4B] FIG. 2 is a schematic diagram of coordinates of a reference person in this embodiment. [Figure 4C] FIG. 2 is a schematic diagram of the coordinates of an avatar in this embodiment. [Figure 5A] FIG. 10 is a schematic diagram of joint points and distances between joint points of an arbitrary user in this embodiment. [Figure 5B] FIG. 2 is a schematic diagram of joint points and distances between joints of a reference person in this embodiment. [Figure 5C] FIG. 2 is a schematic diagram of the joint points of an avatar and the distances therebetween in this embodiment. [Figure 6] FIG. 10 is a schematic diagram of a calibration board for absolute scale calculation in this embodiment. [Figure 7] FIG. 10 is a schematic diagram of absolute scale calculation using an arbitrary user and a calibration board in this embodiment. [Figure 8] FIG. 10 is a diagram showing a movement correction coefficient calculation model for an arbitrary user in this embodiment. [Figure 9] FIG. 10 is a diagram showing a model for calculating an avatar movement correction coefficient in this embodiment. [Figure 10A] 1 is a flowchart of the present embodiment. [Figure 10B] 1 is a flowchart of the present embodiment. [Figure 11] FIG. 1 is a block diagram of the present embodiment. DETAILED DESCRIPTION OF THE INVENTION

[0020] Hereinafter, embodiments of the present invention will be described in detail with reference to the accompanying drawings. Consider a service in which a CG (Computer Graphics) avatar on a screen mimics the user's movements in front of a measurement camera, following the user's body movements at an event exhibition or game. In this case, to accurately analyze the user's movements, it is desirable for the user to be positioned near the center of the measurement camera's measurable range (image capture range). The user may move around the measurement camera's measurable range and potentially go outside the measurement camera's measurable range. Users have a variety of physiques, from children to adults. The device of this embodiment realizes a function that naturally restricts users from moving to the edges of the measurement camera's measurable range, regardless of differences in user physique.

[0021] This embodiment relates to a device that guides a user so that the avatar follows the user's movements and does not deviate from the measurement range of the measurement camera, regardless of the user's physique, in a service where the avatar follows the user's movements in a usage scenario such as a television event exhibition or a game. This naturally limits the user's range of movement.

[0022] As shown in FIG. 1, the avatar display control system includes a measurement camera 1 that captures an image of a user 61 in real space, an avatar display control device 2, a monitor 4 that displays an avatar to the user 61, and a calibration board 7.

[0023] The avatar display control device 2 is configured as a computer, and each function is realized by the CPU (Central Processing Unit) executing an avatar display control program. The avatar display control device 2 analyzes the video captured by the measurement camera 1 to extract the subject area and bone structure, calculates movement correction coefficients, etc., and renders the avatar based on the positions of the reference points of the avatar's bones. The avatar display control device 2 needs to calculate the movement correction coefficients to variably adjust the amount of movement of the avatar according to the physique of the user 61. The monitor 4 presents the avatar, which is the rendering result, as augmented reality to the arbitrary user 61. The calibration board 7 has a checkered pattern and is placed at the user's feet within the user's movable range.

[0024] Here, the process in which avatar display control device 2 reflects the bone structure of arbitrary user 61 in the avatar will be explained.

[0025] As shown in Figure 2A, the real space coordinate system Σ r Let P be the three-dimensional point cloud that constitutes the bones of an arbitrary user 61 who is a user of an arbitrary physique. u ∈R 3 An arbitrary user 61 is photographed by the measurement camera 1. As shown in Figure 2B, the real space coordinate system Σ r The three-dimensional point cloud that constitutes the bones of the reference person 62 is P n ∈R 3 The reference person 62 is a virtual person with a known physique, and is assumed to be photographed by the measurement camera 1. Also, as shown in Figure 2C, the CG renderer coordinate system Σ c Then, the three-dimensional point cloud that composes the 63 bones of the avatar "Labo-chan" is P a ∈R 3 This is a CG avatar rendered by a CG camera and has a known bone structure.

[0026] As shown in Non-Patent Documents 1 to 3, a two-dimensional point cloud can be extracted from the image of a monocular measurement camera. In addition, if two or more cameras are used as measurement cameras, the principle of stereo reconstruction is used to extract a two-dimensional point cloud from the real space coordinate system Σ r In addition, some bone extraction libraries can return 3D point clouds as estimation results from images taken with a monocular camera.

[0027] The bone structure of the avatar 63 varies widely depending on the CG model. In this embodiment, a CG model such as Labo-chan, whose proportions are significantly different from those of a human, will be described as an example. An example of the bone structure of a humanoid model is shown in FIG. 3. Markers drawn on various parts of the human body indicate the joint points of the bone structure. Although not shown, this embodiment can be applied even to a case where there is no concept of a neck and the object does not have a human form, such as the CG model of Domo-kun (registered trademark).

[0028] In Fig. 4A, the relative coordinate system Σ of an arbitrary user 61 u Relative coordinate system Σ u The origin of the coordinate system Σ of the reference person 62 is the hip joint point, which is the reference of the bone structure of the user 61. n Relative coordinate system Σ n The origin of the coordinate system Σ of the reference person 62 is the hip joint point, which is the reference of the bone structure of the reference person 62. a Relative coordinate system Σ a The origin of the avatar 63 is the hip joint point (hereinafter referred to as the reference point) which is the reference for the bone structure of the avatar 63. At this time, the three-dimensional point group P n and P a The transformation is performed in the relative coordinate system Σ with the reference point of the bone of the reference person 62 as the origin. n and the relative coordinate system Σ with the reference point of the bone of avatar 63 as the origin. a The conversion between

[0029] That is, the reference point of the bone of the reference person 62 and the reference point of the bone of the avatar 63 are both the origin of the relative coordinate system. na and the relative coordinate system Σ of the reference person 62 n and Avatar 63's relative coordinate system Σ a and the scale coefficients between the joint points are calculated so that the lengths between the joint points are the same.

[0030] Figures 5B and 5C are schematic diagrams of joint points and their distances. The details of how to calculate the scale factor are explained below. The relative coordinate system that shares the reference point is Σ na So, the relative coordinate system Σ na As shown in FIG. 5B, the three-dimensional point cloud P n Each joint point p ni and articulation point p ni+1 Distance between l njAs shown in FIG. 5C, the three-dimensional point cloud P a Each joint point p ai and articulation point p ai+1 Distance between l aj Next, calculate the distance l between the joint points of the avatar 63. aj and the distance l between the joint points of the reference person 62 nj The ratio of aj / l nj Calculate the scale factor s of the distance between the joint points of the reference person 62 and the avatar 63. j The subscript i is an index indicating which joint point it is, and the subscript j is an index indicating which joint point it is.

[0031] At this point, the purpose is to absorb differences in bone structure between the reference person 62 and the avatar 63, such as head size and limb length, so it is desirable that the scale coefficient be calculated based on a reference posture such as a T-pose. A T-pose is a pose in which the user stands upright with both arms extended to the left and right to form the letter T.

[0032] Through the above calculations, the bone structure of the reference person 62 is reflected in the avatar 63. Next, the system calculates the distance between the joint points between the arbitrary user 61 and the reference person 62, and calculates a scale factor.

[0033] 5A and 5B are schematic diagrams of joint points and their distances. The details of the method for calculating the scale factor are explained below. A relative coordinate system Σ that shares the reference point as the origin is un and the relative coordinate system Σ of any user 61 u and the relative coordinate system Σ of the reference person 62 n Then, the relative coordinate system Σ un As shown in FIG. 5A, the three-dimensional point cloud P u Each joint point p ui and articulation point p ui+1 Distance between l uj Next, as shown in FIG. 5B, the three-dimensional point cloud P n Each joint point p ni and articulation point p ni+1 Distance between lnj Furthermore, the distance l between the joint points of any user 61 is calculated. uj and the distance l between the joint points of the reference person 62 nj The ratio of nj / l uj Calculate the scale factor s of the distance between the joint points of the arbitrary user 61 and the reference person 62. j 'Let's say.

[0034] Scale factor s j ' is calculated when the given user 61 first joins. At this time, the given user 61 does not necessarily take a reference pose such as a T-pose. Therefore, in this system, various poses of the reference person 62 are stored in a database in advance, and the pose that is closest to the pose taken by the given user 61 is selected from the database, and the scale coefficient s j It is advisable to calculate '. Note that this pose search can be performed based on the three-dimensional angle of the line connecting each joint point.

[0035] However, this is not limited to this, and the present system calculates the vector correlation of the three-dimensional angle (expressed by two parameters, the azimuth angle θ and the zenith angle φ) of the line connecting each joint point in a relative coordinate system with the reference point as the origin, between the pose of the input arbitrary user 61 and the pose of the reference person 62 in the database, selects the pose with the highest vector correlation, and calculates a scale coefficient s based on the distance between the joint points in the pose of this reference person 62. j ' may be calculated.

[0036] The three-dimensional angle vector of the line connecting each joint point is given by the following equation (1).

number

[0037] After the scale factor is calculated, the system converts the input 3D point cloud P u The length l between each joint point uj to the scale factor s j In equation (2), a certain joint point pui and p ui+1 to the next scaled joint point p s ui+1 This shows the case where

number

[0038] When scaling in three-dimensional space is performed as described above, the three-dimensional point cloud P u It is desirable that the absolute scale of the bone structure of the user 61 is known. For this reason, a calibration board 7 shown in FIG. 6 is placed under the feet of the user 61 in the shooting range to calculate the absolute scale. In this embodiment, the calibration board 7 is, for example, a board with a black and white checkerboard pattern. Since the lengths of the sides of the rectangles that make up the checkerboard pattern are known, the absolute scale of the bone structure of the user 61 can be calculated from the length of the sides. However, the present invention is not limited to this, and it is sufficient that a marker of a known size is included somewhere on the floor of the real space in the shooting range.

[0039] An arbitrary user 61 shown in FIG. 7 has his feet on the calibration board 7. The system can calculate the absolute scale of the distance between the joint points of the heels of both feet of the arbitrary user 61 by calculating how many times the distance between the joint points of the heels of both feet or the toes of both feet that are closest to the contact points is the length of one side of the checkered square marker drawn on the calibration board 7. Then, the absolute scale of the bone structure of the arbitrary user 61 can be determined based on the distance between the joint points of the heels of both feet of the arbitrary user 61.

[0040] By performing homography transformation by approximating that the joint points of both feet of an arbitrary user 61 exist on the calibration board 7, the absolute distance between the joint points of both feet on the ground, which is the real space, can be calculated.

[0041] The homography transformation from point x on the image to point x' in the real space is expressed by the following equation (3).

number

[0042] h 11 From h 33 Since x is estimated by a known method such as the OpenCV findHomography function, if x is a point on the image and x' is a point in real space (where z = 0 on the ground), the absolute distance can be calculated from x'. In this way, the scale coefficient s of each joint point that makes up the bone of an arbitrary user 61 can be calculated. j ' is required.

[0043] Next, consider the correction coefficient for the amount of movement on the screen when the given user 61 moves his or her entire body. Assuming a pinhole camera model, the amount of movement of the given user 61 on the screen is proportional to the amount of movement in real space. When a large user is projected on the screen, the size of the user will be larger than the size of the avatar 63. Accordingly, the amount of movement of the given user 61 on the screen will also be proportionally larger than the amount of movement of the avatar 63 on the screen. The given user 61 visually watches the avatar 63 on the screen of the monitor 4 and moves while providing feedback so that the avatar 63 does not deviate from the edge of the screen. If the amount of movement of the avatar 63 is smaller than the amount of movement of the given user 61, there is a risk that the given user 61 in real space will deviate from the imaging range of the measurement camera 1.

[0044] Real space coordinate system Σ r The horizontal width of the space occupied by an arbitrary user 61 is w u , the horizontal width of the space occupied by the avatar 63 is w a Assume that the amount of movement in one step is d. The distance from the center of the screen to the edge of the screen is W. If a proportionality constant k is used to indicate the relationship between the width of any user 61 and the amount of movement in one step, the amount of movement d is the product of the proportionality constant k and the width w. If any user 61 located at the center of the screen reaches the edge of the screen in q steps, the system corrects the avatar 63 located at the center of the screen so that it reaches the edge of the screen in q steps regardless of the physique. Therefore, when the movement correction coefficient h is used, the equation (4) that satisfies the above is obtained.

number

[0045] From equation (4), the movement correction coefficient h can be calculated using equation (5) below.

number

[0046] Width w a and w u can be obtained from the horizontal length of the area obtained by projecting the maximum rectangular range encompassing the bone configuration point clouds projected onto the screens of the CG camera and measurement camera 1 onto three-dimensional space (real space). The absolute horizontal scale can be calculated from the results of homography transformation of horizontal line segments that are considered to be on the calibration board 7 on the screen. Figures 8 and 9 show overview diagrams.

[0047] The movement correction coefficient h increases as the physique of the user 61 increases relative to the avatar 63. While the avatar 63 is providing a service to imitate the user's movements, the movement correction coefficient h is always multiplied by the movement amount of the reference point of the bone of the avatar 63, thereby positioning the avatar 63 at the position of the user 61 on the screen.

[0048] In this way, the amount of movement of the user 61 on the screen is made equal to the amount of movement of the avatar 63 on the screen, and the position of the avatar 63 on the screen of the monitor 4 is visually fed back to the user 61, thereby making the user 61 aware that the user 61 is cut off from the screen. This prevents the user 61 from moving to the edge of the screen of the measurement camera 1 and straying from the imaging range (measurable range) of the measurement camera 1.

[0049] This conversion considers the amount of horizontal movement on the image in a coordinate system directly facing the measurement camera 1, but similar processing is possible for vertical movement as well. Furthermore, the influence of differences in the user's physique on the depth direction relative to the measurement camera 1 is small, so correction is not considered necessary.

[0050] This embodiment can be used in both VR and AR. In the case of VR, everything displayed on the screen to the user is CG, and there is no need to perform processing to hide the user from the screen. In the case of AR, only the avatar 63 is CG, and the background is a live image obtained by reusing the image of the measurement camera 1. In the case of AR, a sky image of the background is acquired in advance by the measurement camera 1, and if the user protrudes from the avatar 63, the sky image is synthesized to conceal the user. The user area is recognized by, for example, the width w u However, this is not limited to this, and other algorithms such as directly cutting out the person area may be used.

[0051] 10A and 10B are flowcharts showing the display control process of this embodiment. The display control process will be described with reference to the block diagram shown in FIG.

[0052] The avatar display control device 2 is composed of a bone extraction means 21, a three-dimensional angle calculation means between joint points 22, a pose search means 23, pose data 24, a joint point scaling means 25, a homography conversion means 26, an absolute scale calculation means 27, a rectangle calculation means 28, a width calculation means 29, a movement correction coefficient calculation means 30, a reference point correction means 31, a user hiding means 32, a rendering means 33, and a video output means 34.

[0053] Bone extraction means 21, joint point three-dimensional angle calculation means 22, pose search means 23, joint point scaling means 25, homography conversion means 26, absolute scale calculation means 27, rectangle calculation means 28, width calculation means 29, movement correction coefficient calculation means 30, reference point correction means 31, user hiding means 32, rendering means 33, and video output means 34 are realized by a CPU (not shown) included in avatar display control device 2 executing an avatar display control program (not shown). Avatar display control device 2 analyzes input video 51 to extract the subject area and bone structure, calculates movement correction coefficients, etc., reflects the bone structure in avatar 63, and renders the result to generate output video 52.

[0054] First, input video 51 is input to avatar display control device 2 (step S10). Input video 51 includes a marker for scaling the size of the user. The bone extraction means 21 analyzes the input video 51 to extract the subject area and bone structure (step S11). Here, the subject area refers to the subject region on the screen.

[0055] Thereafter, the processes of steps S12 and S13 and the processes of steps S14 and S15 are carried out in parallel. In step S12, the joint point three-dimensional angle calculation means 22 calculates the three-dimensional angles of the joint points of the bone structure of the arbitrary user 61. Then, the pose search means 23 searches for the pose (posture) of the reference person 62 stored in the pose data 24 that is closest to the pose (posture) taken by the arbitrary user 61, and calculates the scale coefficient s between the arbitrary user 61 and the reference person 62. j ' and the scale factor s between the reference person 62 and the avatar 63 j is calculated (step S13).

[0056] The scale factor s between the reference person 62 and the avatar 63 is j may be calculated in advance. Also, the scale factor s between the arbitrary user 61 and the reference person 62 j ' may be calculated manually and input to the system, and is not limited to this. In other words, the pose search means 23 and the pose data 24 are optional components.

[0057] In step S14, the homography transformation means 26 performs homography transformation on the points of both feet of the arbitrary user 61 on the image to points of both feet in real space. These points of both feet are joint points on the ground and are located on the calibration board 7. Then, the absolute scale calculation means 27 calculates the absolute distance (absolute scale) between the joint points on the ground, which is real space (step S15). At this time, the absolute scale calculation means 27 calculates the absolute distance between the joint points in real space based on markers of known sizes displayed on the calibration board 7. The calibration board 7 in this embodiment displays black and white checkered markers of known sizes.

[0058] The avatar display control device 2 only needs to perform homography transformation of the coordinates of both feet of the arbitrary user 61 into coordinates in real space to calculate the absolute scale, and the detailed procedure is not limited.

[0059] Thereafter, the processes of steps S16 to S18 and the processes of steps S19 and S20 are carried out in parallel. In step S20, the joint point scaling means 25 calculates a scale coefficient s j and s j Based on the calculated distances between the joint points of the user 61, the distances between the joint points of the user 61 are scaled, and the three-dimensional angles of the joint points of the bone structure of the user 61 are reflected in the avatar 63 (step S16). Then, the rectangle calculation means 28 detects a rectangle including the subject area of ​​the avatar 63 (step S17). The width calculation means 29 calculates the width w of the rectangle including the subject area of ​​the avatar 63. a is calculated (step S18).

[0060] In step S19, the rectangle calculation means 28 detects a rectangle including the subject area of ​​the arbitrary user 61. The width calculation means 29 calculates the width w of the rectangle including the subject area of ​​the arbitrary user 61. u is calculated (step S20).

[0061] In step S21, the movement correction coefficient calculation means 30 calculates a movement correction coefficient h for correcting the movement amount of the avatar 63. When the movement amount of the avatar 63 is corrected by multiplying it by the movement correction coefficient h, the movement amount of the avatar 63 becomes equal to the movement amount of the arbitrary user 61. Then, the reference point correction means 31 corrects the position of the reference point, which is the joint point that serves as the reference for the bone structure of the avatar 63 (generally, the waist is often used as the reference point) (step S22).

[0062] The rendering means 33 renders the avatar 63 at the position of the reference point of the bone of the avatar 63 (step S23). In parallel with this, the user hiding means 32 hides the image of the arbitrary user 61 in the input video 51 and replaces it with the background video (step S24). Finally, the video output means 34 generates the output video 52 by combining the background video generated by the user hiding means 32 with the avatar 63 rendered by the rendering means 33 (step S25), and then ends the processing of Figure 10B.

[0063] The avatar display control device 2 may correct the positions of the reference points of the bones of the avatar 63 using the reference point correction means 31, and may render the avatar 63 at the positions of the reference points of the bones of the avatar 63. The rectangle calculation means 28 and the width calculation means 29 are optional components. The avatar 63 may be rendered so that the position of the arbitrary user 61 on the screen coincides with the position of the avatar 63 on the screen, and the details of the calculation are not limited to the rectangle calculation means 28 and the width calculation means 29.

[0064] 10A and 10B are related to AR. If the processing is related to VR, the user concealment means 32 is not necessary. In other words, the user concealment means 32 is an optional component.

[0065] In addition to the above functions, if the user 61 actually moves to the edge of the screen and moving any further would cause part of their body to be cut off from the measurement camera 1's field of view, the avatar 63 may be forcibly made transparent. In VR, making the avatar 63 transparent indicates to the user that the avatar itself has disappeared. In AR, making the avatar 63 transparent and canceling the concealment process reveals the user's appearance on the monitor 4, indicating to the user that the AR effect has been canceled. This alerts the user to deviations from the measurable range and guides them to avoid deviations from the measurable range of the measurement camera. The bone structure of the user 61 is appropriately scaled and fitted to the avatar 63, ensuring the identity of the avatar 63. Body movements, such as when the user 61 waves their hand, are reflected in the avatar 63 according to the physique of the avatar 63.

[0066] Furthermore, when the given user 61 is no longer detected in the input video 51, the rendering means 33 of the avatar display control device 2 may fix the pose of the avatar 63 and render it at the edge of the screen. By visually feeding back the fixation of the avatar 63 on the display screen of the monitor 4, the given user 61 can recognize that he or she has deviated from the imaging range of the measurement camera 1.

[0067] Effect of this embodiment In a service where a user moving within a specific measurable range is measured with a measurement camera at an event exhibition or game, and the extracted bone structure is reflected in an avatar, the user can be guided so that they do not deviate from the measurable range measured by the measurement camera regardless of their physique. [Industrial Applicability]

[0068] The present invention is primarily used in situations where a user controls an on-screen avatar while moving within a specific range, and has applicability in event exhibition halls, simulators, and the game industry.

[0069] Although the embodiments have been described in detail above, the present invention is not limited to the above-described embodiments, and includes design modifications and the like within the scope of the present invention.

[0070] In the above-described embodiment, the avatar display control device is described as independent hardware, but the present invention is not limited to this. For example, the present invention can be realized by a program that causes hardware resources such as a CPU, memory, and hard disk of a computer to function as the above-described avatar display control device. This program may be distributed via a communication line or written to a recording medium such as a CD-ROM or flash memory. [Explanation of symbols]

[0071] 1. Measurement camera 2 Avatar display control device 21 Bone Extraction Method 22 Means for calculating three-dimensional angles between joint points 23 Pose search method 24 pose data 25 Articulation point scaling means 26 Homography Transformation Method 27 Absolute Scale Calculation Method 28 Rectangle calculation means 29 Width calculation means 30 Movement correction coefficient calculation means 31 Reference point correction means 32 User Concealment Methods 33 Rendering Method 34 Video output means 4 monitors 51 Input video 52 Output video 61 Any User 62 Standard person 63 Avatar 7 Calibration Board

Claims

1. a bone extraction means for extracting a bone structure of a user from an input image of the user; a joint point scaling means for scaling the user's joint point information and reflecting the scaling information on the avatar; an absolute scale calculation means for calculating an absolute scale by performing a homography transformation on the coordinates of both feet of the user to coordinates in real space; a reference point correcting means for correcting the positions of the reference points of the bones of the avatar in accordance with the absolute scale; a rendering means for rendering the avatar at the position of the reference point of the bone of the avatar corrected by the reference point correction means; An avatar display control device comprising:

2. a user concealment means for generating a background image from the input image by concealing the user; a synthesis means for synthesizing the avatar rendered by the rendering means with the background image; The avatar display control device according to claim 1 , further comprising:

3. the input image includes a marker for scaling the size of the user; 2. The avatar display control device according to claim 1, wherein:

4. When the user is no longer detected from the input video, the rendering means renders the avatar at the edge of the screen.

2. The avatar display control device according to claim 1, wherein:

5. When the user is no longer detected from the input video, the rendering means does not render the avatar.

2. The avatar display control device according to claim 1, wherein:

6. the joint point scaling means reflects the three-dimensional angles of the joint points of the bone structure of the user in the avatar; 2. The avatar display control device according to claim 1, wherein:

7. A program for causing a computer to function as the avatar display control device according to any one of claims 1 to 6.

8. extracting a bone structure of a user from an input image of the user; a step of scaling the user's joint point information and reflecting it in an avatar; a step of calculating an absolute scale by performing a homography transformation on the coordinates of both feet of the user to coordinates in real space; a step of correcting the positions of the reference points of the bones of the avatar according to the absolute scale; Rendering the avatar at the position of the corrected reference point of the bone of the avatar; 2. An avatar display control method comprising causing a computer to execute the above steps.

Images