Video display device, video display method, and program

Abstract

To provide a video display device with which it is possible to display a self-video comparable with a reference video without having to prepare the same imaging environment as for the reference video.SOLUTION: A video display device 1 comprises: a storage unit 11 in which a reference video is stored that is a video of motion of an imitation object whose motion a user is going to imitate; a video acquisition unit 12 that acquires a first self-video which is a video of the user's motion; a skeleton recognition unit 13 that performs skeleton recognition of the user of the first self-video; a generation unit 14 that generates a second self-video by using the result of skeleton recognition, which is a video of a three-dimensional object that moves according to the user's motion, so that a relative positional relationship between a camera for reference images that captures a reference video and the imitation object is equalized with a relative positional relationship between the view point of a video of the three-dimensional object and the three-dimensional object; and a display unit 15 that displays the reference video and the second self-video.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image display device and the like that displays an image of a motion to be imitated by a user and an image corresponding to the user's motion. [Background technology]

[0002] Conventionally, there is known a learning support device that displays a composite image of a reference image, which is an image of a target movement that a learner is to imitate, and a self-image, which is an image of the learner's movement, for learning movements such as surgery (see, for example, Patent Document 1). By referring to such a display, a learner can train to perform the same movements as the target movement. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-071443 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, in conventional learning support devices, in order to make the reference video and the self-video comparable, it was necessary to make the relative positional relationship between the camera that shoots and the subject of the video the same for both videos. For example, if the reference video is a first-person video shot with a head-mounted camera worn by a teacher who performs the action to be imitated, the self-video also needed to be a first-person video shot with a head-mounted camera worn by the learner. In this way, the learner also needed to prepare a shooting environment similar to that of the reference video, which was problematic in terms of cost and time.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to provide an image display device, etc. that can display a self-image that can be compared with a reference image without having to prepare a shooting environment similar to that of the reference image. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, an image display device according to one embodiment of the present invention comprises a memory unit in which a reference image, which is an image of a target whose actions a user is to imitate, an image acquisition unit that acquires a first self-image, which is an image of the user's actions, a skeletal recognition unit that performs skeletal recognition of the user contained in the first self-image, a generation unit that uses the results of skeletal recognition by the skeletal recognition unit to generate a second self-image, which is an image of a three-dimensional object corresponding to the target and which moves in response to the user's actions, such that the relative positional relationship between a reference image camera that captures the reference image and the target and the relative positional relationship between the viewpoint of the image of the three-dimensional object and the three-dimensional object are the same, and a display unit that displays the reference image and the second self-image, wherein the relative positional relationship between the reference image camera and the target is different from the relative positional relationship between the self-image camera that captures the first self-image and a part of the user corresponding to the target.

[0007] With this configuration, it is possible to display a second self-image that can be compared with the reference image without preparing a shooting environment similar to that of the reference image. For example, it is possible to generate and display a second self-image from the user's first-person viewpoint from a first self-image captured by a camera facing the user on a laptop computer, tablet device, smartphone, or the like.

[0008] In addition, in an image display device according to one aspect of the present invention, the generation unit may generate a second self-image by changing the line of sight direction resulting from skeleton recognition by the skeleton recognition unit by a set angle.

[0009] With this configuration, for example, a second self-image from the user's first-person perspective can be generated from a first self-image captured by a camera facing the user.

[0010] In addition, in an image display device according to one aspect of the present invention, the reference image is an image from the perspective of the person being imitated who moves the object to be imitated, and the generation unit may generate a second self-image which is an image from the user's perspective.

[0011] With this configuration, a second self-image from a first-person perspective can be generated regardless of the relative positional relationship between the self-image camera that captures the first self-image and the user.

[0012] In addition, in an image display device according to one aspect of the present invention, the imitation target includes an operation object whose shape changes, the first self-image includes the user's hand, and the generation unit may generate a second self-image including a three-dimensional object of the operation object whose shape changes in response to a gesture of the user's hand included in the first self-image.

[0013] With this configuration, a three-dimensional object as an operation target can be operated by gestures without using a controller or the like.

[0014] In addition, in an image display device according to one embodiment of the present invention, the object to be imitated may include an object to be operated whose shape changes, and the device may further include a reception unit that receives instructions from a controller operated by a user, and the generation unit may generate a second self-image that includes a three-dimensional object of the object to be operated whose shape changes in response to the instructions received by the reception unit.

[0015] With this configuration, the three-dimensional object of the operation target can be operated by using the controller. Therefore, for example, when the reference image is an image of a surgical robot, the user can operate the three-dimensional object of the operation target by using a controller similar to the controller used to operate the surgical robot, and the user can operate the three-dimensional object in an environment similar to that of a real surgical robot.

[0016] In the image display device according to an aspect of the present invention, the display unit may synthesize and display the reference image and the second own image.

[0017] With this configuration, the reference video and the second self-video can be easily compared, and the user can learn to make his / her own movements similar to the movements of the target person in the reference video.

[0018] In addition, in an image display device according to one embodiment of the present invention, the self-image camera and the display device on which the reference image and the second self-image are displayed may be arranged so that the direction from the self-image camera to the subject to be photographed in the optical axis direction is opposite to the direction of the line of sight when looking directly at the display device.

[0019] With this configuration, it becomes possible to generate a second self-image from a first self-image captured by a camera facing the user, such as on a laptop, tablet, or smartphone, and to display the second self-image and a reference image to the user.

[0020] Furthermore, an image display method according to one aspect of the present invention is an image display method processed using a memory unit in which a reference image, which is an image of a target movement of an imitation target whose movement is to be imitated by a user, an image acquisition unit, a skeletal recognition unit, a generation unit, and a display unit, the method comprising the steps of: the image acquisition unit acquiring a first self-image, which is an image of the user's movements; the skeletal recognition unit performing skeletal recognition of the user contained in the first self-image; the generation unit generating a second self-image, which is a three-dimensional object corresponding to the target and which moves in response to the user's movements, using the results of the skeletal recognition, such that the relative positional relationship between a reference image camera that captures the reference image and the target and the relative positional relationship between the viewpoint of the image of the three-dimensional object and the three-dimensional object are the same; and the display unit displaying the reference image and the second self-image, wherein the relative positional relationship between the reference image camera and the target is different from the relative positional relationship between the self-image camera that captures the first self-image and a part of the user corresponding to the target. Effect of the Invention

[0021] According to an image display device or the like according to one aspect of the present invention, it becomes possible to display a self-image that can be compared with a reference image, even if a shooting environment similar to that of the reference image is not prepared. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram showing a configuration of a video display device according to an embodiment of the present invention; [Diagram 2] FIG. 13 is a diagram showing an example of how a computer is used to realize the video display device according to the embodiment. [Diagram 3] FIG. 2 shows an example of a reference video in the embodiment. [Figure 4] FIG. 13 shows an example of a first self-image and a result of skeletal recognition in the embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of a result of hand skeleton recognition in the embodiment; [Figure 6] FIG. 13 shows an example of a second self-image in the embodiment. [Figure 7] FIG. 13 shows an example of display of a first and second self-image in the embodiment. [Figure 8A] FIG. 13 is a diagram showing an example of an operation of an operation target by a gesture in the embodiment; [Figure 8B] FIG. 13 is a diagram showing an example of an operation of an operation target by a gesture in the embodiment; [Figure 9] A flowchart showing the operation of the video display device according to the embodiment. [Figure 10] FIG. 4 is a block diagram showing another configuration of the video display device according to the embodiment; [Figure 11] FIG. 2 is a perspective view showing an example of a controller according to the embodiment; [Figure 12] FIG. 2 shows an example of a configuration of a computer according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The image display device and image display method according to the present invention will be described below using embodiments. In the following embodiments, components and steps with the same reference numerals are the same or equivalent, and repeated description may be omitted. The image display device according to this embodiment uses a first self-image captured in a different shooting environment from a reference image, which is an image of a target action that the user is trying to imitate, to generate a second self-image of a three-dimensional object so that the relative positional relationship between the viewpoint and the three-dimensional object is similar to that of the shooting environment of the reference image, and displays the reference image and the second self-image.

[0024] Fig. 1 is a block diagram showing a configuration of an image display device 1 according to the present embodiment. The image display device 1 according to the present embodiment includes a storage unit 11, an image acquisition unit 12, a skeleton recognition unit 13, a generation unit 14, and a display unit 15. Note that the image display device 1 may be realized by a computer 900 as shown in Fig. 2, for example. This case will be mainly described in the present embodiment.

[0025] The storage unit 11 stores a reference image, which is an image of the motion of a target object to be imitated by the user. The user is a learner who learns the motion while referring to the reference image. The motion to be learned by the user may be, for example, a motion such as surgery, a motion of work in a factory, a motion of nursing care or a hotel, a motion such as cooking, a motion for creating a work such as a craft, a motion such as sports, a motion such as calligraphy, a motion of tying a rope, or other motions. The target object to be imitated may be, for example, a part of the body of the imitated person, or an object operated by the imitated person. The imitated person may be, for example, a teacher of the user who is a learner, and may be skilled in the motion that the learner is to learn. In addition, a part of the body of the imitated person may include, for example, the hand of the imitated person. In addition, the object operated by the imitated person may be, for example, the finger or forceps of a surgical robot, or may be a tool such as forceps, scalpel, tweezers, scissors, or brush that the imitated person has. The reference image is usually an image captured by a camera, but may be a CG (Computer Graphics) image equivalent to an image captured by a camera. The reference image may be, for example, an image from the viewpoint of the person to be imitated who operates the object to be imitated, that is, an image from the first-person viewpoint of the person to be imitated. In this case, the reference image may be, for example, an image captured by a head-mounted camera worn by the person to be imitated. In this embodiment, as shown in FIG. 3, a case will be mainly described where a reference image including an object to be imitated 21, which is a forceps of a surgical robot, is stored in storage unit 11.

[0026] In the storage unit 11, for example, the entire reference video may be stored, or a part of the reference video may be stored. As an example, when the video display device 1 displays a reference video while receiving it from the outside, a part of the reference video, that is, the latest reference video received, may be stored in the storage unit 11, and may be read out and displayed, and may be overwritten sequentially. The storage unit 11 may store information other than the reference video. For example, information of a three-dimensional object may be stored in the storage unit 11, a second self-video generated by the generation unit 14 may be stored in the storage unit 11, and a first self-video acquired by the video acquisition unit 12 may be stored in the storage unit 11.

[0027] The process by which information is stored in the storage unit 11 is not important. For example, information may be stored in the storage unit 11 via a recording medium, information transmitted via a communication line or the like may be stored in the storage unit 11, or information inputted via a device such as a camera may be stored in the storage unit 11. The storage unit 11 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory or a magnetic disk.

[0028] The image acquisition unit 12 acquires a first self-image, which is an image of the user's actions. The image acquisition unit 12 may be, for example, an optical device such as a camera that captures an image, or may acquire an image captured by an optical device such as a camera. In this embodiment, a case in which the image acquisition unit 12 receives an image captured by a camera 901 that is a self-image camera will be mainly described. The first self-image may be, for example, an image of the user's hand actions. The first self-image may be, for example, an image of the user's palm, an image including the user's arm from the wrist to the elbow, an image including the user's arm from the elbow to the shoulder, or an image including the user's shoulder and torso. When the second self-image is generated by changing the line of sight direction by a set angle, it is preferable that the first self-image is captured as determined in advance. For example, the first self-image may be captured in a state in which the self-image camera and the user face each other. In addition, when a second self-image from the user's viewpoint is generated, it is preferable to obtain a first self-image that indicates the user's line of sight, for example, a first self-image that also includes the user's head.

[0029] It is assumed that the relative positional relationship between the reference image camera that captures the reference image and the imitation target is different from the relative positional relationship between the self-image camera that captures the first self-image and the part of the user that corresponds to the imitation target. The part of the user that corresponds to the imitation target is not particularly limited, and may be, for example, the palm of the user's hand or the part of the hand from the elbow of the user. As an example, the reference image may be an image from the first person perspective of the imitation target, and the first self-image may be an image captured by a camera 901 facing the user 30 as shown in FIG. 2. In this embodiment, this case will be mainly described. In the present embodiment, as an example, as shown in Fig. 2, the camera 901 and the display device 902 on which the reference image and the second self-image are displayed are arranged so that the direction from the camera 901 to the user 30 who is the subject of the image capture in the optical axis direction is opposite to the direction of the line of sight of the user 30 who looks directly at the display device 902, that is, the case where the first self-image is captured and the reference image and the second self-image are displayed using a laptop computer with a camera, a tablet terminal with a camera, a smartphone with a camera, etc. will be mainly described. Note that the camera 901 may be, for example, a built-in camera of the computer 900. The built-in camera of the computer 900 may be, for example, a front-facing camera of a laptop computer, a tablet terminal, or an in-camera of a smartphone.

[0030] The skeleton recognition unit 13 performs skeleton recognition of the user included in the first self-image. For example, the skeleton recognition unit 13 may detect a person or a part of a person in a certain frame of the first self-image, and perform skeleton recognition on the detected person or part of a person. The skeleton recognition unit 13 may perform the process on each of a plurality of frames included in the first self-image. That is, the skeleton recognition process may be performed repeatedly. Note that the skeleton recognition may be performed on all frames of the first self-image, or may be performed on discontinuous frames. The part of the person may be, for example, the upper body of a person, a part of an arm including a palm of a person, or a part of a palm of a person. When the generation unit 14 generates a second self-image from the user's viewpoint, it is preferable to perform skeleton recognition including the user's head. This method of skeleton recognition is already known, and a detailed description thereof will be omitted. For example, as shown in FIG. 4, the skeleton recognition unit 13 may identify a user 30 included in the first self-image and recognize a skeleton 31 of the user 30. As an example, the skeleton 31 may include, as shown in Fig. 4, circular node figures corresponding to joints and end parts of the body such as fingertips and the head, and linear link figures corresponding to body parts such as the arm connecting them. Fig. 5 is a diagram showing the skeleton 31 recognized for the hand 32 of the user 30. In this embodiment, a case where a three-dimensional object is operated using the skeleton 31 of the hand 32 recognized by the skeleton recognition unit 13 will be mainly described.

[0031] The generating unit 14 generates a second self-image, which is a three-dimensional object corresponding to the imitation target and is an image of a three-dimensional object that moves according to the user's movement, using the result of the skeleton recognition by the skeleton recognition unit 13. The three-dimensional object corresponding to the imitation target is preferably a three-dimensional object having a shape similar to the imitation target, but may be, for example, a three-dimensional object having a shape that is a simplified version of the shape of the imitation target. The position and posture of the three-dimensional object included in the second self-image may be, for example, a position and posture according to the recognition result of the skeleton of the part of the user corresponding to the imitation target, or may move according to a change in the time series of the recognition result of the skeleton of the part of the user corresponding to the imitation target. In addition, the generating unit 14 generates the second self-image so that the relative positional relationship between the reference image camera that captures the reference image and the imitation target is the same as the relative positional relationship between the viewpoint of the image of the three-dimensional object and the three-dimensional object. The viewpoint of the image of the three-dimensional object may be, for example, a viewpoint when rendering a three-dimensional object arranged in a three-dimensional virtual space into a two-dimensional image. The second self-image may be generated so that the relative positional relationship between the direction of the optical axis of the reference image camera and the imitation target and the relative positional relationship between the direction of the line of sight of the image of the three-dimensional object and the three-dimensional object are also the same. In this way, the first self-image can be converted into a second self-image taken in a shooting environment similar to that of the reference image. For example, when the reference image is a first-person viewpoint image, the second self-image, which is a first-person viewpoint image, can be generated from the first self-image taken by the camera 901 facing the user 30 as shown in FIG. 2. Note that the viewpoint and line of sight directions being the same may mean, for example, that the two are exactly the same, or that the two are the same within a predetermined error range.

[0032] The result of the skeleton recognition shows a skeleton recognized in a two-dimensional image, but the length of each part of the human body is generally determined, such as the length from the shoulder to the elbow, the length from the elbow to the wrist, the length from the wrist to the base of each finger, the length of each finger, the width of the hand, etc., and the range of motion of each joint is also determined. Therefore, by taking these into consideration, the generation unit 14 can estimate the three-dimensional position and posture of the skeleton of the user 30 included in the first self-image based on the result of the skeleton recognition by the skeleton recognition unit 13. For example, the generation unit 14 may specify the position and optical axis direction of the self-image camera that captured the first self-image and the user's skeleton 31 in three-dimensional space based on the result of the skeleton recognition.

[0033] For example, in the skeleton 31 of the hand 32 shown in Fig. 5, even if the hand 32 rotates in the direction of the arrow B1, the length of the double-sided arrow A1 does not change, and even if the hand 32 rotates in the direction of the arrow B2, the length of the double-sided arrow A2 does not change. The double-sided arrow A1 is a double-sided arrow whose two end points are the joint at the base of the index finger and the joint at the base of the little finger, and the double-sided arrow A2 is a double-sided arrow whose two end points are the joint at the wrist and the joint at the base of the middle finger. Therefore, the generation unit 14 can know the angle of the hand in the directions of the arrows B1 and B2 and the change in the angle according to the length of the double-sided arrows A1 and A2 of the hand 32 and the change in the length in the result of the skeleton recognition.

[0034] The direction of rotation in the direction of the arrow B1 may be determined, for example, according to a change in the distance between the joints of each finger. For example, when the distance between the joints on the side closer to the wrist increases while the distance between the joints on the side closer to the fingertip decreases when rotating in the direction of the arrow B1, it may be determined that the wrist side has rotated in a direction approaching the camera, and when the distance between the joints on the side closer to the wrist decreases while the distance between the joints on the side closer to the fingertip increases, it may be determined that the fingertip side has rotated in a direction approaching the camera. The direction of rotation in the direction of the arrow B2 may be determined in a similar manner. For example, when the distance between the joints of the little finger increases while the distance between the joints of the thumb decreases when rotating in the direction of the arrow B2, it may be determined that the little finger side has rotated in a direction approaching the camera, and when the distance between the joints of the little finger decreases while the distance between the joints of the thumb increases, it may be determined that the thumb side has rotated in a direction approaching the camera.

[0035] In addition, in the results of skeleton recognition shown in FIG. 4 and the like, for example, the positions of the shoulder, elbow, and wrist in a plane direction perpendicular to the optical axis direction of the camera can be specified according to the positions of the shoulder, elbow, and wrist in the first self-image. In addition, regarding the positions of the elbow and wrist relative to the shoulder in the optical axis direction of the camera, for example, if the length from the shoulder to the elbow in the first self-image is shorter than the original length from the shoulder to the elbow, it can be estimated that the elbow is approaching the camera side accordingly. The same is true for the elbow to the wrist. In this way, the generation unit 14 may estimate the three-dimensional positions and postures of the skeletons of the arm and palm of the user 30.

[0036] The generating unit 14 may generate a second self-image using the three-dimensional position and posture of the skeleton 31 of the user 30 estimated in this way. The generating unit 14 may generate a second self-image by, for example, changing the line of sight of the result of the skeleton recognition by the skeleton recognition unit 13 by a set angle. Specifically, the generating unit 14 may identify the viewpoint and line of sight corresponding to the position and optical axis direction of the self-image camera in the three-dimensional virtual space and the skeleton 31 of the user based on the result of the skeleton recognition, and change the viewpoint and line of sight as determined in advance. In this way, the generating unit 14 may change the line of sight and also change the viewpoint. More specifically, the generating unit 14 may generate a second self-image in which the line of sight from the hand of the user 30 to the shoulder is changed to the line of sight from the shoulder of the user 30 to the hand, i.e., a second self-image from a first-person perspective, as in the first self-image shown in FIG. 4. Since the positional relationship between the viewpoint and line of sight and the skeleton 31 is relative, the generation unit 14 may change the skeleton of the user 30 in the three-dimensional virtual space, for example, instead of changing the line of sight. The change in the skeleton of the user 30 may be, for example, a rotation of the skeleton of the user 30, and may further include movement. In this way, for example, a change in the angle of the skeleton in the three-dimensional virtual space may also be considered to be a change in the line of sight.

[0037] The generating unit 14 may also arrange a three-dimensional object based on the position and posture of the skeleton of the user 30 in a three-dimensional virtual space, for example. The generating unit 14 may arrange a three-dimensional object of forceps in a three-dimensional virtual space, for example, using the skeleton from the elbow to the fingertip of the user 30, so that the longitudinal direction is along the direction from the elbow to the wrist of the user 30 and the tip is at the position of the palm of the user 30. The generating unit 14 may also arrange a three-dimensional object of forceps using the skeleton of the palm of the user 30, for example, so that the longitudinal direction is along the direction of the arrow A2 in FIG. 5, the angle centered on the longitudinal direction changes according to the direction of the arrow A1 in FIG. 5, and the tip is at the position of the tip of the middle finger of the user 30. This three-dimensional object may be stored in the storage unit 11, for example, and may be read out and used. The three-dimensional object corresponds to, for example, an object to be imitated, and may be, for example, a tool such as forceps or a scalpel, or may include a hand. When the three-dimensional object includes a hand, the generation unit 14 may place a three-dimensional object of a hand having a shape according to the skeleton of the hand indicated by the result of the skeleton recognition in the three-dimensional virtual space.

[0038] Although the case has been described here in which the relative positional relationship between the skeleton of the user 30 and the viewpoint or line of sight is changed, and then the three-dimensional object is arranged according to the skeleton of the user 30, the order may be reversed. For example, the three-dimensional object may be arranged according to the skeleton of the user 30, and then the relative positional relationship between the three-dimensional object and the viewpoint or line of sight may be changed.

[0039] Furthermore, the generation unit 14 may generate a second self-image, which is an image from the viewpoint of the user 30, for example. In this case, the generation unit 14 may identify the user's skeleton 31 including the position of the head of the user 30 in the three-dimensional virtual space based on the result of the skeleton recognition, for example, and may specify the position of the head of the user 30 as a viewpoint and the direction from the viewpoint toward the hand of the user 30 as a line of sight direction. Then, similar to the above description, the generation unit 14 may place a three-dimensional object based on the position and posture of the skeleton of the user 30 in the three-dimensional virtual space.

[0040] In the three-dimensional virtual space, for example, a three-dimensional object corresponding to the right hand of the user 30 and a three-dimensional object corresponding to the left hand of the user 30 may be arranged. In this case, a three-dimensional object corresponding to the right hand of the user 30 may be arranged based on the result of the skeleton recognition of the right hand of the user 30, and a three-dimensional object corresponding to the left hand may be arranged based on the result of the skeleton recognition of the left hand of the user. FIG. 6 is a diagram showing an example of a second self-image including three-dimensional objects 33a and 33b generated in this way. The three-dimensional objects 33a and 33b may correspond to the right hand and left hand of the user 30, respectively. Note that when the three-dimensional objects 33a and 33b are not particularly distinguished from each other, they may be called three-dimensional objects 33. In addition, in the second self-image, the area other than the three-dimensional objects 33a and 33b may be transparent, for example.

[0041] The generation unit 14 may generate a two-dimensional image by rendering the three-dimensional object 33 based on the viewpoint and the line of sight in the three-dimensional virtual space. This two-dimensional image is a two-dimensional image of the three-dimensional object 33 viewed from the viewpoint in the line of sight in the three-dimensional virtual space. The generation unit 14 may, for example, change the position and posture of the three-dimensional object 33 in the three-dimensional virtual space according to the result of the skeleton recognition repeatedly performed by the skeleton recognition unit 13, and repeatedly generate two-dimensional images that are the result of rendering the three-dimensional object 33 after the change. The second self-image may be composed of, for example, a plurality of two-dimensional images generated in this way.

[0042] The size of the three-dimensional object 33 included in the second self-image is determined according to the size of the three-dimensional object in the three-dimensional virtual space, the distance from the viewpoint to the three-dimensional object, and the angle of view at the time of rendering. As an example, the size of the three-dimensional object in the three-dimensional virtual space and the angle of view at the time of rendering are predetermined values, and the distance from the viewpoint to the three-dimensional object may be determined according to the distance from the self-image camera to the part of the user corresponding to the imitation target. The distance from the self-image camera to the part of the user corresponding to the imitation target may be determined according to the size of the part of the user corresponding to the imitation target included in the first self-image. In this case, the user can adjust the size of the three-dimensional object 33 included in the second self-image by, for example, changing the distance between the self-image camera and the user. Also, the user may be able to adjust, for example, the predetermined value. Also, as an example, the size of the three-dimensional object in the three-dimensional virtual space and the angle of view at the time of rendering may be automatically adjusted by the generation unit 14 so that the size of the three-dimensional object 33 included in the second self-image and the size of the imitation target included in the reference image are the same. This automatic adjustment may be performed, for example, once at the start of the generation of the second self-image, or may be performed repeatedly during the generation of the second automatic image. The size of the three-dimensional object 33 and the size of the imitation target being the same may be, for example, strictly the same, or may be the same within a predetermined error range. In addition, the area of ​​the imitation target included in the reference image may be identified by, for example, pattern matching or segmentation.

[0043] The display unit 15 displays the reference image stored in the storage unit 11 and the second self-image generated by the generation unit 14. It is preferable that the display unit 15 displays both images so that the two images can be compared. The display unit 15 may, for example, display a composite image of the reference image and the second self-image. In this case, for example, as shown in FIG. 7, the imitation target 21 and the three-dimensional objects 33a and 33b may be displayed together. Note that in FIG. 7, the three-dimensional objects 33a and 33b are shown by dashed lines so that the imitation target 21 and the three-dimensional objects 33a and 33b can be distinguished from each other. In addition, for example, the second self-image may be composited on the front side, i.e., the upper side, of the reference image. As described above, the second self-image composited into the reference image may be transparent in the area other than the three-dimensional object 33. In addition, the opacity of the area of ​​the three-dimensional object 33 in the second self-image composited into the reference image may be, for example, 100% or less than 100%. When the opacity of the second self-image is greater than 0% and less than 100%, i.e., when the second self-image is translucent, the user will be able to see both the three-dimensional object 33 and the imitation target 21, even if they overlap.

[0044] In addition, the display unit 15 may display both images while switching them in a time-division manner, for example. More specifically, the display unit 15 may repeatedly display the reference image for a first time period and the second self-image for the next second time period. In this case, the display unit 15 may sequentially display the reference image divided for each first time period, sandwiching the display of the second self-image for the second time period. The first and second times are not particularly limited, but may be, for example, within a range of 0.1 to 1 second, respectively. Even in this case, the user can see both images. In addition, the display while switching between the reference image and the second self-image may be considered as repeating the process of displaying the second self-image synthesized in front of the reference image at 0% opacity for the first time period and then displaying it at 100% opacity for the second time period. In this case, the area of ​​the second self-image other than the three-dimensional object 33 may be opaque (for example, a single color such as white). In this case, instead of switching the opacity between 0% and 100%, the opacity may be changed continuously from 0% to 100%. For example, the opacity may be changed continuously within the range from 0% to 100% like a sine wave, sawtooth wave, or triangular wave.

[0045] Display unit 15 may or may not include a display device (such as a liquid crystal display or an organic EL display) that performs the display. The display of the display target may be performed in another device. In that case, display unit 15 may transmit the image of the display target to the outside of the device. Display unit 15 may be realized by hardware, or may be realized by software such as a driver that drives the display device.

[0046] In addition, when the imitation target includes an operation object whose shape changes, the generating unit 14 may generate a second self-image including, for example, a three-dimensional object of the operation object. The operation object whose shape changes is one whose shape changes according to the operation of the operator. The operation object may have an opening and closing part, such as scissors, forceps, tweezers, tongs, and the like, and the opening and closing part may change in shape between an open state and a closed state. In this case, the first self-image may include the hand 32 of the user 30. Then, the generating unit 14 may generate a second self-image including a three-dimensional object of the operation object whose shape changes according to a gesture of the hand 32 of the user 30 included in the first self-image. The gesture may be, for example, the shape of the hand 32 of the user 30, or may be a movement of the shape of the hand 32. The former gesture is static, and the latter gesture is dynamic. For example, the generation unit 14 may identify a gesture of the hand 32 of the user 30 from the first self-image itself, or may identify a gesture of the hand 32 of the user 30 using a result of skeleton recognition in the first self-image. A dynamic gesture may be identified using, for example, a change in the shape of the hand 32 or a change in the shape of the skeleton 31 in the first self-image.

[0047] In the case where the operation object is a pair of forceps having an opening and closing part and a static gesture is specified, the generating unit 14 may generate a second self-image including a three-dimensional object 33 of forceps with an open opening and closing part when the hand 32 is open, particularly when the tip of the index finger and the tip of the thumb of the hand 32 are separated, as shown in Fig. 8A, and may generate a second self-image including a three-dimensional object 33 of forceps with a closed opening and closing part when the tip of the index finger and the tip of the thumb of the hand 32 are in contact with each other, as shown in Fig. 8B. Note that the opening and closing part of the three-dimensional object 33 of the forceps may change between two states, i.e., an open state and a closed state, depending on the gesture, and may also change the degree to which the opening and closing part is open. In the latter case, the generation unit 14 may obtain the degree to which the tip of the index finger and the tip of the thumb of the hand 32 are close to an open state (FIG. 8A) or a closed state (FIG. 8B) using, for example, the result of skeleton recognition in the first self-image, and may change the degree to which the opening and closing part of the three-dimensional object 33 is opened depending on the degree. In this case, for example, the degree of opening and closing may be changed so that the opening and closing part of the three-dimensional object 33 becomes closer to a closed state as the tip of the index finger and the tip of the thumb of the hand 32 become closer to a closed state. In addition, when the operation object is a pair of forceps having an opening and closing part and a dynamic gesture is specified, the generating unit 14 may generate a second self-image including a three-dimensional object 33 of the forceps whose opening and closing part has changed from an open state to a closed state when the hand 32 changes from an open state to a state in which the tip of the index finger and the tip of the thumb are in contact with each other, and may generate a second self-image including a three-dimensional object 33 of the forceps whose opening and closing part has changed from a closed state to an open state when the tip of the index finger and the tip of the thumb of the hand 32 change from a state in which the tips are in contact with each other to a state in which the tips are separated from each other. In this way, the shape of the three-dimensional object 33 included in the second self-image can be changed using a gesture of the hand 32 of the user 30 without using a controller or the like. The position and posture of the three-dimensional object 33 may be changed according to, for example, the position and posture of the hand 32 as a result of skeleton recognition by the skeleton recognition unit 13.

[0048] Next, the operation of the image display device 1 will be described with reference to the flowchart of FIG.

[0049] (Step S101) The display unit 15 starts displaying the reference video stored in the storage unit 11. After this, the display of the reference video by the display unit 15 is assumed to be continued.

[0050] (Step S102) Video acquisition unit 12 judges whether to acquire a first self-image. If the first self-image is to be acquired, the process proceeds to step S103. If not, the process of step S102 is repeated until it is determined that the first self-image is to be acquired. Note that video acquisition unit 12 may, for example, periodically determine that the first self-image is to be acquired.

[0051] (Step S103) The video acquisition unit 12 acquires a first self-image. Note that the acquisition of this first self-image may be, for example, acquisition of one frame constituting the first self-image. The video acquisition unit 12 may receive the first self-image from the camera 901, for example.

[0052] (Step S104) Skeleton recognition unit 13 performs skeleton recognition of user 30 included in the first self-image. This skeleton recognition may be performed, for example, for one frame constituting the first self-image.

[0053] (Step S105) The generator 14 identifies a gesture of the hand 32 of the user 30 using the first self-image or the result of the skeleton recognition. The static gesture may be identified using, for example, one frame or the result of the skeleton recognition performed on one frame. The dynamic gesture may be identified using, for example, multiple consecutive frames or the result of the skeleton recognition performed on multiple consecutive frames.

[0054] (Step S106) The generator 14 generates a second self-image including a three-dimensional object 33 using the result of the skeleton recognition and the result of identifying the gesture of the hand 32 of the user 30. The generation of this second self-image may be, for example, the generation of one frame constituting the second self-image. Furthermore, the shape of the three-dimensional object 33 included in the second self-image may correspond to the identified gesture of the hand 32 of the user 30. Furthermore, this second self-image may be generated so that the relative positional relationship between the reference image camera and the imitation target is the same as the relative positional relationship between the viewpoint of the second self-image and the three-dimensional object 33.

[0055] (Step S107) Display unit 15 displays the generated second self-image together with the reference image. For example, the second self-image and the reference image may be composited and displayed. In this way, user 30 can see both the reference image and the second self-image. Then, the process returns to step S102.

[0056] Note that the order of the processes in the flowchart of Fig. 9 is an example, and the order of the steps may be changed as long as the same results can be obtained. In addition, in the flowchart of Fig. 9, the process ends when the power is turned off or an interrupt is issued to end the process.

[0057] Next, the operation of the image display device 1 according to the present embodiment will be described using a specific example. In this specific example, the reference image is an image of a situation in which a forceps to be imitated 21 is being operated by a surgical robot, as shown in Fig. 3. Also, as shown in Fig. 2, the image display device 1 is realized by a notebook computer 900, and a first self-image including a hand 32 of a user 30 is captured using a camera 901 built into the computer 900.

[0058] First, it is assumed that the user 30 operates the computer 900 to input an instruction to display the reference image and the second self-image. Then, in response to the instruction, the display unit 15 starts displaying the reference image stored in the storage unit 11 on the display device 902 (step S101). In addition, the image acquisition unit 12 acquires a first self-image of the user 30 captured by the camera 901, and passes it to the skeleton recognition unit 13 (steps S102, S103). As shown in FIG. 4, the skeleton recognition unit 13 recognizes the skeleton 31 of the user 30 in the first self-image captured by the camera 901 facing the user 30, and passes the recognized skeleton 31 to the generation unit 14 (step S104). Upon receiving the skeleton 31 as the recognition result, the generation unit 14 identifies the hand gesture of the user 30 using the shape of the hand part of the skeleton 31 (step S105). Furthermore, the generation unit 14 uses the result of the skeleton recognition and the result of the specified gesture to arrange three-dimensional objects 33a and 33b having a shape corresponding to the specified gesture so that the position and posture correspond to the skeleton 31 of the user 30 in the three-dimensional virtual space, and generates a second self-image by rendering the three-dimensional objects 33a and 33b to be an image of the first person viewpoint of the user 30, and passes it to the display unit 15 (step S106). For example, a second self-image as shown in FIG. 6 is generated. Upon receiving the second self-image, the display unit 15 synthesizes the second self-image with the reference image and displays it (step S107). As a result, the user 30 can see the reference image and the second self-image displayed on the display device 902 of the computer 900 as shown in FIG. 2. In this way, by repeatedly acquiring a first self-image, recognizing a skeleton, identifying a gesture, generating a second self-image based on them, and displaying the reference image and the second self-image, user 30 can move three-dimensional objects 33a and 33b of the forceps according to the movement of his or her own hand so as to match the movement of the forceps included in the reference image. Thus, user 30 can train to move three-dimensional objects 33a and 33b in the same way as the imitation targets included in the reference image.

[0059] As described above, according to the image display device 1 of the present embodiment, it is possible to generate and display a second self-image that can be compared with the reference image, even if the same shooting environment as when the reference image was shot is not prepared. Therefore, the user 30 can train to perform the same movement as the movement of the object to be imitated while comparing the reference image with the second self-image. For example, even if the reference image is a first-person viewpoint image, the user 30 can shoot his / her own image using a front camera such as a notebook computer, tablet terminal, or smartphone, and there is no need to prepare a head-mounted camera, so there is an advantage that the cost and time required for this can be reduced. In addition, when changing the shape of the three-dimensional object 33 of the operation target according to the gesture of the hand of the user 30, it is not necessary to use a controller for changing the shape of the three-dimensional object 33, and training can be performed with a simple configuration.

[0060] In the present embodiment, the case where the shape of the three-dimensional object 33 is changed using a gesture of the hand 32 of the user 30 has been described, but this is not necessarily the case. The user 30 may change the shape of the three-dimensional object 33 by operating a controller. In this case, the image display device 1 may further include a reception unit 16 that receives an instruction from a controller 4 operated by the user 30, as shown in FIG. 10. The reception unit 16 may receive an instruction from the controller 4 by wire or wirelessly, for example. The reception unit 16 may or may not include a device for receiving (for example, a communication device, etc.). The reception unit 16 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

[0061] The controller 4 operated by the user 30 may be, for example, as shown in FIG. 11. The controller 4 shown in FIG. 11 has a button 4a, and when the user 30 presses the button 4a, for example, an instruction to close the opening and closing part of the three-dimensional object 33 may be transmitted to the image display device 1. In this case, the generation unit 14 may generate a second self-image including a three-dimensional object of the operation target whose shape changes in response to an instruction received by the reception unit 16. More specifically, when the user 30 presses the button 4a, the generation unit 14 may generate a second self-image including a three-dimensional object 33 whose opening and closing part is closed in response to an instruction to close the opening and closing part received by the reception unit 16. Note that, when the user 30 does not press the button 4a, a second self-image including a three-dimensional object 33 whose opening and closing part is open may be generated. In this way, the three-dimensional object 33 of the operation target can be operated using the controller 4. Therefore, for example, when the reference image is an image of a surgical robot, the user 30 can operate the three-dimensional object 33 to be operated using a controller 4 similar to the controller used to operate the surgical robot, and the user 30 can operate the three-dimensional object 33 in an environment similar to that of a real surgical robot.

[0062] In addition, in the present embodiment, the case where the image display device 1 according to the present embodiment is realized by a notebook computer with a camera or a tablet terminal with a camera has been mainly described, but this is not essential. For example, the image display device 1 according to the present embodiment may be realized by a desktop computer connected to an external camera. In recent years, for web meetings and the like, cameras are sometimes arranged around the display of a desktop computer so as to face a user who is viewing the display. The camera may be used as a self-image camera, and the display may be used as a display for displaying a reference image and a second self-image.

[0063] In addition, in this embodiment, the three-dimensional object may be changed. For example, the imitation target may be changed in the reference image. Specifically, the imitation target may be changed from forceps to a scalpel. In such a case, the three-dimensional object may be changed in the second self-image according to the change in the imitation target in the reference image. This change may be performed, for example, manually. When the three-dimensional object is changed manually, for example, the user may be able to change the three-dimensional object by a hand gesture. In this case, for example, the user may change the three-dimensional object by performing a gesture of wiping his / her hand. In addition, the change of the three-dimensional object may be performed automatically. In this case, for example, the generation unit 14 may identify the type of the imitation target included in the reference image, and a three-dimensional object according to the identified type may be arranged in the three-dimensional virtual space. The type of the imitation target may be identified by, for example, pattern matching or object recognition.

[0064] In the above embodiment, the image display device 1 is described as a stand-alone device, but the image display device 1 may be a stand-alone device or a server device in a server-client system. In the latter case, the image acquisition unit and the display unit may acquire and display images via a communication line.

[0065] In addition, in the above embodiments, each process or function may be realized by centralized processing in a single device or a single system, or may be realized by distributed processing in multiple devices or multiple systems.

[0066] In addition, in the above embodiments, the transfer of information between each component may be performed, for example, by one component outputting information and the other component receiving information if the two components transferring the information are physically different, or, if the two components transferring the information are physically the same, by shifting from a processing phase corresponding to one component to a processing phase corresponding to the other component.

[0067] In the above embodiment, information related to the processing performed by each component, such as information accepted, acquired, selected, generated, transmitted, or received by each component, and information such as thresholds, formulas, and addresses used by each component in processing, may be temporarily or long-term stored in a recording medium (not shown) even if not specified in the above description. Furthermore, each component or a storage unit (not shown) may store information in the recording medium (not shown). Furthermore, each component or a reading unit (not shown) may read information from the recording medium (not shown).

[0068] In the above embodiment, if the information used by each component, such as the thresholds, addresses, and various setting values ​​used by each component in processing, may be changed by the user, the user may be able to change the information as appropriate, even if not specified in the above description. If the information is changeable by the user, the change may be realized, for example, by a reception unit (not shown) that receives a change instruction from the user, and a change unit (not shown) that changes the information in response to the change instruction. The reception unit (not shown) may receive the change instruction from an input device, may receive information transmitted via a communication line, or may receive information read from a predetermined recording medium.

[0069] Furthermore, in the above embodiment, when two or more components included in the image display device 1 have a communication device, an input device, etc., the two or more components may physically have a single device, or may have separate devices.

[0070] In the above embodiment, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component may be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or semiconductor memory. During execution, the program execution unit may execute the program while accessing the storage unit or recording medium. The software that realizes the image display device 1 in the above embodiment is a program such as the following. In other words, this program causes a computer that can access a memory unit in which a reference image, which is an image of the movements of a target whose movements the user is to imitate, to function as an image acquisition unit that acquires a first self-image, which is an image of the user's movements, a skeletal recognition unit that performs skeletal recognition of the user contained in the first self-image, a generation unit that uses the results of skeletal recognition by the skeletal recognition unit to generate a second self-image, which is an image of a three-dimensional object that corresponds to the target and moves in response to the user's movements, so that the relative positional relationship between the reference image camera that captures the reference image and the target and the relative positional relationship between the viewpoint of the image of the three-dimensional object and the three-dimensional object are the same, and a display unit that displays the reference image and the second self-image, wherein the relative positional relationship between the reference image camera and the target is different from the relative positional relationship between the self-image camera that captures the first self-image and the part of the user that corresponds to the target.

[0071] In addition, in the above program, the functions realized by the program do not include functions that can only be realized by hardware, such as an interface card in an acquisition unit that acquires information or a display unit that displays information.

[0072] This program may be executed by being downloaded from a server or the like, or may be executed by reading out a program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, etc.). This program may be used as a program constituting a program product.

[0073] Furthermore, the program may be executed by a single computer or a plurality of computers. That is, the program may be executed by centralized processing or distributed processing.

[0074] FIG. 12 is a diagram showing an example of the configuration of a computer 900 that executes the above program to realize the image display device 1 according to the above embodiment. In FIG. 12, the computer 900 includes a camera 901, a display device 902, a keyboard 903, a pointing device 904 such as a touchpad or a mouse, an MPU (Micro Processing Unit) 911, a ROM 912 for storing programs such as a boot-up program, a RAM 913 connected to the MPU 911 and temporarily storing instructions of application programs and providing a temporary storage space, a storage unit 914 for storing application programs, system programs, and data, and a communication module 915 for providing a connection to a LAN, a WAN, or the like. The MPU 911, the ROM 912, and the like may be connected to each other by a bus. The storage unit 914 may be, for example, a hard disk or a solid state drive (SSD). The camera 901, the display device 902, the keyboard 903, the pointing device 904, and the like may be, for example, devices built into the computer 900, or may be external devices.

[0075] A program for causing the computer 900 to execute the functions of the image display device 1 according to the above-described embodiment may be loaded into the RAM 913 at the time of execution. Note that the program may be loaded directly from the storage unit 914 or a network, for example.

[0076] The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 900 to execute the functions of the image display device 1 according to the above embodiment. The program may include only an instruction portion that calls appropriate functions or modules in a controlled manner to obtain a desired result. How the computer 900 operates is well known, and a detailed description will be omitted.

[0077] The above-mentioned embodiments are merely examples for specifically implementing the present invention, and are not intended to limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims, not by the description of the embodiments, and is intended to include modifications within the literal scope of the claims and the scope of equivalent meanings. [Explanation of symbols]

[0078] 1. Video display device 11 Storage section 12 Image acquisition section 13 Skeleton Recognition Unit 14 Generation part 15 Display section 16 Reception Department

Claims

1. A storage unit that stores a reference video, which is a video of a target motion that a user is to imitate; an image acquisition unit that acquires a first self-image, which is an image of a user's actions; a skeleton recognition unit that recognizes a skeleton of the user included in the first self-image; a generation unit that generates a second self-image, which is a three-dimensional object corresponding to the object to be imitated and which moves in response to the user's actions, using a result of the skeleton recognition by the skeleton recognition unit, such that a relative positional relationship between a reference image camera that captures the reference image and the object to be imitated and a relative positional relationship between a viewpoint of the image of the three-dimensional object and the three-dimensional object are the same; a display unit that displays the reference image and the second self-image, An image display device in which the relative positional relationship between the reference image camera and the object to be imitated is different from the relative positional relationship between the self-image camera that captures the first self-image and the part of the user that corresponds to the object to be imitated.

2. The image display device according to claim 1 , wherein the generation unit generates a second self-image by changing a line of sight direction resulting from skeleton recognition by the skeleton recognition unit by a set angle.

3. the reference image is an image from a viewpoint of an imitation subject who moves the imitation target, The image display device according to claim 1 , wherein the generating unit generates a second self-image that is an image from a viewpoint of the user.

4. The imitation target includes an operation target object whose shape changes, the first self-image includes a hand of the user; The image display device according to claim 1 , wherein the generating unit generates a second self-image including a three-dimensional object as an operation target whose shape changes in response to a gesture of the user's hand included in the first self-image.

5. The imitation target includes an operation target object whose shape changes, a reception unit that receives an instruction from a controller operated by the user, The image display device according to claim 1 , wherein the generating unit generates a second self-image including a three-dimensional object of an operation target whose shape changes in response to the instruction received by the receiving unit.

6. 6. The image display device according to claim 1, wherein the display unit combines the reference image and the second self-image and displays the combined image.

7. An image display device as described in any one of claims 1 to 5, wherein the self-image camera and the display device on which the reference image and the second self-image are displayed are positioned so that the direction from the self-image camera to the subject to be photographed in the optical axis direction is opposite to the direction of the line of sight when looking directly at the display device.

8. A video display method that is processed using a storage unit that stores a reference video, which is a video of a target motion that a user is to imitate, a video acquisition unit, a skeleton recognition unit, a generation unit, and a display unit, comprising: The image acquisition unit acquires a first self-image, which is an image of a user's movement; a step of performing skeleton recognition of the user included in the first self-image by the skeleton recognition unit; a step in which the generation unit generates a second self-image, which is a three-dimensional object corresponding to the object to be imitated and which is an image of the three-dimensional object that moves in response to the user's actions, using the result of the skeleton recognition such that a relative positional relationship between a reference image camera that captures the reference image and the object to be imitated and a relative positional relationship between a viewpoint of the image of the three-dimensional object and the three-dimensional object are the same; The display unit displays the reference image and the second self-image, An image display method in which the relative positional relationship between the reference image camera and the object to be imitated is different from the relative positional relationship between the self-image camera that captures the first self-image and the part of the user that corresponds to the object to be imitated.

9. A computer that can access a storage unit that stores a reference video, which is a video of a target motion that a user is to imitate, an image acquisition unit that acquires a first self-image which is an image of the user's actions; a skeleton recognition unit that recognizes a skeleton of the user included in the first self-image; a generation unit that generates a second self-image, which is a three-dimensional object corresponding to the object to be imitated and which moves in response to the user's actions, using a result of the skeleton recognition by the skeleton recognition unit, such that a relative positional relationship between a reference image camera that captures the reference image and the object to be imitated and a relative positional relationship between a viewpoint of the image of the three-dimensional object and the three-dimensional object are the same; a display unit that displays the reference image and the second self-image; A program in which the relative positional relationship between the reference image camera and the object to be imitated is different from the relative positional relationship between the self-image camera that captures the first self-image and the part of the user that corresponds to the object to be imitated.

Images