Video synthesizer, video synthesis method and program

Abstract

To provide a video synthesizer capable of supporting a movement object that a user moves to follow a simulation object included in a reference video.SOLUTION: A video synthesizer 1 includes a storage part 11 for storing a reference video, the video of movement of a simulation object whose movement is to be simulated by a user, a video acquisition part 12 for acquiring a self-video, the video of movement of a movement object to be moved by the user, a synthesis part 13 for generating a synthetic video formed by synthesizing the reference video and the self-video, a guide generation part 14 for generating a guide for making the simulation object follow the movement object based on the simulation object included in the reference video and the movement object included in the self-video, and an output part 15 for outputting the synthetic video and the guide.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image synthesizing device that synthesizes and outputs a reference image, which is an image of an object to be imitated, and a self-image, which is an image of an object to be moved by a user. [Background technology]

[0002] Conventionally, there is known a learning support device that displays a reference image, which is an image of a target movement that a learner is to imitate, and a self-image, which is an image of the learner's movement, for learning movements such as surgery (see, for example, Patent Document 1). By referring to such a display, a learner can train to perform the same movements as the target movement. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-071443 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, if the learner is a beginner, he or she may not be able to properly make the object that he or she is controlling follow the movements of the object to be imitated contained in the reference video, which may reduce the effectiveness of learning.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to provide an image synthesis device etc. that can assist a user in making an object being operated follow an object to be imitated contained in a reference image. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, an image synthesis device according to one aspect of the present invention includes a memory unit for storing a reference image, which is an image of the movement of an object to be imitated by a user; an image acquisition unit for acquiring a self-image, which is an image of the movement of an object to be performed by the user; a synthesis unit for generating a synthetic image by synthesizing the reference image and the self-image; a guide generation unit for generating a guide for causing the object to follow the object to be imitated based on the object to be imitated contained in the reference image and the object of movement contained in the self-image; and an output unit for outputting the synthetic image and the guide.

[0007] With this configuration, a guide for making the action target follow the imitation target is output, thereby supporting the action target that the user is moving to make follow the imitation target included in the reference video. By referring to the guide, the user can more easily make the action target follow the imitation target, thereby improving the effectiveness of learning.

[0008] In addition, in an image synthesis device according to one aspect of the present invention, the guide generation unit may generate a guide indicating the following direction in the forward and backward directions based on the user, depending on the relationship between the size of the object to be imitated in the reference image and the size of the object to be moved in the self-image.

[0009] With this configuration, the user can refer to the guide to make the motion target follow the imitation target in the forward and backward directions relative to the user.

[0010] In addition, in an image synthesis device according to one aspect of the present invention, the guide generation unit may generate a guide indicating the following direction in the up / down and left / right directions based on the user, depending on the positional relationship between a representative position of the object to be imitated in the reference image and a representative position of the object to be moved in the self-image.

[0011] With this configuration, the user can refer to the guide to cause the motion target to follow the imitation target in the up-down and left-right directions relative to the user.

[0012] In addition, in an image synthesis device according to one aspect of the present invention, the guide generation unit may use multiple feature points of the object to be imitated in the reference image to obtain a first relative positional relationship between the object to be imitated and the camera that captured the reference image, use multiple feature points of the object to be moved in the self-image to obtain a second relative positional relationship between the object to be moved and the camera that captured the self-image, use the first and second relative positional relationships to obtain a third relative positional relationship between the object to be imitated and the object to be moved, and generate a guide using the third relative positional relationship.

[0013] Such a configuration allows for more accurate guide generation.

[0014] In addition, in an image synthesis device according to one aspect of the present invention, the object to be imitated and the object to be moved change shape, and the guide generation unit may generate a guide for causing the shape of the object to be imitated to follow the shape of the object to be imitated, using the shape of the object to be imitated contained in the reference image and the shape of the object to be moved contained in the self-image.

[0015] With this configuration, the user can make the shape of the operation target follow the shape of the object to be imitated by referring to the guide.

[0016] In addition, in an image synthesis device according to one aspect of the present invention, a guide generation unit may generate a guide figure that is a guide, a synthesis unit may generate a synthetic image including the guide figure, and an output unit may output the synthetic image including the guide figure.

[0017] With this configuration, the user can make the action target follow the imitation target by referring to the guide figure included in the composite image.

[0018] In addition, a video synthesis method according to one aspect of the present invention is a video synthesis method processed using a memory unit in which a reference video, which is a video of an object to be imitated by a user, is stored, a video acquisition unit, a synthesis unit, a guide generation unit, and an output unit, and includes a step in which the video acquisition unit acquires a self-video, which is a video of an object to be moved by the user, a step in which the synthesis unit generates a synthesized video by synthesizing the reference video and the self-video, a step in which the guide generation unit generates a guide for causing the object to follow the object to be imitated based on the object to be imitated contained in the reference video and the object to be moved contained in the self-video, and a step in which the output unit outputs the synthesized video and the guide. Effect of the Invention

[0019] According to an image synthesizing device or the like according to one aspect of the present invention, it is possible to provide assistance to a user in causing an object to be operated by the user to follow an object to be imitated that is included in a reference image. [Brief description of the drawings]

[0020] [Figure 1] FIG. 1 is a block diagram showing a configuration of a video synthesizing device according to an embodiment of the present invention; [Diagram 2] FIG. 2 shows an example of a usage state of the image synthesizer according to the embodiment. [Diagram 3] FIG. 2 shows an example of a reference video in the embodiment. [Figure 4] FIG. 13 shows an example of a self-image in the embodiment. [Diagram 5] FIG. 13 is a diagram showing an example of a composite image including a guide figure according to the embodiment; [Figure 6] FIG. 13 is a diagram showing an example of a composite image including a guide figure according to the embodiment; [Figure 7] FIG. 13 shows an example of a self-image in the embodiment. [Figure 8] FIG. 2 shows an example of a reference video in the embodiment. [Figure 9] A flowchart showing the operation of the image synthesizing device according to the embodiment. [Figure 10]FIG. 2 shows an example of a configuration of a computer according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0021] The video synthesis device and the video synthesis method according to the present invention will be described below using embodiments. In the following embodiments, components and steps with the same reference numerals are the same or equivalent, and repeated description may be omitted. The video synthesis device according to the present embodiment generates a guide for making a motion target operated by a user follow a imitation target included in a reference video.

[0022] FIG. 1 is a block diagram showing the configuration of the image synthesizing device 1 according to the present embodiment. FIG. 2 is a diagram showing a situation in which a user 30, who is a learner, is learning the movement of a target to be imitated using the image synthesizing device 1. The image synthesizing device 1 according to the present embodiment includes a storage unit 11, an image acquisition unit 12, a synthesis unit 13, a guide generation unit 14, and an output unit 15. Note that the image synthesizing device 1 may be realized by a computer 900 as shown in FIG. 10 described later, as an example, or may be realized by dedicated hardware. In the present embodiment, the former case will be mainly described. Note that, as shown in FIG. 2, the front-rear direction, the up-down direction, and the left-right direction based on the user 30 who is looking directly at the display device 902 may be referred to as the front-rear direction, the up-down direction, and the left-right direction, respectively, in the following description.

[0023] The storage unit 11 stores a reference image, which is an image of the motion of a target object whose motion the user 30 imitates. The user 30 is a learner who learns the motion while referring to the reference image. The motion that the user 30 learns may be, for example, a motion such as surgery, a motion of work in a factory, a motion of nursing care or a hotel, a motion such as cooking, a motion for creating a work such as a craft, a motion such as sports, a motion such as calligraphy, a motion of tying a rope, or other motions. The target object may be, for example, a part of the body of the imitated person, or an object operated by the imitated person. The imitated person may be, for example, a teacher of the user 30 who is a learner, and may be skilled in the motion that the learner is to learn. In addition, a part of the body of the imitated person may include, for example, the hand of the imitated person. In addition, the object operated by the imitated person may be, for example, a tool such as forceps, scalpel, tweezers, scissors, or brush that the imitated person has. The reference image may be, for example, an image of a situation in which an imitation target such as forceps is being operated by a surgical robot. The reference image is usually an image captured by a camera, but may be a CG (Computer Graphics) image equivalent to the image captured by the camera.

[0024] The reference image may be, for example, an image from the viewpoint of the person to be imitated who operates the object to be imitated, that is, an image from the first-person viewpoint of the person to be imitated. In this case, the reference image may be, for example, an image taken by a camera placed near the head of the person to be imitated. For example, the camera may be a head-mounted camera. In this embodiment, a case will be mainly described in which the reference image shown in FIG. 3 including the object to be imitated 41, which is a pair of forceps used by the person to be imitated, is stored in storage unit 11.

[0025] In the storage unit 11, for example, the entire reference video may be stored, or only a part of the reference video may be stored. As an example, when the video synthesis device 1 displays a reference video while receiving it from the outside, a part of the reference video, that is, the most recently received reference video, may be stored in the storage unit 11, and may be read out and displayed, and may be overwritten sequentially. The storage unit 11 may store information other than the reference video. For example, a self-video image acquired by the video acquisition unit 12 may be stored in the storage unit 11.

[0026] The process by which information is stored in the storage unit 11 is not important. For example, information may be stored in the storage unit 11 via a recording medium, information transmitted via a communication line or the like may be stored in the storage unit 11, or information inputted via a device such as a camera may be stored in the storage unit 11. The storage unit 11 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory or a magnetic disk.

[0027] The image acquisition unit 12 acquires a self-image, which is an image of the action of an action target that is an object to be operated by the user 30. The image acquisition unit 12 may be, for example, an optical device such as a camera that captures an image, or may acquire an image captured by an optical device such as a camera. In this embodiment, a case where the image acquisition unit 12 receives an image captured by a camera 901 attached to a camera stand 951 will be mainly described. The camera 901 that captures the self-image may be, for example, a camera that is installed in the shooting environment by a support unit such as the camera stand 951, or may be a camera that is attached to the user 30, such as a head-mounted camera. The action target corresponds to the imitation target. Usually, the action target and the imitation target are of the same type. Therefore, the action target may be, for example, a part of the body of the user 30, or may be an object operated by the user 30. The part of the body of the user 30 may include, for example, the hand of the user 30. In addition, the object operated by the user 30 may be, for example, a tool such as forceps, scalpel, tweezers, scissors, or brush that the user 30 has. For example, if the object to be imitated is forceps, it is preferable that the object to be operated is also forceps. If the object to be imitated and the object to be operated are objects operated by the imitated person or user 30, the object to be imitated and the object to be operated may be, for example, the same. It is preferable that the same object to be imitated and the object to be operated are, for example, the same type and the same size.

[0028] As an example, the self-image may be an image from the viewpoint of the user 30 who operates the operation target, that is, an image from the first-person viewpoint of the user 30. In this case, the self-image may be an image taken by a camera 901 arranged near the head of the user 30, for example, as shown in FIG. 2. It is preferable that the optical axis of the camera 901 faces the operation target. In this embodiment, a case where the self-image shown in FIG. 4 including the operation target 51, which is the forceps used by the user 30, is acquired by the image acquisition unit 12 will be mainly described. Note that the self-image is usually an image taken by a camera, but it may also be a CG image corresponding to the image taken by the camera. As an example, when the reference image is an image of a situation in which an imitation target such as forceps is being operated by a surgical robot, the self-image may be a real image of a situation in which the operation target such as forceps is being operated by the surgical robot in response to the operation of the user 30, or a CG image corresponding to that image. In the latter case, a self-image that is a CG image of a situation in which an operation target such as forceps is being operated by a surgical robot in response to the operation of the user 30 may be generated in real time.

[0029] It is preferable that the relative positional relationship between the reference image camera that captures the reference image and the imitation target is the same as or close to the relative positional relationship between the self-image camera that captures the self-image and the motion target. It is also preferable that each frame included in the reference image and each frame included in the self-image are, for example, the same size. The frames being the same size may mean that the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the frames are the same.

[0030] The synthesis unit 13 synthesizes the reference image and the self-image to generate a synthetic image. For example, the synthesis unit 13 may generate a synthetic image by synthesizing the reference image and the self-image so that the synthesis ratio of both images is a predetermined ratio. As an example, the synthesis unit 13 may synthesize a self-image with a predetermined opacity in front of the reference image, that is, on the upper side. In this case, the opacity of the self-image is set to 50%, so that the reference image and the self-image can be synthesized at a ratio of 50% each. The synthesis ratio may be changed, for example, in response to an instruction from the user 30. Furthermore, the synthesis unit 13 may synthesize both images so that the synthesis ratio of the reference image and the self-image changes continuously and repeatedly over time. In this case, for example, the synthesis ratio may change periodically, such as a sine wave, a triangular wave, or a sawtooth wave. The change in the synthesis ratio may be, for example, the opacity of one of the images. As an example, the synthesis unit 13 may generate a synthetic image by synthesizing the reference image and the self-image as shown in FIG. 5 and FIG. 6. In Fig. 5 and Fig. 6, the moving target 51 is indicated by a dashed line so that the imitation target 41 and the moving target 51 can be distinguished from each other. Note that guide figures 61, 62 in Fig. 5 and Fig. 6 will be described later. Furthermore, the synthesis unit 13 may generate a synthesized image in which both images are displayed while being switched in a time-division manner. More specifically, the synthesis unit 13 may generate a synthesized image in which the reference image is displayed for a first period of time, and then the second self-image is displayed for a second period of time, and this is repeated.

[0031] Furthermore, when the guide generating unit 14 generates a guide to be displayed such as a guide graphic or character string, the synthesis unit 13 may generate a synthetic video including the display target such as the guide graphic or character string generated by the guide generating unit 14. In this case, the guide is displayed together with the reference video and the self-video. In this embodiment, the case where a synthetic video including the guide graphic generated by the guide generating unit 14 is generated will be mainly described.

[0032] The guide generating unit 14 generates a guide for making the motion target follow the imitation target based on the imitation target included in the reference image and the motion target included in the self image. This guide may be, for example, a guide to be displayed such as a guide figure or a character string, a guide to be output as sound such as voice, or other guide for making the motion target follow the imitation target. In this embodiment, (1) generation of a guide indicating a following direction using the size of the imitation target and the motion target, (2) generation of a guide indicating a following direction using the position of the imitation target and the motion target, (3) generation of a guide for shape following using the shape of the imitation target and the motion target, and (4) generation of a guide indicating a following direction using a relative positional relationship between the camera and the imitation target and the motion target will be described. The guide generating unit 14 may generate any one of the guides (1) to (4), or may generate two or more of the guides (1) to (4). As an example, the guide generating unit 14 may generate the guides (1) and (2), may generate the guides (1) to (3), or may generate the guides (3) and (4).

[0033] (1) Generation of a guide showing the direction to follow using the size of the object to be imitated and the object to be moved The guide generating unit 14 may generate a guide indicating a following direction in the forward / backward direction based on the user 30 according to the relationship between the size of the imitation target in the reference image and the size of the motion target in the self-image. The following direction may be, for example, a moving direction for making the motion target follow the imitation target. When the motion target in the self-image is in a following state in which the motion target in the reference image can appropriately follow the motion target in the self-image, it is considered that the sizes of both are approximately the same. The following state may be, for example, a case in which the imitation target and the motion target are displayed so as to overlap in the composite image. On the other hand, when the motion target is located behind the following state, the size of the motion target is larger than the size of the imitation target. Conversely, when the motion target is located forward than the following state, the size of the motion target is smaller than the size of the imitation target. Therefore, the guide generating unit 14 can specify which side of the motion target must be moved in the forward / backward direction based on the user 30 in order to bring the position of the motion target closer to the position of the motion target according to the relationship between the size of the imitation target and the size of the motion target, that is, the size relationship between the two, and can generate a guide according to the result of the specification. For example, when the size of the action target is larger than the size of the imitation target, the guide generating unit 14 may generate a guide for moving the action target forward with respect to the user 30, i.e., toward the back of the user 30. Also, for example, when the size of the action target is smaller than the size of the imitation target, the guide generating unit 14 may generate a guide for moving the action target backward with respect to the user 30, i.e., toward the front of the user 30.

[0034] The guide generating unit 14 may specify the area of ​​the imitation target in the reference image or the area of ​​the action target in the self image, for example, by segmentation or template matching. For example, when the imitation target or action target is forceps, the area of ​​the forceps may be specified by segmentation or template matching. When segmentation is performed, for example, an area previously classified as forceps may be set as the area of ​​the imitation target or action target, or an area with movement may be set as the area of ​​the imitation target or action target. The area of ​​the imitation target or action target may be, for example, the entire area or a part of the area. The guide generating unit 14 may generate the guide as described above using the size of the area of ​​the imitation target or action target specified in this way, or may generate the guide as described above using an area configured by predetermined feature points in the specified area of ​​the imitation target or action target. In the latter case, the guide generating unit 14 may, for example, specify the area of ​​the imitation target 41 in the reference image as shown in Fig. 3, specify a triangular area D41 in the specified area with the fulcrum P41, the tip P42 of one gripping part, and the tip P43 of the other gripping part as vertices, and may similarly specify the area of ​​the action target 51 in the self-image as shown in Fig. 4, specify a triangular area D51 in the specified area with the fulcrum P51, the tip P52 of one gripping part, and the tip P53 of the other gripping part as vertices. Then, the guide generating unit 14 may compare the sizes of the areas D41 and D51, and if the area D51 is larger, generate a guide for moving the action target 51 forward with respect to the user 30, and if the area D41 is larger, generate a guide for moving the action target 51 backward with respect to the user 30. The sizes of the regions D41 and D51 may be compared, for example, in terms of their areas or in terms of the lengths of the longest sides of the triangles.

[0035] When the guide generating unit 14 generates a guide figure, for example, an arrow guide figure 61 shown in FIG. 5 may be generated. The guide figure 61 is a guide for moving the action target 51 in the forward direction based on the user 30. For example, the guide figure may be added to a position in the composite image that does not overlap with the imitation target or the action target. When the area or position of the imitation target or the action target is specified when generating the guide, the guide generating unit 14 may instruct the composition unit 13 to add the guide figure to a position away from the specified area of ​​the imitation target or the action target. When a character string guide or a voice guide is generated, for example, a character string or a voice guide indicating the moving direction of the action target, such as "forward side" or "back side", may be generated. In addition, when the size of the imitation target and the action target are the same, for example, the generation of the guide (1) may not be performed. As an example, even when the sizes of the two are sufficiently close, it may be determined that the sizes of the two are the same.

[0036] (2) Generating a guide showing the direction to follow using the positions of the object to be imitated and the object to be moved The guide generating unit 14 may generate a guide indicating the following direction in the up-down direction and the left-right direction based on the user 30 according to the positional relationship between the representative position of the imitation target in the reference image and the representative position of the action target in the self-image. Here, the representative position of the imitation target or the action target may be, for example, the position of the center of gravity of the area of ​​the imitation target or the action target, or may be a predetermined position of the imitation target or the action target. The area of ​​the imitation target or the action target may be, for example, the entire area or a part of the area. The area may be specified, for example, as the above-mentioned areas D41 and D51. The predetermined position may be, for example, the position of the fulcrum P41 and P51 shown in FIG. 3 and FIG. 4.

[0037] When the motion target in the self-image is in a tracking state where it can appropriately track the imitation target in the reference image, the representative position of the imitation target and the representative position of the motion target are considered to be the same. For example, even if the two representative positions are sufficiently close, it may be determined that the two representative positions are the same. On the other hand, when the motion target is located at a position away from the imitation target in at least one of the directions of up, down, left and right based on the user 30, the representative position of the motion target will be located at a position away from the representative position of the imitation target accordingly. Therefore, the guide generating unit 14 may specify in which direction the motion target must be moved in the up, down and left and right directions based on the user 30 in order to bring the position of the motion target closer to the position of the imitation target according to the positional relationship between the representative position of the imitation target and the representative position of the motion target, and generate a guide according to the specified result. The positional relationship may be, for example, a vector from the representative position of the motion target to the representative position of the imitation target. The positional relationship between the two representative positions may be, for example, a positional relationship when the two representative positions are placed in the same coordinate system such as the coordinate system of the composite image.

[0038] The guide generating unit 14 may generate, as a guide according to the positional relationship between the representative position of the imitation target and the representative position of the action target, a guide indicating, for example, the direction of a vector from the representative position of the action target to the representative position of the action target. The guide generating unit 14 may also generate a guide indicating, for example, the degree of movement according to the magnitude of the vector, that is, whether more movement is required or less movement is sufficient. When the guide is a guide figure, the guide generating unit 14 may generate, for example, a guide figure that is an arrow figure indicating the direction of the vector, or may generate a guide figure that is an arrow figure indicating the magnitude of the vector. When a character string guide or a voice guide is generated, for example, a character string guide or a voice guide indicating the movement direction of the action target, such as "upward" or "right side", may be generated.

[0039] For example, when a vector from the representative position of the motion target to the representative position of the imitation target is a leftward vector in the image and the guide is a guide figure, the guide generating unit 14 may generate an arrow guide figure 62 shown in Fig. 6. The guide figure 62 is a guide for moving the motion target 51 in the left direction based on the user 30. Note that, when the representative positions of the imitation target and the motion target match, for example, the generation of the guide (2) may not be performed. As an example, when the representative positions of both are sufficiently close to each other, it may be determined that the representative positions of both match.

[0040] In addition, when generating the guides (1) and (2), the guide generating unit 14 may, for example, first generate the guide (1), and after the alignment of the imitation target and the action target in the front-rear direction of the user 30 is completed according to the guide, generate the guide (2), or may generate the guides (1) and (2) together. In this embodiment, the former case will be mainly described. In addition, when generating the guides (1) and (2) together, the guide generating unit 14 may generate a guide indicating the following direction in the front-rear direction according to the relationship of the sizes of the imitation target and the action target in the video, as described above, and may generate a guide indicating the following direction in the up-down direction and the left-right direction according to the positional relationship of the representative positions of the imitation target and the action target in the video. In this case, for example, two independent guides (for example, two arrows, etc.) may be generated, or one guide (for example, one arrow indicating the following direction in the front-rear direction, and the up-down direction and the left-right direction) may be generated.

[0041] (3) Creating guides for shape tracking using the shapes of the object to be imitated and the object to be moved In the case where the object to be imitated or the object to be moved changes shape, the guide generating unit 14 may generate a guide for making the shape of the object to be imitated follow the shape of the object to be imitated by using the shape of the object to be imitated included in the reference image and the shape of the object to be moved included in the self-image. The object to be imitated or the object to be moved that changes shape may change in response to the operation of the imitated person or the user 30, for example. The object to be imitated or the object to be moved that changes shape may have an opening and closing part, such as scissors, forceps, tweezers, tongs, etc., and the opening and closing part may change shape between an open state and a closed state.

[0042] The guide generating unit 14 may specify the shape of the imitation target or the action target, for example, by using a learned learning model or template matching. For example, when the imitation target or the action target is something that opens and closes like forceps, the shape of the open state and the shape of the closed state may be specified, and a shape between the two may also be specified. The shape between the open state and the closed state may be, for example, a shape indicated by a ratio of the open state or a ratio of the closed state. The ratio of the open state may be, for example, 100% in the open state and 0% in the closed state, and may be closer to 100% as it approaches the open state. When specifying the shape by using a learned learning model or template matching, for example, it may be specified by a method similar to object recognition whether the image includes an imitation target or action target with an open state shape or an imitation target or action target with a closed state shape. Also, when specifying the shape indicated by the ratio of the open state, object recognition including the ratio may be performed.

[0043] When the action target in the self-image is in a tracking state where the action target in the reference image can be properly tracked, the shape of the action target and the shape of the action target are the same. On the other hand, when the user 30 is unable to make the shape of the action target follow the shape of the action target, the shapes of the action target and the action target are different. When it is determined that the shape of the action target and the shape of the action target do not match, the guide generating unit 14 may use the shape of the action target and the shape of the action target to specify a change in shape required to make the shape of the action target follow the shape of the action target, and generate a guide accordingly. For example, when the shape of the action target 41 is in an open state as shown in FIG. 3 and the shape of the action target 51 is in a closed state as shown in FIG. 7, and the two shapes do not match, the guide generating unit 14 may generate a guide to change the shape of the action target 51 to a shape specified for the action target 41 (for example, the open state in FIG. 3). This guide may also be, for example, a guide figure. The guide figure indicating that an open motion target is to be closed may be, for example, a figure of two arrows arranged with their tips facing each other in the left-right direction, and the guide figure indicating that a closed motion target is to be opened may be, for example, a figure of both arrows arranged in the left-right direction. When a text guide or a voice guide is generated, a text guide or a voice guide indicating a change in the shape of the motion target, such as "open" or "close", may be generated. When the shapes of the imitation target and the motion target match, for example, the generation of the guide (3) may not be performed.

[0044] (4) Generating a guide showing the direction to follow using the positional relationship between the camera and the object to be imitated or the object to be moved In the generation of the guide described in (1) and (2) above, depending on the angle of the imitation target or the motion target included in the video, an accurate guide may not be generated. Therefore, in order to generate a more accurate guide, the relative positional relationship between the camera and the imitation target and the motion target may be used. In this case, the guide generating unit 14 may obtain a first relative positional relationship between the imitation target and the camera that captured the reference video using a plurality of feature points of the imitation target in the reference video, obtain a second relative positional relationship between the motion target and the camera that captured the self-video using a plurality of feature points of the motion target in the self-video, obtain a third relative positional relationship between the imitation target and the motion target using the first and second relative positional relationships, and generate a guide using the third relative positional relationship.

[0045] As an example, the feature points of the imitation target or the motion target may be feature points P44 and P45 corresponding to a predetermined vertex of one of two members intersecting at a fulcrum constituting the imitation target 41, which is a pair of forceps, and feature point P46 corresponding to the tip of the grip of the member, as shown in FIG. 8. Even if the shape of the imitation target or the motion target changes, it is preferable that the positional relationship of the multiple feature points does not change according to the change in the shape. In addition, it is preferable that the number of multiple feature points used to acquire the relative positional relationship is three or more, and it is preferable that the multiple feature points do not exist on a straight line. It is also preferable that the distance between two feature points in the multiple feature points is known. In addition, the relative positional relationship may be represented by a homogeneous transformation matrix, as an example. For example, the first relative positional relationship between the imitation target and the camera that captured the reference video may be a homogeneous transformation matrix that indicates the transformation between the local coordinate system of the imitation target and the local coordinate system of the camera that captured the reference video. The same applies to other relative positional relationships. Note that a method for acquiring a homogeneous transformation matrix indicating the transformation between the local coordinate system of an object having the above-mentioned three or more feature points included in a captured image and the local coordinate system of the camera is already known, and a detailed description thereof will be omitted. Also, the multiple feature points of the imitation target and the multiple feature points of the motion target are usually the same feature points, but this is not necessary. In the latter case, different feature points may be used for the imitation target and the motion target.

[0046] Also, if the local coordinate system of the camera that captured the reference image and the local coordinate system of the camera that captured the self-image are the same, the third relative positional relationship between the imitation target and the motion target can be obtained using the first and second relative positional relationships. In addition, in a composite image in which the reference image and the self-image are composited, the local coordinate systems of both cameras can be considered to be the same. Therefore, when the relative positional relationship is a homogeneous transformation matrix, for example, for two homogeneous transformation matrices that respectively correspond to the first and second relative positional relationships, a homogeneous transformation matrix corresponding to the third relative positional relationship can be generated by multiplying the inverse matrix of one homogeneous transformation matrix by the other homogeneous transformation matrix.

[0047] The third relative positional relationship can, for example, tell the positional relationship between the local coordinate system of the object to be imitated and the local coordinate system of the object to be moved. The first relative positional relationship or the second relative positional relationship can tell the positional relationship between the camera coordinate system, which is the local coordinate system of the camera, and the local coordinate system of the object to be imitated or the local coordinate system of the object to be moved. Therefore, the guide generating unit 14 can specify the degree of movement and rotation required for the local coordinate system of the object to be imitated to match the local coordinate system of the object to be imitated in the camera coordinate system by using the first and third relative positional relationship or the second and third relative positional relationship. The guide generating unit 14 may generate a guide using the result of the specification. Note that the local coordinate system of the object to be imitated matches the local coordinate system of the object to be imitated, meaning that the state is a tracking state.

[0048] For example, the guide generating unit 14 may specify a vector in the camera coordinate system indicating the direction and degree of movement of the motion target so that the local coordinate system of the motion target coincides with the local coordinate system of the imitation target. If the z-axis direction of the camera coordinate system is the optical axis direction of the camera, the guide generating unit 14 can specify in which direction in the z-axis direction the motion target must be moved in order to follow the imitation target, depending on the direction of the specified vector in the z-axis direction. Then, the guide generating unit 14 may generate a guide indicating the following direction in the forward and backward directions based on the user 30, depending on the result of the specification. For example, when the motion target must be moved in the z-axis direction in a direction approaching the camera, the guide generating unit 14 may generate a guide to move the motion target backward based on the user 30.

[0049] In addition, when the x-axis direction of the camera coordinate system is the left-right direction of the image and the y-axis direction of the camera coordinate system is the up-down direction of the image, the guide generating unit 14 can specify in which direction, the x-axis direction or the y-axis direction, the action target must be moved in order to follow the imitation target, according to the orientations of the x-axis and y-axis directions of the specified vector. Then, the guide generating unit 14 may generate a guide indicating the following direction in the up-down and left-right directions based on the user 30 according to the specified result. For example, when the action target must be moved to the right in the x-axis direction of the image, the guide generating unit 14 may generate a guide to move the action target to the right based on the user 30, and when the action target must be moved to the upward direction of the image in the y-axis direction, the guide generating unit 14 may generate a guide to move the action target to the upward direction based on the user 30.

[0050] When acquiring the relative positional relationship, it is possible to acquire not only the parallel movement between the imitation target and the motion target, but also the relative positional relationship including the rotation. Therefore, the guide generating unit 14 may generate a guide that indicates, for example, not only the moving direction of the motion target, but also the rotating direction of the motion target.

[0051] In addition, when the object to be imitated or the object to be moved is elongated like forceps, it may be difficult to set three feature points that are not on a straight line. In such a case, for example, a marker for identifying the feature point may be attached to the object to be imitated or the object to be moved, and the feature point may be identified using the marker. The marker may be, for example, a polygonal marker such as a square. In this case, each vertex of the marker may be the feature point.

[0052] In addition, since the orientation of the imitation target or the action target changes in the video, the same feature points cannot always be used. For example, the imitation target or the action target may be rotated, making it impossible to identify a predetermined feature point in the video. Therefore, the feature points of the imitation target and the action target may be changed, for example, for each period of the reference video. In this case, for example, feature points to be used for each period of the reference video may be set in advance. The setting may be performed, for example, manually, or automatically using a learning model or the like.

[0053] Furthermore, when the guide generating unit 14 specifies the area, representative position, and feature point position of the imitation target or action target, the guide generating unit 14 may specify the area, etc. of the imitation target using a reference image, and specify the area, etc. of the action target using a self-image. In this case, the guide generating unit 14 may specify the area, position, etc. using the same coordinate system in both images, for example. The same coordinate system may be, for example, a coordinate system in which the positions of both images that are at the same position when the two images are combined have the same coordinate values.

[0054] The output unit 15 outputs the composite video and the guide. When a guide figure or a character string of the guide is included in the composite video, the output unit 15 may output the composite video including the guide figure or the character string of the guide. In this case, the guide is also output by outputting the composite video including the guide figure or the like. On the other hand, when the composite video does not include a guide, for example, when the guide is a sound, the output unit 15 may output the composite video and the guide, respectively.

[0055] Here, this output may be, for example, a display on a display device (e.g., a liquid crystal display or an organic EL display), a transmission via a communication line to a specified device, an audio output from a speaker, storage in a recording medium, or delivery to another component. As an example, if the guidance is audio, the output unit 15 may display a composite image and output the audio guidance from a speaker. The output unit 15 may or may not include a device that performs output (e.g., a display device, a communication device, a speaker, etc.). The output unit 15 may be realized by hardware, or may be realized by software such as a driver that drives those devices.

[0056] Next, the operation of the image synthesizer 1 will be described with reference to the flowchart of FIG. (Step S101) The synthesis unit 13 determines whether or not to synthesize the reference video and the self-video. If synthesis is to be performed, the process proceeds to step S102; if not, the process of step S101 is repeated until it is determined that synthesis is to be performed. Note that the synthesis unit 13 may determine to perform synthesis at each display interval of frames included in each video. For example, when the frame rate is A (fps), the synthesis unit 13 may determine to perform synthesis every 1 / A (seconds).

[0057] (Step S102) The synthesis unit 13 reads a frame of a reference video to be synthesized from the storage unit 11. Note that the frame of the reference video to be read T seconds after the start of the synthesis process may be, for example, a frame located T seconds from the beginning.

[0058] (Step S103) The image acquisition unit 12 acquires a frame of a self-image. The image acquisition unit 12 may acquire the latest frame of a self-image from the camera 901, for example.

[0059] (Step S104) The guide generating unit 14 generates a guide using the frame of the reference video read out in step S102 and the frame of the self-video acquired in step S103.

[0060] (Step S105) The synthesis unit 13 generates a synthetic video frame by synthesizing the reference video frame read in step S102 and the self-video frame acquired in step S103. Note that, if the generated guide is a guide figure, the synthesis unit 13 may generate a synthetic video frame including the guide figure.

[0061] (Step S106) The output unit 15 outputs the composite video generated in step S105 and the guide generated in step S104. For example, when a composite video including a guide figure is generated, the output unit 15 may output the composite video. Then, the process returns to step S101. In this manner, the frames of the reference video and the self-video are composited and output together with the generated guide, whereby the two videos are composited and a guide for causing the action target to follow the imitation target is also provided.

[0062] Note that the order of the processes in the flowchart of Fig. 9 is an example, and the order of the steps may be changed as long as the same results can be obtained. In addition, in the flowchart of Fig. 9, the process ends when the power is turned off or an interrupt is issued to end the process.

[0063] Next, the operation of the image synthesizing device 1 according to this embodiment will be described using a specific example. In this specific example, the reference image is an image of a situation in which a forceps as an imitation target 41 is being used by an expert. The reference image is, for example, the image shown in FIG. 3. Also, as shown in FIG. 2, it is assumed that the user 30 uses the camera 901 to capture a self-image of the action target 51 as forceps being used by the user 30. The self-image is, for example, the image shown in FIG. 4.

[0064] First, when the user 30 operates the image synthesizer 1 to start the process of synthesizing the reference image and the user's own image, the synthesizer 13 reads out the first frame of the reference image stored in the memory 11 (steps S101 and S102). The image acquisition unit 12 also receives a frame of the user's own image captured by the camera 901 via wired or wireless communication (step S103). The guide generator 14 also generates a guide at that time (step S104). In this specific example, it is assumed that a guide figure for moving the action target forward of the user 30 is generated using the reference image in FIG. 3 and the user's own image in FIG. 4, and is passed to the synthesizer 13.

[0065] When the guide figure is received, the synthesis unit 13 synthesizes a self-image with an opacity of 50% in front of the reference image, generates a frame of a synthetic image with the guide figure added, and passes it to the output unit 15 (step S105). When the synthetic image frame including the guide figure is received, the output unit 15 outputs the received frame of the synthetic image to the display device 902 (step S106). In this way, the user 30 can see the synthetic image shown in FIG. 5. In FIG. 5, a guide figure 61 is displayed, which indicates that the action target 51 should be moved forward of the user 30. Therefore, the user 30 only needs to move the action target 51 forward of the user 30 in accordance with the guide figure 61.

[0066] By repeating the display of the composite video including the guide figure, the sizes of the imitation target 41 and the action target 51 in the video become approximately the same, and it is assumed that it is no longer necessary to provide a guide for the forward and backward directions of the user 30. Then, when new frames of the reference video and the self-video are acquired (steps S101 to S103), the guide generating unit 14 generates a guide figure for moving the action target 51 in the up-down and left-right directions of the user 30 (step S104). Then, a frame of the composite video including the guide figure is generated and output (steps S105 and S106). The frame of the composite video is assumed to be that shown in FIG. 6. In FIG. 6, a guide figure 62 is displayed indicating that the action target 51 should be moved to the left of the user 30. Therefore, the user 30 only needs to move the action target 51 to the left of the user 30 according to the guide figure 62. In this way, by outputting the composite video including the guide figure, the user 30 can move the action target 51 according to the guide, thereby making it possible for the action target 51 to follow the imitation target 41.

[0067] As described above, according to the image synthesizing device 1 of the present embodiment, even if the reference image and the self-image are two-dimensional images that do not have depth information such as stereo images, a guide for making the action target follow the imitation target can be output, and assistance can be provided to make the action target operated by the user 30 follow the imitation target. By referring to the guide, the user 30 can more easily make the action target follow the imitation target. Therefore, even if the user 30 is a beginner at learning using a reference image, the effect of learning can be improved.

[0068] In addition, by generating a guide indicating the following direction in the forward and backward directions of the user 30 according to the size of the object to be imitated or the object to be moved, it becomes possible to easily align the object to be imitated and the object to be moved in the forward and backward directions of the user 30. In addition, by generating a guide indicating the following direction in the up-down and left-right directions of the user 30 using the representative position of the object to be imitated or the object to be moved, it becomes possible to easily align the object to be imitated and the object to be moved in the up-down and left-right directions of the user 30. In addition, by generating a guide for making the shape of the object to be imitated follow the shape of the object to be imitated according to the shape of the object to be imitated or the object to be moved, it becomes possible to easily match the shape of the object to be imitated to the shape of the object to be imitated. Furthermore, by using the relative positional relationship between the object to be imitated and the object to be moved acquired using the relative positional relationship between the object to be imitated or the object to be moved and the camera, it becomes possible to generate a more accurate guide.

[0069] Furthermore, when the guide is a guide figure, the user 30 can easily understand how to move the object of operation by referring to the guide figure contained in the output composite image.

[0070] In the above embodiment, the image synthesizer 1 is mainly described as a stand-alone device, but the image synthesizer 1 may be a stand-alone device or a server device in a server-client system. In the latter case, the acquisition unit and the output unit may acquire information and output information via a communication line.

[0071] In addition, in the above embodiments, each process or function may be realized by centralized processing in a single device or a single system, or may be realized by distributed processing in multiple devices or multiple systems.

[0072] In addition, in the above embodiments, the transfer of information between each component may be performed, for example, by one component outputting information and the other component receiving information if the two components transferring the information are physically different, or, if the two components transferring the information are physically the same, by shifting from a processing phase corresponding to one component to a processing phase corresponding to the other component.

[0073] In the above embodiment, information related to the processing performed by each component, such as information accepted, acquired, selected, generated, transmitted, or received by each component, and information such as thresholds, formulas, and addresses used by each component in processing, may be temporarily or long-term stored in a recording medium (not shown) even if not specified in the above description. Furthermore, each component or a storage unit (not shown) may store information in the recording medium (not shown). Furthermore, each component or a reading unit (not shown) may read information from the recording medium (not shown).

[0074] In the above embodiment, if the information used by each component, such as the thresholds, addresses, and various setting values ​​used by each component in processing, may be changed by the user, the user may be able to change the information as appropriate, even if not specified in the above description. If the information is changeable by the user, the change may be realized, for example, by a reception unit (not shown) that receives a change instruction from the user, and a change unit (not shown) that changes the information in response to the change instruction. The reception unit (not shown) may receive the change instruction from an input device, may receive information transmitted via a communication line, or may receive information read from a predetermined recording medium.

[0075] In addition, in the above embodiment, when two or more components included in the image synthesis device 1 have a communication device, an input device, etc., the two or more components may physically have a single device, or may have separate devices.

[0076] In the above embodiment, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component may be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. During execution, the program execution unit may execute the program while accessing a storage unit or a recording medium. The software that realizes the image synthesis device 1 in the above embodiment is a program such as the following. In other words, this program may be a program that causes a computer that can access a storage unit in which a reference image that is an image of an imitation target whose behavior is to be imitated by a user to function as an image acquisition unit that acquires a self-image that is an image of an action target whose behavior is to be operated by a user, a synthesis unit that generates a synthetic image by synthesizing the reference image and the self-image, a guide generation unit that generates a guide to make the action target follow the imitation target based on the imitation target included in the reference image and the action target included in the self-image, and an output unit that outputs the synthetic image and the guide.

[0077] In addition, the functions realized by the above program do not include functions that can only be realized by hardware, such as a modem or an interface card in an acquisition unit that acquires information or an output unit that outputs information.

[0078] This program may be executed by being downloaded from a server or the like, or may be executed by reading out a program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, etc.). This program may be used as a program constituting a program product.

[0079] Furthermore, the program may be executed by a single computer or a plurality of computers. That is, the program may be executed by centralized processing or distributed processing.

[0080] FIG. 10 is a diagram showing an example of the configuration of a computer 900 that executes the above program to realize the image synthesizing device 1 according to the above embodiment. In FIG. 10, the computer 900 includes an MPU (Micro Processing Unit) 911, a ROM 912 for storing programs such as a boot-up program, a RAM 913 connected to the MPU 911 to temporarily store instructions of application programs and provide a temporary storage space, a storage unit 914 for storing application programs, system programs, and data, and a communication module 915 for providing a connection to a LAN, a WAN, or the like. The MPU 911, the ROM 912, and the like may be connected to each other by a bus together with a camera 901, a display device 902, a keyboard 903, and a pointing device 904 such as a touchpad or a mouse. The storage unit 914 may be, for example, a hard disk or a solid state drive (SSD). The camera 901, the display device 902, the keyboard 903, the pointing device 904, and the like may be, for example, external devices, or may be devices built into the computer 900.

[0081] A program for causing the computer 900 to execute the functions of the image synthesizing device 1 according to the above-described embodiment may be loaded into the RAM 913 at the time of execution. Note that the program may be loaded directly from the storage unit 914 or a network, for example.

[0082] The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 900 to execute the functions of the image synthesizing device 1 according to the above embodiment. The program may include only an instruction portion that calls appropriate functions or modules in a controlled manner to obtain a desired result. How the computer 900 operates is well known, and a detailed description will be omitted.

[0083] The above-mentioned embodiments are merely examples for specifically implementing the present invention, and are not intended to limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims, not by the description of the embodiments, and is intended to include modifications within the literal scope of the claims and the scope of equivalent meanings. [Explanation of symbols]

[0084] 1. Video composition equipment 11 Storage section 12 Image acquisition section 13 Synthesis section 14 Guide Generation Unit 15 Output section 30 users 41 Imitation Target 51 Target 61, 62 Guide shapes

Claims

1. A storage unit that stores a reference video, which is a video of a target motion that a user wishes to imitate; an image acquisition unit that acquires a self-image, which is an image of a motion of an action target that is to be performed by the user; a synthesis unit that generates a synthetic image by synthesizing the reference image and the self-image; a guide generating unit that generates a guide for causing the motion target to follow the imitation target based on the imitation target included in the reference video and the motion target included in the self-video; an output unit that outputs the composite image and the guide.

2. The image synthesis device according to claim 1 , wherein the guide generation unit generates a guide indicating a following direction in a forward / backward direction based on the user in accordance with a relationship between a size of the object to be imitated in the reference image and a size of the object to be moved in the self-image.

3. The image synthesis device of claim 1, wherein the guide generation unit generates a guide indicating a following direction in the up / down and left / right directions based on the user in accordance with a positional relationship between a representative position of the imitation target in the reference image and a representative position of the motion target in the self-image.

4. The image synthesis device of claim 1, wherein the guide generation unit uses a plurality of feature points of the object to be imitated in the reference image to obtain a first relative positional relationship between the object to be imitated and a camera that captured the reference image, uses a plurality of feature points of the object to be moved in the self-image to obtain a second relative positional relationship between the object to be moved and a camera that captured the self-image, uses the first and second relative positional relationships to obtain a third relative positional relationship between the object to be imitated and the object to be moved, and generates a guide using the third relative positional relationship.

5. The imitation target and the action target change shape, The image synthesis device of claim 1 , wherein the guide generation unit generates a guide for causing the shape of the subject to follow the shape of the subject to be imitated using the shape of the subject to be imitated contained in the reference image and the shape of the subject to be moved contained in the self-image.

6. The guide generating unit generates a guide figure that is the guide, The synthesis unit generates a synthetic image including the guide figure, The image synthesizing device according to claim 1 , wherein the output unit outputs the synthetic image including the guide figure.

7. A video synthesis method that is processed using a storage unit that stores a reference video, which is a video of a target action that a user is to imitate, a video acquisition unit, a synthesis unit, a guide generation unit, and an output unit, The image acquisition unit acquires a self-image, which is an image of an action of an action target that is to be performed by the user; a step of the synthesis unit generating a synthetic image by synthesizing the reference image and the self-image; A step of generating a guide for causing the motion target to follow the imitation target, based on the imitation target included in the reference video and the motion target included in the self-video, by the guide generation unit; The output unit outputs the composite image and the guide.

8. A computer that can access a storage unit that stores a reference video, which is a video of a target motion that a user is to imitate, an image acquisition unit that acquires a self-image, which is an image of an action of an action target that is an object to be performed by the user; a synthesis unit that generates a synthetic image by synthesizing the reference image and the self-image; a guide generating unit that generates a guide for causing the motion target to follow the imitation target based on the imitation target included in the reference video and the motion target included in the self-video; A program for causing the output unit to function as an output unit that outputs the composite image and the guide.

Images