Video synthesizing device, video synthesizing method, and program

Abstract

To provide a video synthesizing device with which it is possible to prevent a user from feeling that a reference video and a self- video are merged, and possible for a user as a learner to easily follow the motion of an object to be imitated.SOLUTION: A video synthesizing device 1 comprises: a storage unit 11 in which a reference video is stored that is a video of motion of an imitation object whose motion a user is going to imitate; a video acquisition unit 12 that acquires a self-video which is a video of motion of an object to be operated on by the user; a synthesis unit 13 that synthesizes the reference video and the self-video to generate a synthesized video; and a video output unit 14 that outputs the synthesized video. The synthesis unit 13 synthesizes both videos so that the synthesis ratio of the reference video and the self-video repeatedly changes in succession with the passage of time.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The present invention relates to an image synthesizing device that synthesizes and outputs a reference image, which is an image of a motion to be imitated, and a self-image, which is an image of a user's motion to be performed. [Background technology]

[0002] Conventionally, there is known a learning support device that alternately displays a reference image, which is an image of a target movement that the learner is to imitate, and a self-image, which is an image of the learner's movement, for learning movements such as surgery (see, for example, Patent Document 1). By referring to such a display, the learner can train to perform the same movements as the target movement. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2014-071443 A Summary of the Invention [Problem to be solved by the invention]

[0004] However, when the reference video and the self-video are alternately displayed, the learner feels as if the two videos are fused together. Therefore, if the learner can properly follow the movements of the object of imitation contained in the reference video, there is no problem. However, if the learner cannot properly follow the movements of the object of imitation, the difference between the reference video and the self-video becomes unclear, and it becomes difficult for the learner to follow the movements of the object of imitation. For example, it may be difficult for the learner to easily recognize which is the self-video.

[0005] The present invention has been made to solve the above-mentioned problems, and aims to provide an image synthesis device or the like that can prevent the impression that the reference image and the user's own image have been merged, and enables the learner to easily follow the movements of the object to be imitated. [Means for solving the problem]

[0006] In order to achieve the above-mentioned object, an image synthesis device according to one aspect of the present invention comprises a memory unit in which a reference image, which is an image of an object to be imitated by a user, is stored, an image acquisition unit in which a self-image, which is an image of an object to be imitated by the user, a synthesis unit in which the reference image and the self-image are synthesized to generate a synthesized image, and an image output unit in which the synthetic image is output, and the synthesis unit synthesizes the reference image and the self-image so that the synthesis ratio between the reference image and the self-image changes continuously and repeatedly over time.

[0007] With this configuration, the reference image and the self-image can be composited so that the composite ratio of the two images changes continuously and repeatedly over time. In this way, since the reference image and the self-image are not alternately displayed, it is possible to prevent the user from feeling that the reference image and the self-image have been merged, and the user can easily grasp the self-image, and as a result, the user can easily follow the movements of the person being imitated.

[0008] In the image synthesizing device according to an aspect of the present invention, the synthesizing section may synthesize the reference image and the self-image so that a change over time in a synthesis ratio between the two images forms a sine wave.

[0009] With this configuration, for example, when the composition is performed so that the proportion of one image changes between 0 and 100%, there will be periods when only one image is primarily displayed, and there will also be periods when both images are displayed overlapping, allowing the user to easily distinguish between the two images and easily match the object of the action that the user is controlling to the action of the object to be imitated.

[0010] In addition, an image synthesis device according to one aspect of the present invention may further include a distance acquisition unit that acquires the distance between the imitation target and the action target in the synthesized image, and the synthesis unit may perform synthesis such that the greater the distance, the more of the self-image is displayed.

[0011] With this configuration, when the distance between the imitation target and the motion target is large, the self-image is displayed more, so that the motion target can be moved to reduce the distance between them. When the distance between them is small, the imitation target is displayed more, so that the motion of the imitation target can be confirmed in more detail.

[0012] In addition, in an image synthesis device according to one aspect of the present invention, a background modification unit is further provided that modifies a background area of ​​an image to be changed, which is at least one of a reference image and a self-image, to transparent or a predetermined solid color, and the synthesis unit may synthesize the image to be changed with the image after modification by the background modification unit.

[0013] With this configuration, it is possible to make the imitation target and the action target more visible in the composite image. In a composite image in which a reference image and a self-image, each of which includes a background, are composited, for example, when the imitation target overlaps with the background area of ​​the self-image or the action target overlaps with the background area of ​​the reference image, it becomes difficult to grasp the position and state of the imitation target and the action target, but by changing the background area of ​​at least one of the images to transparent or single color, such a situation can be improved.

[0014] In addition, an image synthesis device according to one aspect of the present invention further includes an identification unit that identifies a point of interest in an image of a specific target, which is at least one of a reference image and a self-image, and a blurring processing unit that performs a process of blurring areas other than the identified point of interest in the image of the specific target, and the synthesis unit may synthesize an image after blurring processing by the blurring processing unit for the image of the specific target.

[0015] With this configuration, blurring is performed on areas other than the area of ​​interest, making it easier for the area of ​​interest to be noticed.

[0016] In the image synthesizing device according to an aspect of the present invention, the specifying unit may specify a target area that is a moving area.

[0017] With this configuration, it becomes possible to automatically identify the location of interest.

[0018] In addition, the image synthesis device according to one aspect of the present invention may further include a reception unit that receives attention point information indicating a attention point from a user, and the identification unit may identify the attention point according to the attention point information received by the reception unit.

[0019] With this configuration, the user can specify a location of interest.

[0020] In addition, a video synthesis method according to one aspect of the present invention is a video synthesis method processed using a memory unit in which a reference video, which is a video of an object to be imitated by a user, is stored, a video acquisition unit, a synthesis unit, and a video output unit, and includes a step in which the video acquisition unit acquires a self-image, which is a video of an object to be imitated by the user, a step in which the synthesis unit synthesizes the reference video and the self-image to generate a synthetic video, and a step in which the video output unit outputs the synthetic video, and in the step of generating the synthetic video, the reference video and the self-image are synthesized so that the ratio of synthesis between the two videos changes continuously and repeatedly over time. Effect of the Invention

[0021] According to an aspect of the image synthesis device of the present invention, it is possible to prevent the user from feeling that the reference image and the user's own image have been merged, and it becomes possible for the user, who is a learner, to easily follow the movements of the person being imitated. [Brief description of the drawings]

[0022] [Figure 1] FIG. 1 is a block diagram showing a configuration of a video synthesizing device according to an embodiment of the present invention; [Diagram 2] FIG. 2 shows an example of a usage state of the image synthesizer according to the embodiment. [Diagram 3] FIG. 2 shows an example of a reference video in the embodiment. [Figure 4] FIG. 13 shows an example of a self-image in the embodiment. [Figure 5A] FIG. 13 is a diagram showing an example of a change in the synthesis ratio in the embodiment; [Figure 5B] FIG. 13 is a diagram showing an example of a change in the synthesis ratio in the embodiment; [Figure 5C] FIG. 13 is a diagram showing an example of a change in the synthesis ratio in the embodiment; [Figure 5D] FIG. 13 is a diagram showing an example of a change in the synthesis ratio in the embodiment; [Figure 6] FIG. 13 shows an example of a composite image according to the embodiment. [Figure 7] FIG. 13 shows an example of a composite image according to the embodiment. [Figure 8] A flowchart showing the operation of the image synthesizing device according to the embodiment. [Figure 9] FIG. 2 is a block diagram showing another example of the configuration of the image synthesizing device according to the embodiment. [Figure 10] FIG. 2 is a block diagram showing another example of the configuration of the image synthesizing device according to the embodiment. [Figure 11] FIG. 2 is a block diagram showing another example of the configuration of the image synthesizing device according to the embodiment. [Figure 12] FIG. 2 shows a plurality of regions on an image in the embodiment. [Figure 13] FIG. 2 shows an example of a configuration of a computer according to the embodiment. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0023] The video synthesis device and the video synthesis method according to the present invention will be described below using embodiments. In the following embodiments, components and steps with the same reference numerals are the same or equivalent, and repeated explanations may be omitted. The video synthesis device according to this embodiment synthesizes a reference video and a self-video so that the ratio of the two videos is changed continuously and repeatedly over time.

[0024] FIG. 1 is a block diagram showing a configuration of an image synthesizing device 1 according to this embodiment. FIG. 2 is a diagram showing a situation in which a user 30, who is a learner, is learning the movement of a target to be imitated using the image synthesizing device 1. The image synthesizing device 1 according to this embodiment includes a storage unit 11, an image acquisition unit 12, a synthesis unit 13, and an image output unit 14. Note that the image synthesizing device 1 may be realized by a computer 900 as shown in FIG. 13, etc., which will be described later, as an example, or may be realized by dedicated hardware. In this embodiment, the former case will be mainly described.

[0025] The storage unit 11 stores a reference image, which is an image of the motion of a target object whose motion the user is to imitate. The user is a learner who learns the motion while referring to the reference image. The motion that the user learns may be, for example, a motion such as surgery, a motion of work in a factory, a motion of nursing care or a hotel, a motion such as cooking, a motion for creating a work such as a craft, a motion such as sports, a motion such as calligraphy, a motion of tying a rope, or other motions. The target object to be imitated may be, for example, a part of the body of the imitated person, or an object operated by the imitated person. The imitated person may be, for example, a teacher of the user who is a learner, and may be skilled in the motion that the learner is to learn. In addition, a part of the body of the imitated person may include, for example, the hand of the imitated person. In addition, the object operated by the imitated person may be, for example, a tool such as forceps, scalpel, tweezers, scissors, or brush that the imitated person has. The reference image is usually an image captured by a camera, but may be a CG (Computer Graphics) image equivalent to the image captured by the camera.

[0026] The reference image may be, for example, an image from the viewpoint of the person to be imitated who operates the object to be imitated, that is, an image from the first-person viewpoint of the person to be imitated. In this case, the reference image may be, for example, an image captured by a head-mounted camera worn by the person to be imitated. In this embodiment, a case will be mainly described in which the reference image shown in FIG. 3, including the object to be imitated 41, which is a pair of forceps used by the person to be imitated, is stored in storage unit 11.

[0027] In the storage unit 11, for example, the entire reference video may be stored, or only a part of the reference video may be stored. As an example, when the video synthesis device 1 displays a reference video while receiving it from the outside, a part of the reference video, that is, the most recently received reference video, may be stored in the storage unit 11, and may be read out and displayed, and may be overwritten sequentially. The storage unit 11 may store information other than the reference video. For example, a self-video image acquired by the video acquisition unit 12 may be stored in the storage unit 11.

[0028] The process by which information is stored in the storage unit 11 is not important. For example, information may be stored in the storage unit 11 via a recording medium, information transmitted via a communication line or the like may be stored in the storage unit 11, or information inputted via a device such as a camera may be stored in the storage unit 11. The storage unit 11 is preferably realized by a non-volatile recording medium, but may also be realized by a volatile recording medium. The recording medium may be, for example, a semiconductor memory or a magnetic disk.

[0029] The image acquisition unit 12 acquires a self-image, which is an image of the operation of an operation target that is an object to be operated by the user. The image acquisition unit 12 may be, for example, an optical device such as a camera that captures an image, or may acquire an image captured by an optical device such as a camera. In this embodiment, a case where the image acquisition unit 12 receives an image captured by a camera 901 will be mainly described. The camera 901 that captures the self-image may be, for example, a camera that is attached to the user 30, such as a head-mounted camera, or may be a camera that is installed in the shooting environment by a support unit such as a tripod. The operation target corresponds to the imitation target. Usually, the operation target and the imitation target are of the same type. Therefore, the operation target may be, for example, a part of the user's body, or may be an object operated by the user. The part of the user's body may include, for example, the user's hand. In addition, the object operated by the user may be, for example, a tool such as forceps, scalpel, tweezers, scissors, or brush that the user has. For example, when the imitation target is forceps, it is preferable that the operation target is also forceps.

[0030] The self-image may be, for example, an image from the viewpoint of the user who operates the operation target, that is, an image from the user's first-person viewpoint. In this case, the self-image may be, for example, an image captured by a camera 901 worn on the head of the user 30 as shown in FIG. 2. It is preferable that the optical axis of the camera 901 faces the operation target. In this embodiment, a case where a self-image shown in FIG. 4 including an operation target 51, which is a forceps used by the user 30, is captured by the image capture unit 12 will be mainly described. It is preferable that the relative positional relationship between the reference image camera that captures the reference image and the imitation target and the relative positional relationship between the self-image camera that captures the self-image and the operation target are the same or close to each other. It is also preferable that each frame included in the reference image and each frame included in the self-image are, for example, the same size. The frames being the same size may be that the number of pixels in the vertical direction and the number of pixels in the horizontal direction of the frame are the same.

[0031] The synthesis unit 13 synthesizes the reference video and the self-video to generate a synthetic video. During this synthesis, the synthesis unit 13 synthesizes both videos so that the synthesis ratio of the reference video and the self-video changes continuously and repeatedly along time. The synthesis ratio of the reference video and the self-video may be the display ratio of the reference video or the self-video displayed in the synthetic video. This ratio may be, for example, the opacity or transparency of one of the videos. In this embodiment, a case where the synthesis ratio of the reference video and the self-video is the opacity ratio of the self-video may be mainly described. In this case, the self-video may be synthesized in front of the reference video, that is, on the upper side. The synthesis ratio of both videos changing continuously along the time direction may mean, for example, that the synthesis ratio changes continuously over the entire time direction of the synthesis ratio (see, for example, FIG. 5A and FIG. 5B), or that the synthesis ratio changes continuously over at least a part of the time direction of the synthesis ratio (see, for example, FIG. 5C). Furthermore, the repetition of the synthesis ratio in the time direction may be, for example, a periodic repetition, or a non-periodic repetition. The period of the change in the periodically changing composition ratio is not particularly limited, but may be, for example, about 0.5 seconds to 3 seconds. The faster the movement of the imitation target, the shorter the period is preferable. In addition, the composition ratio of the reference image and the self-image may change between 100:0 and 0:100, for example, or may change so that there is no period in which the composite image becomes one of the images, such as between 95:5 and 5:95. In this embodiment, the former case will be mainly described. In addition, the continuous change in the composition ratio in the time direction may be a smooth change, or may not be. In the former case, the time change in the composition ratio may be time-differentiable.

[0032] As an example, the synthesis unit 13 may synthesize both the reference video and the self-video so that the change over time of the synthesis ratio of the two videos is a sine wave. In this case, the opacity of the self-video to be synthesized in front of the reference video (i.e., the synthesis ratio) may change along a time series as shown in FIG. 5A. In this case, there are periods in which the reference video is the main one in the synthesized video (periods in which the opacity of the self-video is around 0%) and periods in which the self-video is the main one in the synthesized video (periods in which the opacity of the self-video is around 100%), so that the user can see only the reference video and can see only the self-video. Therefore, the user can easily distinguish and grasp the imitation target included in the reference video and the action target included in the self-video. On the other hand, in the other periods, the synthesized video includes both the reference video and the self-video, so that the user can see the imitation target included in the reference video and the action target included in the self-video together. Therefore, the user can easily move the action target included in the self-video in accordance with the action of the imitation target included in the reference video. That is, the user can cause the motion target to move in accordance with the motion of the imitation target.

[0033] The synthesis unit 13 may, for example, specify a synthesis ratio between the reference video and the self-video when synthesizing, and synthesize the reference video and the self-video at the measured ratio. For example, when the change over time of the synthesis ratio between the reference video and the self-video is shown in FIG. 5A, the synthesis unit 13 may specify the opacity of the self-video corresponding to time t when synthesizing both videos at time t, and synthesize both videos so that the opacity of the self-video to be synthesized in front of the reference video corresponding to time t becomes the specified value. The change over time of the synthesis ratio between the reference video and the self-video may be shown, for example, by a graph, a table, a function, or the like, and the synthesis unit 13 may specify the synthesis ratio between the reference video and the self-video when synthesizing using them. When the synthesis ratio is shown in FIG. 5A, the function indicating the opacity (%) of the self-video at time t may be shown, for example, by the following formula. In the following formula, P is the period of opacity. Self-image opacity (%) = 50 × (sin(2πt / P-π / 2) + 1)

[0034] 6 and 7 are diagrams showing examples of a composite image. FIG. 6 is a composite image generated with the opacity at time T1 in FIG. 5A, and FIG. 7 is a composite image generated with the opacity at time T2 in FIG. 5A. In the composite image shown in FIG. 6, the opacity of the self-image is lower than 50%, so that the imitation target 41 is displayed more clearly than the action target 51. On the other hand, in the composite image shown in FIG. 7, the opacity of the self-image is higher than 50%, so that the action target 51 is displayed more clearly than the imitation target 41.

[0035] Also, the synthesis unit 13 may synthesize both images so that the change over time of the synthesis ratio of the reference image and the self-image is other than a sine wave. For example, the synthesis unit 13 may synthesize both images so that the change over time of the synthesis ratio of the reference image and the self-image is a triangular wave as shown in Fig. 5B, or may synthesize both images so that the change over time of the synthesis ratio is a sawtooth wave as shown in Fig. 5C. Also, the synthesis unit 13 may synthesize the two images by adjusting the size of one image to the size of the other image when the size of each frame included in the reference image and the size of each frame included in the self-image are different, for example.

[0036] The video output unit 14 outputs the composite video generated by the composition unit 13. Here, this output may be, for example, a display on a display device (for example, a liquid crystal display or an organic EL display) or may be a transmission via a communication line to an apparatus that displays the composite video. Note that the video output unit 14 may or may not include a device that performs output (for example, a display device or a communication device). The display device on which the video output by the video output unit 14 is displayed may be, for example, a display placed in the environment where the user 30 is present, as shown in FIG. 2, or may be a head-mounted display worn by the user 30. Also, the video output unit 14 may be realized by hardware, or may be realized by software such as a driver that drives those devices.

[0037] Next, the operation of the image synthesizer 1 will be described with reference to the flowchart of FIG. (Step S101) The synthesis unit 13 determines whether or not to synthesize the reference video and the self-video. If synthesis is to be performed, the process proceeds to step S102; if not, the synthesis unit 13 repeats the process of step S101 to determine that synthesis is to be performed. Note that the synthesis unit 13 may determine to perform synthesis at each display interval of frames included in each video. For example, when the frame rate is A (fps), the synthesis unit 13 may determine to perform synthesis every 1 / A (seconds).

[0038] (Step S102) The synthesis unit 13 reads a frame of the reference video to be synthesized from the storage unit 11. Note that the frame of the reference video to be read T seconds after the synthesis process starts may be, for example, a frame located T seconds from the beginning.

[0039] (Step S103) The image acquisition unit 12 acquires a frame of a self-image. The image acquisition unit 12 may acquire the latest frame of a self-image from the camera 901, for example.

[0040] (Step S104) The synthesis unit 13 specifies the synthesis ratio between the reference video and the synthetic video. For example, in the case where the synthesis ratio changes in the time direction as shown in Fig. 5A and T seconds have passed since the synthesis process started, the synthesis unit 13 may specify the synthesis ratio corresponding to T seconds.

[0041] (Step S105) The synthesis unit 13 generates a frame of a synthetic video by synthesizing the frame of the reference video read out in step S102 and the frame of the self-video acquired in step S103 in the ratio specified in step S104.

[0042] (Step S106) The video output unit 14 outputs the frame of the composite video generated in step S105. Then, the process returns to step S101. In this manner, the frames of the reference video and the self video are composited and output, thereby composite of both videos.

[0043] Note that the order of the processes in the flowchart of Fig. 8 is an example, and the order of the steps may be changed as long as the same results can be obtained. For example, the processes in steps S102 to S104 may be executed in an order other than that shown in Fig. 8. In addition, in the flowchart of Fig. 8, the process ends due to power off or an interrupt to end the process.

[0044] Next, the operation of the image synthesizing device 1 according to this embodiment will be described using a specific example. In this specific example, the reference image is an image of a situation in which a forceps as an imitation target 41 is being used by an expert. The reference image is, for example, the image shown in FIG. 3. Also, as shown in FIG. 2, it is assumed that a self-image of the action target 51 as forceps being used by the user 30 is captured using a camera 901 worn on the user's head. The self-image is, for example, the image shown in FIG. 4.

[0045] First, when the user 30 operates the image synthesizer 1 to start the process of synthesizing the reference image and the self-image, the synthesizer 13 reads out the first frame of the reference image stored in the storage unit 11 (steps S101 and S102). The image acquisition unit 12 also receives a frame of the self-image captured by the camera 901 via wired or wireless communication (step S103). The synthesizer 13 also specifies the synthesis ratio at that time (step S104). In this specific example, the self-image is synthesized in front of the reference image using the opacity of the self-image shown in FIG. 5A. It is assumed that the opacity of the self-image is 0% at the start of the synthesis process. Then, the synthesizer 13 generates a frame of a synthesized image in which the self-image with an opacity of 0% is synthesized in front of the reference image, and passes it to the image output unit 14 (step S105). That is, a frame of a synthesized image including only the reference image is generated. Upon receiving the frame of the composite video, the video output unit 14 outputs the received frame of the composite video to the display device 902 (step S106). In this manner, the user 30 becomes able to view the composite video.

[0046] The ratio of the self-image to the reference image is changed as shown in FIG. 5A. Therefore, the ratio of the self-image gradually increases in the composite image, and the composite image shown in FIG. 6 is displayed at time T1, and the composite image shown in FIG. 7 is displayed at time T2. After the composite image including only the self-image is displayed, the ratio of the self-image gradually decreases in the composite image. In this way, the ratio of the self-image in the composite image is changed periodically within the range from 0% to 100% according to the sine wave. While watching the composite image displayed on the display device 902, the user 30 can learn by moving the motion target so that the motion is similar to that of the motion target.

[0047] As described above, according to the image synthesizing device 1 of the present embodiment, the reference image and the self-image can be synthesized so that the ratio of the synthesis of the two images changes continuously and repeatedly over time. Therefore, since the reference image and the self-image are not alternately displayed, it is possible to prevent the user from feeling that the reference image and the self-image are fused together. As a result, the user who watches the synthesized image can easily distinguish between the imitation target 41 in the reference image and the motion target 51 in the self-image, and can move the motion target to follow the motion of the imitation target, thereby improving the effect of learning using the reference image. In addition, when the ratio of the synthesis of the two images changes periodically and repeatedly, it is possible to easily predict the time when the reference image is mainly displayed or the time when the self-image is mainly displayed, and as a result, it becomes easier to identify the imitation target and the motion target in the synthesized image.

[0048] In addition, as shown in FIG. 9, the image synthesizing device 1 according to the present embodiment further includes a distance acquisition unit 21 that acquires the distance between the imitation target and the action target in the synthetic image, and the synthesizing unit 13 may perform synthesis such that, for example, the greater the distance acquired by the distance acquisition unit 21, the more the self-image is displayed. The distance between the imitation target and the action target in the synthetic image is the distance between the imitation target and the action target in the synthetic image when a reference image read out at a certain time point is synthesized with a self-image acquired at that time point to generate a synthetic image. This distance may be, for example, the distance between the imitation target and the action target in the frame of the synthetic image synthesized immediately before, or may be the distance between the imitation target and the action target in the frame of the synthetic image synthesized by synthesizing the frame of the reference image at the time of synthesis and the frame of the self-image to acquire this distance, or may be the distance between the position of the imitation target in the frame of the reference image at the time of synthesis and the position of the action target in the latest frame of the self-image.

[0049] The area of ​​the object to be imitated or the object to be acted on in the frame of the video may be identified, for example, by object detection, segmentation, pattern matching, or the like in the frame. The position of the object to be imitated or the position of the object to be acted on may be, for example, a representative position of the area of ​​the object to be imitated or the object to be acted on identified in the frame. The representative position of an area may be, for example, the position of the center of gravity of the area, or the position of the center of a bounding box surrounding the area.

[0050] For example, when the acquired distance is greater than the threshold value TH, the synthesis unit 13 may synthesize the reference image and the self-image so that the self-image is displayed more than when the acquired distance is smaller than the threshold value TH. Note that, for example, when the acquired distance is equal to the threshold value TH, the synthesis unit 13 may synthesize the reference image and the self-image so that the self-image is displayed more than when the acquired distance is smaller than the threshold value TH, or may not. Synthesizing so that the self-image is displayed more may be, for example, synthesizing so that the average value of the ratio of the self-image in the synthetic image in the time direction becomes larger over a predetermined period. When the ratio of synthesis changes periodically, the predetermined period may be a period corresponding to an integer multiple of one period of the ratio of synthesis. In this case, when the acquired distance is greater than the threshold value TH, the self-image is displayed more than when the acquired distance is smaller than the threshold value TH when the synthetic image is displayed. As an example, synthesis may be performed so that the average value of the ratio of the self-image in the synthetic image exceeds 50% when the acquired distance is greater than the threshold value TH. When the distance between the imitation subject and the motion subject is large, the motion subject is usually moved so that its position approaches the position of the imitation subject, rather than adjusting the motion of the motion subject to that of the imitation subject. Therefore, by performing synthesis so that the motion subject is displayed more than when the distance between the imitation subject and the motion subject is small, the motion can be easily performed. On the other hand, when the distance between the imitation subject and the motion subject is small, the motion subject is usually moved so that its motion is adjusted to that of the imitation subject. Therefore, by performing synthesis so that the motion subject is displayed more than when the distance between the imitation subject and the motion subject is large, the motion can be easily imitated.

[0051] As an example, when the acquired distance is greater than the threshold TH, the synthesis unit 13 may synthesize the reference image and the self-image according to the opacity of the self-image that changes over time as shown in Fig. 5D. When the acquired distance is smaller than the threshold TH, the synthesis unit 13 may synthesize the reference image and the self-image according to the opacity of the self-image that changes over time as shown in Fig. 5A. In this way, when the acquired distance is greater than the threshold TH, the reference image and the self-image can be synthesized so that more of the self-image is displayed than when the acquired distance is smaller than the threshold TH.

[0052] Also, as shown in FIG. 10, for example, the image synthesizing device 1 according to the present embodiment further includes a background change unit 22 that changes the background area of ​​at least one of the reference image and the self-image to be changed to transparent or a predetermined single color, and the synthesizing unit 13 may synthesize the image after the change by the background change unit 22 for the image to be changed. The image to be changed may be, for example, only the reference image, only the self-image, or both images. When the image to be changed is only the reference image, the synthesizing unit 13 may synthesize the reference image after the background change and the self-image after the background change. When the image to be changed is only the self-image, the synthesizing unit 13 may synthesize the reference image after the background change and the self-image after the background change. When the image to be changed is the reference image and the self-image, the synthesizing unit 13 may synthesize the reference image after the background change and the self-image after the background change.

[0053] The background change unit 22 may, for example, specify the area of ​​the imitation target or the action target, and specify the area other than the specified area as the background area. The area of ​​the imitation target or the action target may be specified as described above. Note that the area of ​​an object (for example, an organ or a simulated organ) that is the subject of a predetermined process (for example, a suturing process performed by the imitation target or the action target, which is forceps) by the imitation target or the action target may also be considered as other than the background. The background change unit 22 may make the specified background area, for example, transparent, or may change it to a predetermined monochromatic color. Note that when both the reference image and the self-image are images to be changed, and the background areas of both images are changed to a monochromatic color, the background color of the reference image and the background color of the self-image may be different. In this way, when the composite image is displayed, it becomes easy to know which image is mainly displayed by using the color of the background. Also, software that makes the background area of ​​an image transparent or a specific monochromatic color is known. The background change unit 22 may use such software to change the background area to transparent or monochromatic in each frame of the image to be changed.

[0054] In this way, by changing the background area of ​​at least one of the images to transparent or monochrome, the imitation target or the action target can be made more visible in the composite image. For example, a transparent or monochrome background area of ​​the self-image is displayed in the area of ​​the imitation target, or a transparent or monochrome background area of ​​the reference image is displayed in the area of ​​the action target, so that the user can easily identify the imitation target or the action target.

[0055] 11, the image synthesizing device 1 according to the present embodiment further includes a receiving unit 23 that receives attention point information indicating an attention point from a user, a specifying unit 24 that specifies an attention point for an image of a specific target which is at least one of a reference image and a self-image, and a blurring processing unit 25 that performs processing to blur an area other than the attention point specified for the image of the specific target, and the synthesizing unit 13 may synthesize an image after blurring processing by the blurring processing unit 25 for the image of the specific target. The image of the specific target may be, for example, only the reference image, only the self-image, or both images.

[0056] The reception unit 23 may receive, for example, attention point information that is at least one of a plurality of area identifiers that respectively identify predetermined areas 61 to 65 in a frame of a video as shown in FIG. 12. In this case, the area identified by the area identifier that is the attention point information may be the attention point. The area identifier may be, for example, "center," "upper left," "upper right," "lower left," or "lower right" that respectively identify the areas 61 to 65. In addition, the reception unit 23 may receive, as an example, attention point information that is a result of voice recognition. In this case, the user is able to specify the attention point by voice. It goes without saying that the number and positions of attention points that can be identified in a video may be other than those shown in FIG. 12.

[0057] Reception unit 23 may, for example, receive information input from an input device (e.g., a keyboard, a mouse, a touch panel, etc.), or may receive information transmitted via a wired or wireless communication line. Note that reception unit 23 may or may not include a device for receiving the information (e.g., an input device, a communication device, etc.). Reception unit 23 may be realized by hardware, or may be realized by software such as a driver that drives a predetermined device.

[0058] The identification unit 24 may identify a point of interest according to, for example, point of interest information received by the reception unit 23. In this case, the identification unit 24 may identify a point of interest that is a region identified by a region identifier that is the point of interest information received by the reception unit 23. Specifically, when point of interest information with a region identifier "center" is received, the identification unit 24 may identify the region 61 identified by the region identifier "center" as the point of interest. When point of interest information is received and the images to be identified are both a reference image and a self-image, for example, the points of interest in both images may be the same or different. In the latter case, for example, point of interest information may be received for each image.

[0059] The identification unit 24 may also identify a point of interest without using the received point of interest information. In this case, the identification unit 24 may, for example, identify a point of interest that is a part of movement in the image of the identification target. The point of interest that is a part of movement in the image may be, for example, a part where a change in position per unit time of corresponding feature points in the two frames, that is, a distance, exceeds a threshold value, by extracting feature points in each of two frames that are separated by a predetermined time interval in the image. As an example, a part where the magnitude of an optical flow vector in the image exceeds a predetermined threshold may be set as a point of interest that is a part of movement in the image. This threshold value may be set so that, for example, an imitation target or a moving target is included in the point of interest. Note that the shape of the point of interest may be predetermined, such as, for example, a rectangular shape or an elliptical shape. In addition, for example, the size of the point of interest may also be predetermined. In this case, the identification unit 24 may, for example, identify a point of interest that is an area of ​​a predetermined size and shape that includes a part of movement in the image of the identification target. When a point of interest is identified without using point of interest information and the images to be identified are both a reference image and a self-image, for example, the points of interest in both images may be different. When a point of interest is identified without using received point of interest information, the image synthesizing device 1 does not need to have the receiving unit 23.

[0060] The blurring processing unit 25 performs a process of blurring the area of ​​the image of the identified target other than the attention point identified by the identification unit 24. As an example, the identification unit 24 may pass the image of the identified target and the area indicating the attention point in the image to the blurring processing unit 25, and the blurring processing unit 25 may perform a process of blurring the area other than the attention point using the received image and the attention point. The process performed by the blurring processing unit 25 may be, for example, a process known as a blurring process in image processing software. As an example, this process may be a process of replacing the color of a pixel with the average color of the surroundings in a predetermined range centered on the pixel.

[0061] In this way, the blurring process is performed on the area other than the attention area, so that the user can easily focus on the attention area. For example, when the reference image and the self-image that have not been blurred are combined and both images are included in the combined image, the background of the self-image may be displayed in the area of ​​the imitation target, or the background of the reference image may be displayed in the area of ​​the motion target, and in such a situation, it may be difficult for the user to easily identify the imitation target or the motion target. On the other hand, when the blurring process is performed on the area other than the attention area of ​​at least one of the images, for example, the blurred area of ​​the self-image is displayed in the area of ​​the imitation target, or the blurred area of ​​the reference image is displayed in the area of ​​the motion target, so that the user can easily identify the imitation target or the motion target.

[0062] In addition, in the present embodiment, the reference image, the self-image, and the composite image are mainly described as two-dimensional single images, but these images may be stereo images. In this case, the image output unit 14 may output the composite image, which is a stereo image, to a head-mounted display worn by the user 30.

[0063] In the above embodiment, the image synthesizer 1 is described as a standalone device, but the image synthesizer 1 may be a standalone device or a server device in a server-client system. In the latter case, the reception unit, acquisition unit, and output unit may receive input, acquire information, and output information via a communication line.

[0064] In addition, in the above embodiments, each process or function may be realized by centralized processing in a single device or a single system, or may be realized by distributed processing in multiple devices or multiple systems.

[0065] In addition, in the above embodiments, the transfer of information between each component may be performed, for example, by one component outputting information and the other component receiving information if the two components transferring the information are physically different, or, if the two components transferring the information are physically the same, by shifting from a processing phase corresponding to one component to a processing phase corresponding to the other component.

[0066] In the above embodiment, information related to the processing performed by each component, such as information accepted, acquired, selected, generated, transmitted, or received by each component, and information such as thresholds, formulas, and addresses used by each component in processing, may be temporarily or long-term stored in a recording medium (not shown) even if not specified in the above description. Furthermore, each component or a storage unit (not shown) may store information in the recording medium (not shown). Furthermore, each component or a reading unit (not shown) may read information from the recording medium (not shown).

[0067] In the above embodiment, if the information used by each component, such as the thresholds, addresses, and various setting values ​​used by each component in processing, may be changed by the user, the user may be able to change the information as appropriate, even if not specified in the above description. If the information is changeable by the user, the change may be realized, for example, by a reception unit (not shown) that receives a change instruction from the user, and a change unit (not shown) that changes the information in response to the change instruction. The reception unit (not shown) may receive the change instruction from an input device, may receive information transmitted via a communication line, or may receive information read from a predetermined recording medium.

[0068] In addition, in the above embodiment, when two or more components included in the image synthesis device 1 have a communication device, an input device, etc., the two or more components may physically have a single device, or may have separate devices.

[0069] In the above embodiment, each component may be configured by dedicated hardware, or a component that can be realized by software may be realized by executing a program. For example, each component may be realized by a program execution unit such as a CPU reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. During execution, the program execution unit may execute the program while accessing a storage unit or a recording medium. The software that realizes the image synthesis device 1 in the above embodiment is a program as follows. That is, this program may be a program that causes a computer that can access a storage unit in which a reference image, which is an image of an imitation target whose behavior is to be imitated by a user, to function as an image acquisition unit that acquires a self-image, which is an image of an action target whose behavior is to be operated by a user, a synthesis unit that synthesizes the reference image and the self-image to generate a synthetic image, and a video output unit that outputs the synthetic image, and the synthesis unit synthesizes both images so that the ratio of the synthesis of the reference image and the self-image changes continuously and repeatedly over time.

[0070] In addition, the functions realized by the above program do not include functions that can only be realized by hardware, such as a modem or an interface card in an acquisition unit that acquires information or an output unit that outputs information.

[0071] This program may be executed by being downloaded from a server or the like, or may be executed by reading out a program recorded on a predetermined recording medium (for example, an optical disk such as a CD-ROM, a magnetic disk, a semiconductor memory, etc.). This program may be used as a program constituting a program product.

[0072] Furthermore, the program may be executed by a single computer or a plurality of computers. That is, the program may be executed by centralized processing or distributed processing.

[0073] FIG. 13 is a diagram showing an example of the configuration of a computer 900 that executes the above program to realize the image synthesizing device 1 according to the above embodiment. In FIG. 13, the computer 900 includes an MPU (Micro Processing Unit) 911, a ROM 912 for storing programs such as a boot-up program, a RAM 913 connected to the MPU 911 to temporarily store instructions of application programs and provide a temporary storage space, a storage unit 914 for storing application programs, system programs, and data, and a communication module 915 for providing a connection to a LAN, a WAN, or the like. The MPU 911, the ROM 912, and the like may be connected to each other by a bus together with a camera 901, a display device 902, a keyboard 903, and a pointing device 904 such as a touchpad or a mouse. The storage unit 914 may be, for example, a hard disk or a solid state drive (SSD). The camera 901, the display device 902, the keyboard 903, the pointing device 904, and the like may be, for example, external devices, or may be devices built into the computer 900.

[0074] A program for causing the computer 900 to execute the functions of the image synthesizing device 1 according to the above-described embodiment may be loaded into the RAM 913 at the time of execution. Note that the program may be loaded directly from the storage unit 914 or a network, for example.

[0075] The program does not necessarily include an operating system (OS) or a third-party program that causes the computer 900 to execute the functions of the image synthesizing device 1 according to the above embodiment. The program may include only an instruction portion that calls appropriate functions or modules in a controlled manner to obtain a desired result. How the computer 900 operates is well known, and a detailed description will be omitted.

[0076] The above-mentioned embodiments are merely examples for specifically implementing the present invention, and are not intended to limit the technical scope of the present invention. The technical scope of the present invention is indicated by the claims, not by the description of the embodiments, and is intended to include modifications within the literal scope of the claims and the scope of equivalent meanings. [Explanation of symbols]

[0077] 1. Video composition equipment 11 Storage section 12 Image acquisition section 13 Synthesis section 14 Video output section 21 Distance acquisition part 22 Background Change Section 23 Reception 24 Specific section 25 Processing section

Claims

1. A storage unit that stores a reference video, which is a video of a target motion that a user wishes to imitate; an image acquisition unit that acquires a self-image, which is an image of an action of an action target that is an object to be performed by a user; a synthesis unit that synthesizes the reference image and the self image to generate a synthetic image; a video output unit that outputs the composite video, The synthesis unit synthesizes the reference image and the self-image such that a synthesis ratio between the two images changes continuously and repeatedly over time.

2. 2. The image synthesizing device according to claim 1, wherein the synthesizing section synthesizes the reference image and the self-image such that a change over time in a rate at which the two images are synthesized forms a sine wave.

3. A distance acquisition unit that acquires a distance between the imitation target and the action target in the synthetic image, The image synthesizing device according to claim 1 , wherein the synthesizing section performs synthesis such that the self-image is displayed more as the distance increases.

4. A background change unit is further provided for changing a background area of ​​at least one of the reference image and the self image to be changed, to a transparent or predetermined single color; 3. The image synthesizing device according to claim 1, wherein the synthesizing section synthesizes the image to be changed with the image changed by the background changing section.

5. An identification unit that identifies a point of interest in an image of an identification target, which is at least one of the reference image and the self-image; A blurring processing unit that performs a process of blurring an area other than the identified attention point in the image of the identified target, 3. The image synthesizing device according to claim 1, wherein the synthesizing unit synthesizes the image of the specified target with the image that has been blurred by the blurring unit.

6. The image synthesizing apparatus according to claim 5 , wherein the specifying unit specifies a target portion that is a portion where there is movement.

7. A reception unit is further provided for receiving attention point information indicating an attention point from a user, The image synthesizing device according to claim 5 , wherein the specifying unit specifies the point of interest in accordance with the point of interest information received by the receiving unit.

8. A video synthesis method that is processed using a storage unit that stores a reference video, which is a video of a target action that a user is to imitate, a video acquisition unit, a synthesis unit, and a video output unit, comprising: The image acquisition unit acquires a self-image, which is an image of an action of an action target that is an object to be performed by a user; a step of the synthesis unit synthesizing the reference image and the self-image to generate a synthetic image; the video output unit outputs the composite video, In the step of generating a composite image, the reference image and the self-image are composited such that a proportion of the composite image between the reference image and the self-image changes continuously and repeatedly over time.

9. A computer that can access a storage unit that stores a reference video, which is a video of a target motion that a user is to imitate, an image acquisition unit that acquires a self-image, which is an image of an action of an action target that is an object to be performed by a user; a synthesis unit that synthesizes the reference image and the self-image to generate a synthetic image; a video output unit that outputs the composite video; The synthesis unit synthesizes the reference image and the self-image so that a ratio of the two images to be synthesized varies continuously and repeatedly over time.

Images