Video generation method and program

Abstract

To prevent reflection of an unnecessary motion when operating a virtual character in accordance with a person's motion.SOLUTION: In a video generation method, a video generation apparatus acquires information indicating a user's motion from a plurality of motion capture units worn by the user and generates a video obtained by reflecting the information on a motion of a virtual character. In the video generation method, the video generation apparatus detects the distance between a preset reference position and the plurality of motion capture units and stops the reflection of the information indicating the user's motion on the motion of the virtual character when detecting that the distance from the reference position to at least one motion capture unit of the plurality of motion capture units satisfies a predetermined condition (YES in step S5).SELECTED DRAWING: Figure 12

Description

[Technical field]

[0001] The present disclosure relates to a video generation method and a program. [Background technology]

[0002] Patent Document 1 discloses an information processing device that recognizes a motion rhythm that indicates the rhythm of a user's movement based on detection data that indicates the movement of detection positions detected by a sensor device at one or more detection positions including at least one position on the user's trunk. Patent Document 1 also discloses a technology that causes an avatar corresponding to the user to perform a motion based on the analysis result of the motion. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Patent Publication No. 2023-022265 Summary of the Invention [Problem to be solved by the invention]

[0004] However, in the technology described in Patent Document 1, since the avatar can be operated in real time, when the video of the avatar is stored or distributed, unnecessary actions are also stored or distributed. Therefore, it is desired to develop a technology that does not reflect unnecessary actions when operating a virtual character such as an avatar in accordance with the movement of a person. [Means for solving the problem]

[0005] The image generation method according to the present disclosure is performed by an image generation device, comprising: a motion acquisition step of acquiring information indicating a user's movement from a plurality of motion capture units worn by a user; a character image generation step of generating an image in which the information indicating the user's movement acquired in the motion acquisition step is reflected in the movement of a virtual character; a detection step of detecting a distance between a preset reference position and the plurality of motion capture units; and a stop step of stopping the reflection of the information indicating the user's movement in the movement of the virtual character when it is detected in the detection step that at least one of the plurality of motion capture units is at a distance from the reference position that satisfies a predetermined condition.

[0006] The program of the present disclosure is executed by a computer that operates an image generating device, and includes a motion acquisition step of acquiring information indicating a user's movement from a plurality of motion capture units worn by a user; a character image generating step of generating an image in which the information indicating the user's movement acquired in the motion acquisition step is reflected in the movement of a virtual character; a detection step of detecting the distance between a predetermined reference position and the plurality of motion capture units; and a stopping step of stopping the reflection of the information indicating the user's movement in the movement of the virtual character when it is detected in the detection step that at least one of the plurality of motion capture units is at a distance from the reference position that satisfies a predetermined condition.

[0007] Another image generation method of the present disclosure is one in which an image generation device executes the following steps: an image acquisition step for acquiring image of a person; a motion detection step for detecting information indicating the movement of the person being photographed from the image acquired in the image acquisition step; a character image generation step for generating an image in which the information indicating the movement of the person detected in the motion detection step is reflected in the movement of a virtual character; a detection step for detecting the distance between a predetermined reference position and the person; and a stopping step for stopping the reflection of the information indicating the movement of the person in the movement of the virtual character when it is detected in the detection step that the distance between the reference position and the person has become a distance that satisfies a predetermined condition.

[0008] Another program of the present disclosure is executed by a computer that operates an image generating device, comprising: an image acquisition step of acquiring an image of a person; a motion detection step of detecting information indicating the movement of the person being photographed from the image acquired in the image acquisition step; a character image generation step of generating an image in which the information indicating the movement of the person detected in the motion detection step is reflected in the movement of a virtual character; a detection step of detecting the distance between a predetermined reference position and the person; and a stopping step of stopping the reflection of the information indicating the movement of the person in the movement of the virtual character when it is detected in the detection step that the distance between the reference position and the person has become a distance that satisfies a predetermined condition. Effect of the Invention

[0009] According to the present disclosure, when operating a virtual character in accordance with the movements of a person, it is possible to prevent unnecessary movements from being reflected. [Brief description of the drawings]

[0010] [Figure 1] 1 is a schematic diagram showing an example of a configuration of an image generation system including an image generation device according to a first embodiment. [Diagram 2] FIG. 2 is a functional block diagram showing details of the video production system of FIG. 1. [Diagram 3]3 is a schematic diagram showing a front view of an example of a motion capture unit included in the video production system of FIG. 2 worn by a user. FIG. [Figure 4] 4 is a schematic diagram showing a top view of the example user of FIG. 3. [Diagram 5] 3 is a diagram showing an example of a virtual character generated by the image generating device appearing in FIG. 2. [Figure 6] FIG. 3 is a diagram for explaining an example of a detection process executed by the image generating device in FIG. 2, and is a schematic diagram showing an example of a positional relationship between the image generating device and a user as viewed from the right side. [Figure 7] 7 is a schematic diagram showing the example of the positional relationship in FIG. 6 as viewed from above. [Figure 8] 3 is a schematic diagram showing an example of a case in which the distance of a motion capture unit to an image production device in the image production system of FIG. 2 becomes less than a predetermined distance, as viewed from the right side. FIG. [Figure 9] 9 is a diagram showing an example of a virtual character generated by the image generating device in FIG. 2 in the case shown in FIG. 8. FIG. [Figure 10] FIG. 4 is a diagram for explaining another example of the detection process executed by the image generating device in FIG. 2, and is a schematic diagram showing an example of the positional relationship between the image generating device and a user as viewed from the right side. [Figure 11] 11 is a schematic diagram showing the example of the positional relationship in FIG. 10 as viewed from above. [Figure 12] 4 is a flow diagram illustrating an example of an image generation method executed by the image generation device according to the first embodiment. FIG. [Figure 13] FIG. 11 is a schematic diagram showing a configuration example of an image generating device according to a second embodiment. [Figure 14] FIG. 14 is a functional block diagram showing details of the image generating device of FIG. 13. [Figure 15] 15 is a schematic diagram showing an example of a person photographed by the image generating device of FIG. 14 as viewed from the front. FIG. [Figure 16] 16 is a schematic diagram showing a top view of the example of the person in FIG. 15. FIG. [Figure 17]FIG. 15 is a diagram showing an example of a virtual character generated by the image generating device of FIG. 14. [Figure 18] FIG. 15 is a diagram for explaining an example of a detection process executed by the image generating device of FIG. 14, and is a schematic diagram showing an example of a positional relationship between the image generating device and a person as viewed from the right side. [Figure 19] 19 is a schematic diagram showing the example of the positional relationship in FIG. 18 as viewed from above. [Figure 20] 15 is a schematic diagram showing an example of a case where the distance of a person to the image generating device of FIG. 14 becomes less than a predetermined distance, as viewed from the right side. FIG. [Figure 21] 21 is a diagram showing an example of a virtual character generated by the image generating device in FIG. 14 in the case shown in FIG. 20. FIG. [Figure 22] FIG. 15 is a diagram for explaining another example of the detection process executed by the image generating device of FIG. 14, and is a schematic diagram showing an example of the positional relationship between the image generating device and a person as viewed from the right side. [Figure 23] 23 is a schematic diagram showing the example of the positional relationship in FIG. 22 as viewed from above. FIG. [Figure 24] 11 is a flow diagram illustrating an example of an image generating method executed by an image generating device according to the second embodiment. FIG. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, specific embodiments to which the present invention is applied will be described in detail with reference to the drawings. However, the present disclosure is not limited to the following embodiments. In addition, the following description and drawings are appropriately simplified for clarity of explanation.

[0012] <Embodiment 1> A configuration example of a video generation system including a video generation device according to the first embodiment will be described with reference to Figs. 1 to 11. Fig. 1 is a schematic diagram showing a configuration example of a video generation system including a video generation device according to the present embodiment. Fig. 2 is a functional block diagram showing details of the video generation system of Fig. 1. Fig. 3 is a schematic diagram showing a front view of an example in which a user wears a motion capture unit included in the video generation system of Fig. 2. Fig. 4 is a schematic diagram showing a top view of the example of the user of Fig. 3. Fig. 5 is a diagram showing an example of a virtual character generated by the video generation device in Fig. 2.

[0013] As shown in Fig. 1, an image generating device 10 according to this embodiment constitutes an image generating system 1 together with a plurality of motion capture units 20. For convenience, in Fig. 1, the description will be given on the assumption that the image generating device 10 is an information processing device such as a smartphone or a PC (Personal Computer), but, for example, the functions of the image generating device 10 described below may be incorporated into a general-purpose information processing device. Alternatively, the image generating device 10 described below may be configured as a dedicated device rather than being incorporated into a general-purpose information processing device.

[0014] First, the motion capture unit 20 will be described. As shown in Fig. 2, each motion capture unit 20 includes, for example, a control device 21, a sensor unit 22, and a communication unit 23. The control device 21 only needs to be able to control the sensor unit 22 and the communication unit 23. For example, the control device 21 includes a communication control unit 211 that controls communication by the communication unit 23, and a sensor control unit 212 that controls the sensor unit 22 and obtains detection results.

[0015] The sensor unit 22 is a sensor that acquires information indicating the movement of the user of the motion capture unit 20, including acceleration information and angular velocity information, and may be a sensor that can detect 6DoF (six degrees of freedom) of the mounted motion capture unit 20. For example, the sensor unit 22 is a 6DoF sensor that can measure three-axis acceleration (X, Y, Z) and angular velocity (roll, pitch, yaw). The sensor unit 22 may detect an angle instead of the angular velocity. The sensor unit 22 may also be, for example, an inertial measurement unit (IMU). The sensor unit 22 may also include a geomagnetic sensor.

[0016] The communication unit 23 communicates with the image generation device 10. The communication unit 23 is a communication unit that communicates in accordance with a short-range wireless communication standard such as Bluetooth (registered trademark; the same applies below) LE (Low Energy).

[0017] The multiple motion capture units 20 are arranged at different positions on the user's body. For example, the multiple motion capture units 20 are arranged at positions where the shape of the user's body can be expressed by a combination of their positions, that is, where the movement of the user can be expressed by a combination of changes in their positions. In the following description, the number of motion capture units 20 arranged on the user is n.

[0018] For example, as illustrated in Fig. 3 and Fig. 4, motion capture units 20He, 20Bc, 20RHa, 20LHa, 20RF, and 20LF are worn by a user U as n motion capture units 20. In this example, n=6. Here, the motion capture units 20He, 20Bc, 20RHa, 20LHa, 20RF, and 20LF are for head mounting, body center mounting, right hand mounting, left hand mounting, right foot mounting, and left foot mounting, respectively. In the following, motion capture units that do not depend on the mounting position will be described as motion capture units 20. The positions, number, and arrangement of the motion capture units 20 are not limited to the examples in Figs. 3 and 4.

[0019] Next, a description will be given of the image generating device 10. The image generating device 10 includes, for example, a control device 11, a display unit 121, an operation unit 122, and a communication unit 13, as shown in FIG.

[0020] The control device 11 may be realized, for example, by a processor such as a CPU (Central Processing Unit), a working memory, and a non-volatile storage device storing a control program. The program may include a program that, when executed by the processor, realizes the functions of the image generating device 10 described below. The control device 11 may also be realized using an integrated circuit programmable by a user, such as an FPGA (field-programmable gate array) or a microcomputer.

[0021] The display unit 121 may be configured with a display device such as a liquid crystal panel, an organic electroluminescence panel, or the like. The operation unit 122 is provided for operating the image generating device 10, and may include a touch sensor incorporated in the display unit 121. In that case, as shown in Fig. 1, the image generating device 10 includes the display unit 121 and a touch panel 12 as an example of the operation unit 122. In this way, the operation unit 122 may be configured to include a software key, or may be configured to include a hardware key, or may be configured without including a software key.

[0022] The communication unit 13 communicates with the motion capture unit 20. The communication unit 13 is a communication unit capable of communicating with the communication unit 23, and is, for example, a communication unit that operates in accordance with a short-range wireless communication standard such as Bluetooth LE. The communication unit 13 may further include a communication interface that communicates in accordance with a wireless communication standard such as Wi-Fi (registered trademark, the same applies below) or the fifth generation mobile communication system. Through this communication interface, the communication unit 13 can transmit images generated by a character image generation unit 114 (described later) to an external device via a network.

[0023] The main functions of the image generating device 10 may be realized, for example, as functions of the control device 11. For this reason, the control device 11 includes, for example, a communication control unit 111, a motion acquisition unit 112, a detection unit 113, a character image generating unit 114, a display control unit 115, and an operation control unit 116, as described below.

[0024] The communication control unit 111 receives information indicating the user's motion from the n motion capture units 20 worn by the user via the communication unit 13, and passes the information to the motion acquisition unit 112. Hereinafter, the acquired information indicating the user's motion is referred to as user motion information. The user motion information for a certain motion capture unit 20 is obtained, for example, as information indicating a change in the position of the motion capture unit 20, or as time-series position information. The user motion information may be referred to as capture information, since it is information that captures the motion of the user U. It should be noted that the user of the motion capture unit 20 is also the user of the video generation device 10. The communication control unit 111 may also control the transmission of the video generated by the character video generation unit 114 to an external device via the communication unit 13.

[0025] The motion acquisition unit 112 receives user motion information transmitted from the n motion capture units 20 worn by the user via the communication control unit 111. Here, the motion acquisition unit 112 may request the communication control unit 111 to acquire user motion information periodically or at a predetermined timing, and receive the user motion information from the communication control unit 111 in response to the request. However, the n motion capture units 20 may transmit user motion information periodically or at a predetermined timing, and the motion acquisition unit 112 may receive the user motion information via the communication unit 13 and the communication control unit 111. The motion acquisition unit 112 passes the received user motion information to the character image generation unit 114 and the detection unit 113.

[0026] The character image generating unit 114 generates an image in which the movement of the virtual character reflects the user action information acquired by the action acquiring unit 112. When the user U is standing upright as in the examples of Figures 3 and 4, an image V1 including a virtual character standing upright is generated, such as the virtual character Vc1 illustrated in Figure 5.

[0027] Information required to generate a virtual character, such as information on the shape and other characteristics required to render the virtual character, may be stored in a storage device built into the image generating device 10. Similarly, the background part of the virtual character in the image may be stored in a storage device built into the image generating device 10. Of course, a fixed image may be used as the background part, but a changing image may also be used. By making the background part an image that changes, the virtual character can be moved in a game, a virtual space, or other types of applications that change depending on the scene. Also, the image generating device 10 may generate a virtual character and transmit only an image showing the generated virtual character or information showing the movement of the virtual character to an external device without generating the background part.

[0028] However, in this embodiment, any method can be used for generating a virtual character, for superimposing or embedding a virtual character in an image, or for reflecting user operation information in the movement of a virtual character.

[0029] Furthermore, character image generation unit 114 may pass the generated image to display control unit 115 so that it is displayed on display unit 121, and as a result, the generated image is displayed on display unit 121. Furthermore, when the image is to be stored in a storage device built into image generation device 10, character image generation unit 114 stores the generated image in that storage device. Furthermore, when the image is to be transmitted to an external device, character image generation unit 114 passes the generated image to communication control unit 111 so that it is transmitted via communication unit 13.

[0030] The detection unit 113 detects the distance between a preset reference position and the n motion capture units 20. The detection unit 113 may calculate the positions of the n motion capture units 20 based on the user motion information received from the motion acquisition unit 112, and calculate the distance from the reference position. This is because the user motion information includes information indicating the position of the user U as the position of the motion capture unit 20, or information indicating the position of the user U can be calculated from the user motion information.

[0031] Alternatively, the detection unit 113 may calculate the distance between the reference position and each motion capture unit 20 without relying on user operation information. For example, the position of the image generation device 10 may be used as a reference, and the distance between the image generation device 10 and each motion capture unit 20 may be calculated based on information obtained by communication between them. To give a more specific example, when the communication unit 13 communicates with each motion capture unit 20 in accordance with the above-mentioned Bluetooth LE standard, the detection unit 113 may calculate the distance between the position of the image generation device 10 and each motion capture unit 20 using a positioning technology of Bluetooth LE or Wi-Fi. Note that, when the reference position is the position of the image generation device 10 as in the example described later, the calculated distance becomes the distance between the reference position and each motion capture unit 20. When the reference position is set to a position other than the above, the distance between the reference position and each motion capture unit 20 can be calculated based on the difference between the position of the image generation device 10 and the reference position in addition to the distance between the position of the image generation device 10 and each motion capture unit 20.

[0032] Furthermore, when at least one of the n motion capture units 20 is at a distance from the reference position that satisfies a predetermined condition, the detection unit 113 detects this and notifies the character image generation unit 114. In the following, it is assumed that a notification (hereinafter, a condition match notification) is made when it is detected that the distance satisfies the predetermined condition. In this case, when the detection unit 113 detects that the distance no longer satisfies the predetermined condition, it is preferable that the detection unit 113 passes a notification indicating return to the character image generation unit 114. However, the detection unit 113 may determine whether or not at least one of the n motion capture units 20 is at a distance from the reference position that satisfies a predetermined condition, and notify the character image generation unit 114 of the determination result. In this case, the character image generation unit 114 may determine whether or not to execute a reflection stop process described later based on the determination result.

[0033] When character image generation unit 114 receives the above-mentioned condition match notification from detection unit 113, it executes a process of stopping the reflection of the user action information in the movement of the virtual character (hereinafter, reflection stop process). That is, when detection unit 113 detects that at least one of n motion capture units 20 has become a distance from the reference position that satisfies a predetermined condition, character image generation unit 114 executes the reflection stop process.

[0034] An example of the detection process in the detection unit 113 and the reflection stop process by the character image generation unit 114 will be described with reference to Figs. 6 to 9. Fig. 6 is a diagram for explaining an example of the detection process executed by the image generation device 10 in Fig. 2, and is a schematic diagram showing an example of the positional relationship between the image generation device 10 and the user U as viewed from the right side. Fig. 7 is a schematic diagram showing an example of the positional relationship in Fig. 6 as viewed from above. Fig. 8 is a schematic diagram showing an example of a case in which the distance of the motion capture unit 20 to the image generation device 10 in the image generation system 1 in Fig. 2 becomes less than a predetermined distance as viewed from the right side. Fig. 9 is a diagram showing an example of a virtual character generated by the image generation device 10 in Fig. 2 in the case shown in Fig. 8.

[0035] As described above, detection unit 113 detects that the distance from the reference position satisfies a predetermined condition, and passes a condition satisfaction notification to character image generation unit 114. Here, an example is given in which this reference position is the position of image generation device 10. That is, in the example described here, detection unit 113 detects the distance between the reference position and n motion capture units 20, using the position of image generation device 10 as the reference position. Note that, although the description will be given on the premise that the position of image generation device 10 as the reference position is a position on a horizontal plane, the height of the reference position may also be taken into consideration.

[0036] As the above-mentioned predetermined condition, various conditions can be adopted, such as a first condition that the distance is less than a first predetermined distance (hereinafter, the first distance) on the horizontal plane, and a second condition that the distance is equal to or greater than a second predetermined distance (hereinafter, the second distance) on the horizontal plane. Of course, as the predetermined condition, a third condition that the distance is less than the first distance or equal to or greater than the second distance on the horizontal plane may be adopted. Here, the second distance is assumed to be greater than the first distance. The first distance and the second distance can be expressed by Th1 and Th2 in Figs. 6 and 7, respectively, and may be predetermined values, but may be variable values ​​as described later. Note that Figs. 6 and 7 show an example in which the image generating device 10 is mounted on a mounting base 10Ho. Of course, the predetermined condition is not limited to an example defined only by the distance on the horizontal plane, and a condition that the distance falls outside a rectangular parallelepiped area having a center at a position a predetermined distance away from a reference position in a predetermined direction and at a predetermined elevation angle may also be adopted.

[0037] The first distance Th1 may be preset to, for example, 1.0 m from the image generating device 10. The second distance Th2 may be preset to, for example, 5.0 m from the image generating device 10. The first distance Th1 and the second distance Th2 may be determined based on the position of a predetermined motion capture unit 20 such as the motion capture unit 20Bc when performing calibration, which will be described later. For example, if the position of the motion capture unit 20Bc when performing calibration is 3.0 m from the image generating device 10, the first distance Th1 and the second distance Th2 may be set to ±D (m) from that position. If D=2.0 m, the first distance Th1 is set to 1.0 m from the image generating device 10, and the second distance Th2 is set to 5.0 m from the image generating device 10. Of course, D is not limited to 2.0 m, and the predetermined motion capture unit 20 is not limited to the motion capture unit 20Bc, such as the motion capture unit 20He.

[0038] By adopting the first condition as the predetermined condition, the detection unit 113 and the character image generation unit 114 carry out the following process. When at least one of the motion capture units 20 is at a distance less than the first distance Th1 in the horizontal plane from the position of the image generation device 10, the detection unit 113 issues a condition satisfaction notification, and the character image generation unit 114 executes a reflection stop process.

[0039] 8, when motion capture unit 20LHa becomes less than first distance Th1 from the position of image generation device 10, detection unit 113 issues a condition satisfaction notification, and character image generation unit 114 executes reflection stop processing. When motion capture unit 20LHa becomes less than first distance Th1 from image generation device 10, this reflection stop processing stops the image of the virtual character in the image at that time, for example, in the state shown by virtual character Vc2 in image V2 in FIG.

[0040] In fact, when the motion capture unit 20 becomes less than the first distance Th1 from the image generating device 10, this occurs when the user U wearing the motion capture unit 20 performs the following action from the image generating state. That is, a situation occurs in which the user U performs an action of operating the operation unit of the image generating device 10 to stop image generation, storage, distribution, etc., or to change settings, causing the distance to become less than the first distance Th1. However, in this situation, the character image generating unit 114 executes a reflection stop process, so that such an action of the user U does not need to be reflected in the movement of the virtual character. For example, as described above, the image V2 generated by this reflection stop process is an image of the virtual character Vc2 that reflects the movement of the user U just before the motion capture unit 20LHa becomes less than the first distance Th1 from the image generating device 10. Therefore, according to this embodiment, when operating the virtual character in accordance with the movement of the user U, it is possible to prevent unnecessary actions from being reflected.

[0041] This effect will be described by taking a specific usage scene as an example. For example, when the user U uses the image generating device 10 to transmit the motion of a virtual character to an external device, such as by live streaming the motion of the virtual character, the motion during the period when at least one of the motion capture units 20 is less than the first distance Th1 from the image generating device 10 is not transmitted. Therefore, even in such a case, the image generating device 10 can transmit an appropriate motion of the virtual character. Also, for example, when the user U uses the image generating device 10 to store the motion of a virtual character, the motion during the period when at least one of the motion capture units 20 is less than the first distance Th1 from the image generating device 10 is not stored. Therefore, even in such a case, the image generating device 10 can reduce the number of editing steps after storage.

[0042] Furthermore, the reflection stop processing may be accompanied by the following correction processing. That is, when the reflection of user action information on the movement of the virtual character is stopped by the reflection stop processing, the character image generation unit 114 may perform a correction to transition the virtual character from the action at the time of the stop to a predetermined action. This transition may be an instantaneous switch from the virtual character before the correction to the virtual character after the correction, or a gradual change from the virtual character before the correction to the virtual character after the correction.

[0043] The image generated by this modification process transitions from a stopped image V2 to an image V3 including a virtual character Vc3, as shown in Fig. 9. The image V3 is an image in which the virtual character Vc2 reflecting the user's action information immediately before the motion capture unit 20LHa becomes less than the first distance Th1 from the image generation device 10 in the image V2 is stopped, and the virtual character Vc2 is modified to the virtual character Vc3.

[0044] Here, the virtual character Vc3 is an example of a virtual character showing a form during calibration, which will be described later, and such a correction can be read out by storing information on the virtual character during calibration. However, the virtual character Vc3 does not have to be a virtual character showing a form of calibration, and may be, for example, a predetermined virtual character. In that case, a character indicating that an operation is being performed may be added to the virtual character.

[0045] In this embodiment, by employing such correction processing, when operating a virtual character in accordance with the movement of a person, it is possible to prevent unnecessary movements from being reflected as they are.

[0046] This effect will be described by taking a specific usage scene as an example. For example, when the user U uses the image generating device 10 to transmit the motion of a virtual character to an external device, such as by live streaming the motion of the virtual character, the motion during the period when at least one of the motion capture units 20 is less than the first distance Th1 from the image generating device 10 is corrected and transmitted. Even in such a case, the image generating device 10 can transmit an appropriate motion of the virtual character. Also, for example, when the user U uses the image generating device 10 to store the motion of a virtual character, the motion during the period when at least one of the motion capture units 20 is less than the first distance Th1 from the image generating device 10 is corrected and stored. Even in such a case, the image generating device 10 can reduce the number of editing steps after storage.

[0047] Next, a case where the second or third condition is adopted as the predetermined condition will be described. By adopting the second condition as the predetermined condition, the detection unit 113 and the character image generation unit 114 will carry out the following process. When at least one motion capture unit 20 is at a distance equal to or greater than the second distance Th2 in the horizontal plane from the position of the image generation device 10, the detection unit 113 issues a condition satisfaction notification, and the character image generation unit 114 executes the reflection stop process. In this case as well, the correction process described above may be adopted.

[0048] Furthermore, by adopting the third condition as the predetermined condition, the detection unit 113 and the character image generation unit 114 will carry out the following process. When at least one of the motion capture units 20 is at a distance less than the first distance Th1 or at a distance equal to or greater than the second distance Th2 from the position of the image generation device 10 in the horizontal plane, the detection unit 113 issues a condition satisfaction notification, and the character image generation unit 114 executes the reflection stop process. In this case as well, the correction process described above may be adopted.

[0049] Next, an example will be given in which the first distance Th1 and the second distance Th2 are variable values. The first distance Th1 and the second distance Th2 may be changed according to attribute information indicating the attribute of the user U, for example. This attribute information may be information indicating whether the user U is a main user or a guest user of the image generating device 10. This information may be, for example, information on logging in to the image generating device 10, or may receive a main / guest designation operation or a user name designation operation at the start of use.

[0050] The main user is as likely to perform an operation to stop image generation as the guest user, but is more likely to change some settings of the image generation device 10. Therefore, if the first distance Th1 is set larger for the main user than for the guest user, the reflection stop process will not be performed even if the guest user unintentionally approaches the image generation device 10 without intending to operate it. Also, the guest user is more likely to move around without being aware of the execution of the reflection stop process than the main user. Therefore, if the second distance TH2 is set larger for the guest user than for the main user, the reflection stop process will not be performed even if the guest user unintentionally moves far away from the image generation device 10.

[0051] Furthermore, the first distance Th1 and the second distance Th2 may be changed to smaller and larger values, respectively, depending on the intensity of the movement of the user U. By making such changes, it is possible to prevent the reflection stop process from being executed even if the user U unintentionally moves outside the range of the distance Th1 or more and less than Th2 due to intense movement.

[0052] Another example of the reference position will be described with reference to Fig. 10 and Fig. 11. Fig. 10 is a diagram for explaining another example of the detection process executed by the image generating device 10 in Fig. 2, and is a schematic diagram showing an example of the positional relationship between the image generating device 10 and a user as viewed from the right side. Fig. 11 is a schematic diagram showing the example of the positional relationship in Fig. 10 as viewed from above.

[0053] The reference position may be the position of the user (hereinafter, the calibration position) when calibrating the process of acquiring user motion information in the image generating device 10. In other words, the detection unit 113 may detect the distance between the reference position and the n motion capture units 20 by using the calibration position as the reference position.

[0054] The calibration position as the reference position is the position of the user U when the calibration is performed, and this position may be defined as follows. That is, the position of the user U when the calibration is performed may be the position of each of the n motion capture units 20 when the calibration is performed, and the detection unit 113 may detect the distance from the reference position corresponding to each motion capture unit 20. Alternatively, the position of the user U when the calibration is performed may be the position of a specific motion capture unit 20 such as the motion capture unit 20Bc when the calibration is performed, and the detection unit 113 may detect the distance from a reference position common to the n motion capture units 20. Alternatively, the position of the user U when the calibration is performed may be the position of the center of gravity or the center of the positions of the n motion capture units 20 when the calibration is performed, and the detection unit 113 may detect the distance from a reference position common to the n motion capture units 20.

[0055] In the calibration, for example, the user U wears n motion capture units 20 and stands upright at the start of the calibration as illustrated in Fig. 3 and Fig. 4. The position of each motion capture unit 20 acquired in this state is registered as the position of the virtual character standing upright in the center of the image. For example, for the motion capture unit 20Rha, the position represented by the distance Ca shown in Fig. 10 in the depth direction of the horizontal plane is registered as the position of the virtual character standing upright in the center of the image, and the same is also registered for the width direction and vertical direction of the horizontal plane, although not shown. The same is true for the motion capture units 20 other than the motion capture unit 20Rha.

[0056] In this way, the image corresponding to the position registered when calibration is performed becomes, for example, image V1 including virtual character Vc1. After calibration is completed, the movement of user U is reflected in the movement of the virtual character with the calibration position as the reference position. Note that the movement acquisition unit 112 acquires the current position of each motion capture unit 20 for this reflection, and the calibration position that is the reference for this current position may be the position of each motion capture unit 20 at the time of calibration.

[0057] As described above, the detection unit 113 detects that the distance from the reference position satisfies a predetermined condition, and passes a condition satisfaction notification to the character image generation unit 114. In the example described here, the detection unit 113 detects the distance between the reference position and the n motion capture units 20, with the calibration position being the reference position.

[0058] In this case, the above-mentioned predetermined condition may be, for example, a condition of being equal to or greater than a third predetermined distance (hereinafter, the third distance) on the horizontal plane, as exemplified in Fig. 10 and Fig. 11. Note that Fig. 10 and Fig. 11 show an example in which the position of the motion capture unit 20Rha at the time of calibration is set as the reference position. Of course, the predetermined condition is not limited to an example defined only by the distance on the horizontal plane, and a condition of being outside a rectangular parallelepiped area centered on the calibration position may also be used. Also, although the calibration position as the reference position has been described on the assumption that it is a position on the horizontal plane, the height of the reference position may also be taken into consideration.

[0059] The third distance Th3 may be preset as, for example, 2.0 m. By adopting a condition of the third distance Th3 or more as the predetermined condition, the detection unit 113 and the character image generation unit 114 perform the following process. When at least one motion capture unit 20 is at a distance of the third distance Th3 or more from the calibration position in the horizontal plane, the detection unit 113 notifies the condition satisfaction, and the character image generation unit 114 executes the reflection stop process. For example, when at least one motion capture unit 20 such as the motion capture unit 20LHa is at a distance of the third distance Th3 or more from the calibration position in the horizontal plane, the image of the virtual character is stopped by this reflection stop process in the image at that time, for example, in the state shown by the virtual character Vc2 in the image V2 in FIG. 9. As a result, even when the calibration position is adopted as the reference position, the same effect as when the position of the image generation device 10 is adopted as the reference position is achieved. Moreover, even when the calibration position is adopted as the reference position, the above-mentioned correction process may be adopted.

[0060] Next, a video generation method according to the present embodiment (hereinafter, the present method) will be described. The present method is executed by a video generation device such as the video generation device 10, and will be described below as a method executed by the video generation device 10. Fig. 12 is a flow diagram for explaining an example of the video generation method executed by the video generation device 10 according to the present embodiment.

[0061] First, with the user U standing upright, the image generating device 10 starts communication with each motion capture unit 20 via the communication unit 13 (step S1) and executes calibration (step S2). The position where the user U stands upright when executing calibration may be determined in advance, but it may also be determined roughly as a position facing the image generating device 10 and a predetermined distance away from the image generating device 10.

[0062] Next, the image generating device 10 sets a range for a predetermined condition based on the result of this calibration (step S3). As a result, the predetermined condition is set to a range with the position of the image generating device 10 as the reference position or the calibration position as the reference position.

[0063] Next, the image generating device 10 starts capturing to reflect the movement of the user U in the virtual character (step S4). This capturing includes a process in which the action acquiring unit 112 successively acquires user action information indicating the movement of the user U, and the character image generating unit 114 generates an image in which the user action information is reflected in the movement of the virtual character. Note that the image in which the captured movement of the user U is reflected in the virtual character can be displayed on the display unit 121, stored in a built-in storage device, or distributed to an external device. Whether one or more of the display, storage, and distribution of the image is performed can be determined, for example, according to a setting made by the user U in the image generating device 10.

[0064] After starting capture in this manner, the image generation device 10 generates an image, and also causes the detection unit 113 to detect the distance of the motion capture unit 20 from the reference position and detect whether the distance satisfies predetermined conditions (step S5).

[0065] If the answer is NO in step S5, that is, if the distance does not satisfy the predetermined condition, the image generating device 10 judges whether the capture has ended (step S10), and ends the process if it has ended, and continues the detection in step S5 if it has not ended. The end of the capture may be determined when an end operation is performed by the user U or the like from the operation unit 122. Note that, during the distribution of the image, the capture may be ended, for example, when communication with an external device by the communication unit 13, for example, communication with a network, is interrupted.

[0066] If the answer is YES in step S5, that is, if the distance satisfies the predetermined condition, then character image generation unit 114 instructs motion acquisition unit 112 to interrupt capture (step S6). Note that, although the capture is interrupted to stop the reflection in step S6, the acquisition of user motion information by motion acquisition unit 112 may continue without interruption, and only the reflection of the user motion information to the virtual character may be stopped. Next, character image generation unit 114 generates an image in which the virtual character has been shifted to a reference position (step S7).

[0067] Even in this state, detection by detection unit 113 continues, and detection unit 113 detects whether the distance has become one that does not satisfy the predetermined condition, that is, whether the distance has returned to one at which the reflection stop process is not performed (step S8).

[0068] If the answer is YES in step S8, the image generating device 10 resumes acquisition of user action information in the action acquiring unit 112, and also resumes processing for generating an image in which the user action information is reflected in the movement of the virtual character in the character image generating unit 114 (step S9). After step S9, the process returns to step S5, and detection is performed as to whether or not the distance has reached a predetermined condition. On the other hand, if the answer is NO in step S8, the image generating device 10 determines whether or not the capture has ended, as in step S10 (step S11), and ends the processing if the answer is YES in step S11, and continues the determination in step S8 if the answer is NO in step S11.

[0069] As illustrated in FIG. 12, in this method, an image generating device exemplified as image generating device 10 executes a motion acquisition step, a character image generating step, a detection step, and a stopping step, as described below.

[0070] The motion acquisition step, as exemplified in step S4, acquires user motion information indicating the motion of the user U from the n motion capture units 20 worn by the user U. The character image generation step, as exemplified in step S4, generates an image in which the user motion information acquired in the motion acquisition step is reflected in the motion of the virtual character. The detection step, as exemplified in step S5, detects the distance between a preset reference position and the n motion capture units 20.

[0071] The stopping step is exemplified by the process of step S6 in the case where step S5 is YES. That is, the stopping step executes a reflection stopping process when the detecting step detects that at least one of the n motion capture units 20 is at a distance from the reference position that satisfies a predetermined condition. Also, as exemplified by step S7, the stopping step may further include a correcting step of transitioning the virtual character from the motion at the time of stopping to a predetermined motion after stopping.

[0072] Furthermore, the character image generating unit 114 may execute a reflection stop process when it is detected that at least one of the n predetermined motion capture units 20 has reached a distance from the reference position that satisfies a predetermined condition. The predetermined motion capture unit 20 may be a motion capture unit 20 attached to the left hand, right hand, left foot, right foot, or a position close to these, which often determines the horizontal range of the user U's body.

[0073] That is, in the stopping step, the reflection stopping process may be executed when the detection unit 113 detects that a motion capture unit 20 worn on the hand or foot of the user U or a position close thereto among the n motion capture units 20 is at a distance from the reference position that satisfies a predetermined condition. In the examples of Figures 3 and 4, the motion capture units 20 worn on the hand or foot of the user U or a position close thereto correspond to the motion capture units 20RHa, 20LHa, 20RF, and 20LF for wearing on the right hand, left hand, right foot, and left foot.

[0074] In this embodiment, the program described above as being stored in the control device 11 may be a program such as the following. That is, the program according to this embodiment is a program in which a computer that operates the image generation device exemplified by the image generation device 10, for example, the control device 11, executes the above-mentioned action acquisition step, character image generation step, detection step, and stopping step. In addition, the various application examples exemplified for the image generation device 10 and the image generation method can also be applied to this program.

[0075] As described above, according to this embodiment, when operating a virtual character in accordance with the movement of a person, it is possible to prevent unnecessary movements from being reflected. This embodiment can be suitably applied, for example, to a case where a user's movements are tracked by a tracking device serving as the motion capture unit 20 and reflected in the movement of a virtual character. For example, this embodiment can be suitably applied to a case where a tracking device is attached to the user's hand or foot, and detection results of an acceleration sensor, an angular velocity sensor, etc. in the tracking device are transmitted to the image generating device 10 to move a virtual character in real time.

[0076] <Embodiment 2> A configuration example of the image generating device according to the second embodiment will be described with reference to Figs. 13 to 23. In the configuration example and function example of the image generating device according to the second embodiment, description of the configuration example and function example common to the configuration example and function example of the image generating device according to the first embodiment will be omitted. Fig. 13 is a schematic diagram showing one configuration example of the image generating device according to this embodiment. Fig. 14 is a functional block diagram showing details of the image generating device of Fig. 13. Fig. 15 is a schematic diagram showing a front view of an example of a person photographed by the image generating device of Fig. 14. Fig. 16 is a schematic diagram showing a top view of the example of the person of Fig. 15. Fig. 17 is a diagram showing an example of a virtual character generated by the image generating device of Fig. 14.

[0077] 13, the following description is based on the premise that image generation device 40 according to the present embodiment is an information processing device such as a smartphone or PC equipped with an imaging unit 43, but for example, the functions of image generation device 40 described below may be incorporated into a general-purpose information processing device. Alternatively, image generation device 40 described below may be configured as a dedicated device rather than being incorporated into a general-purpose information processing device.

[0078] As shown in FIG. 14, the image generating device 40 includes, for example, a control device 41, a display unit 421, an operation unit 422, and an imaging unit 43.

[0079] The control device 41, the display unit 421, and the operation unit 422 have the same configurations as the control device 11, the display unit 121, and the operation unit 122 according to the first embodiment, respectively, and therefore their explanations are omitted.

[0080] The imaging unit 43 is, for example, an imaging device built into the image generating device 40, or an imaging device connected externally by wire or wirelessly to the image generating device 40. Hereinafter, the imaging device will be simply referred to as a camera.

[0081] Furthermore, the image generating device 40 may include a communication unit (not shown). This communication unit includes a communication interface that performs communication in accordance with wireless communication standards such as Wi-Fi and the fifth generation mobile communication system. This communication unit can transmit images generated by a character image generating unit 114 (described later) to an external device via a network using this communication interface.

[0082] The main functions of the image generating device 40 may be realized, for example, as functions of the control device 41. For this reason, the control device 41 includes, for example, an imaging control unit 411, a motion detection unit 412, a detection unit 413, a character image generating unit 414, a display control unit 415, and an operation control unit 416, as described below.

[0083] The photography control unit 411 includes a video acquisition unit (not shown), and the video acquisition unit acquires a video of a person photographed by the imaging unit 43. The photography control unit 411 receives the video of the person via the imaging unit 43, and passes it to the action detection unit 412 and the detection unit 413. The photography control unit 411 may also control the transmission of the video generated by the character video generation unit 414 to an external device via the above-mentioned communication unit.

[0084] The motion detection unit 412 receives the video acquired by the shooting control unit 411, and detects human motion information, which is information indicating the motion of the person being shot, from the video. The motion detection unit 412 extracts, for example, the characteristics of the person's skeleton from the video and detects the motion of the characteristics as human motion information, but the method of detecting the human motion information is not limited. The motion of a certain characteristic is obtained by detecting a change in the position of the characteristic part. In addition, in order to make it easier to detect the human motion information, the person may have markers attached to predetermined locations such as the hands, feet, head, and abdomen. The human motion information is information that captures the motion of a person, so it may be called capture information. Note that the person to be shot can be said to be the user of the video generation device 40, so the person may be called the user. The motion detection unit 412 passes the detected human motion information to the character video generation unit 414 and the detection unit 413.

[0085] Similar to character image generation unit 114 according to embodiment 1, character image generation unit 414 generates an image in which the human movement information received from movement detection unit 412 is reflected in the movement of the virtual character. When human U is standing upright as in the examples of Fig. 15 and Fig. 16, image V4 including a virtual character standing upright is generated, such as virtual character Vc4 illustrated in Fig. 17.

[0086] The detection unit 413 detects the distance between a preset reference position and the person U. The detection unit 413 receives the image acquired by the shooting control unit 411 and detects this distance based on the image. The distance between the reference position and the person U may be detected as the distance between the reference position and a plurality of predetermined parts of the person U, or may be detected as the distance between the reference position and a predetermined part such as the head of the person U. Furthermore, the distance between the reference position and the person U may be detected as a statistical value such as the average value or median value of the distances between the reference position and a plurality of predetermined parts of the person U.

[0087] The detection unit 413 can detect the above-mentioned distance, for example, as follows. First, the distance between the image generating device 40 and each part of the person U is detected. For example, when performing calibration described later, a frame imitating a person in an upright state and a line indicating the bottom end are displayed on the display unit 421 by superimposing the captured image on the captured image. Then, the person U stands upright according to the frame and line, and calibration is performed, so that the distance between the image generating device 40 and each part of the person U at the time of performing calibration is determined. Then, the detection unit 413 detects the distance between a preset reference position and each part of the person U from the image of each part captured at the time of performing calibration, the change in size of each part during motion detection, the change in the captured image position, and the like. Note that, when the reference position is the position of the image generating device 40 as in the example described later, the detected distance is the distance between the reference position and the person U or each part of the person U. When the reference position is set to a position other than the above, the distance between the reference position and the person U or each part of the person U can be calculated based on the difference between the position of the image generating device 40 and the reference position in addition to the distance between the position of the image generating device 10 and the person U or each part of the person U.

[0088] In addition, in a configuration in which the detection unit 413 detects the distance based on the video, for example, a configuration may be adopted in which the imaging unit 43 has a function of calculating the distance to the target object, and the detection unit 413 obtains the distance calculated by the imaging unit 43. In order to calculate the distance, the imaging unit 43 may be a camera equipped with a depth sensor, a compound eye camera or two cameras, or a device that captures three-dimensional images such as LiDAR (registered trademark).

[0089] Alternatively, instead of the shooting control unit 411 passing the video to the detection unit 413, the detection unit 413 may detect the distance between the reference position and the person U based on the person movement information received from the movement detection unit 412. The movement detection unit 412 may detect the movement so that the person movement information includes information indicating the position of the person U, or may detect the person movement information so that the information indicating the position of the person U can be calculated from the person movement information.

[0090] Furthermore, when the distance between the reference position and the person U or the distance between the reference position and a predetermined part of the person U becomes a distance that satisfies a predetermined condition, the detection unit 413 detects this and notifies the character image generation unit 414. In the following, it is assumed that a notification (hereinafter, a condition match notification) is made when it is detected that the distance has become a distance that satisfies the predetermined condition. In this case, when the detection unit 413 detects that the distance no longer satisfies the predetermined condition, it is preferable that the detection unit 413 passes a notification indicating return to the character image generation unit 414. However, the detection unit 413 may determine whether or not the distance between the reference position and the person U becomes a distance that satisfies the predetermined condition, and notify the character image generation unit 414 of the determination result. In this case, the character image generation unit 414 may decide whether or not to execute a reflection stop process described later based on the determination result.

[0091] When the character image generation unit 414 receives the above-mentioned condition match notification from the detection unit 413, it executes a process of stopping the reflection of the person action information to the movement of the virtual character (hereinafter, the reflection stop process). That is, when the detection unit 413 detects that the distance between the reference position and the person U has become a distance that satisfies a predetermined condition, the character image generation unit 414 executes the reflection stop process.

[0092] An example of the detection process in the detection unit 413 and the reflection stop process by the character image generation unit 414 will be described with reference to Figs. 18 to 21. Fig. 18 is a diagram for explaining an example of the detection process executed by the image generation device 40 in Fig. 14, and is a schematic diagram showing an example of the positional relationship between the image generation device 40 and the person U as viewed from the right side. Fig. 19 is a schematic diagram showing an example of the positional relationship in Fig. 18 as viewed from above. Fig. 20 is a schematic diagram showing an example of a case where the distance of the person U to the image generation device 40 in Fig. 14 becomes less than a predetermined distance as viewed from the right side. Fig. 21 is a diagram showing an example of a virtual character generated by the image generation device 40 in Fig. 14 in the case shown in Fig. 20.

[0093] As described above, detection unit 413 detects that the distance from the reference position satisfies a predetermined condition, and passes a condition satisfaction notification to character image generation unit 414. Here, an example is given in which this reference position is the position of image generation device 40. That is, in the example described here, detection unit 413 detects the distance between the reference position and person U, using the position of image generation device 40 as the reference position. Note that, although the description will be given on the premise that the position of image generation device 40 as the reference position is a position on a horizontal plane, the height of the reference position may also be taken into consideration.

[0094] The above-mentioned predetermined condition may be, for example, a fourth condition that the person U is less than a fourth predetermined distance (hereinafter, the fourth distance) on the horizontal plane from the image generating device 40. Alternatively, the predetermined condition may be a fifth condition that the person U is equal to or greater than a fifth predetermined distance (hereinafter, the fifth distance) on the horizontal plane from the image generating device 40. Of course, the predetermined condition may be a sixth condition that the person U is less than the fourth distance or equal to or greater than the fifth distance on the horizontal plane, and various other conditions may be used without being limited to these.

[0095] Here, the fifth distance is greater than the fourth distance. The fourth distance and the fifth distance can be expressed by Th4 and Th5 in Fig. 18 and Fig. 19, respectively, and may be preset values, or may be variable values ​​as described later. Note that Fig. 18 and Fig. 19 show an example in which the image generating device 40 is mounted on a mounting base 40Ho. Of course, the predetermined condition is not limited to an example defined by only the distance on the horizontal plane, and a condition such as being outside a rectangular parallelepiped area centered at a position a predetermined distance away from a reference position in a predetermined direction and at a predetermined elevation angle can also be adopted.

[0096] The fourth distance Th4 may be preset to, for example, 1.0 m from the image generating device 40. The fifth distance Th5 may be preset to, for example, 5.0 m from the image generating device 40. The fourth distance Th4 and the fifth distance Th5 may be determined based on the position of the person U or the position of a predetermined part of the person U when the calibration is performed, which will be described later. For example, when the position of the person U when the calibration is performed is 3.0 m from the image generating device 40, the fourth distance Th4 and the fifth distance Th5 may be set to ±D (m) from that position. When D=2.0 m, the fourth distance Th4 is set to 1.0 m from the image generating device 40, and the fifth distance Th5 is set to 5.0 m from the image generating device 40. Of course, D is not limited to 2.0 m.

[0097] Although not described above, the detection unit 413 can adopt a configuration for detecting various distances, such as the distance between the reference position and the person U, the distance between the reference position and a predetermined part of the person U, or the distance between the reference position and a plurality of predetermined parts of the person U. For example, when detecting the distance between the reference position and a predetermined part, the predetermined condition may be a condition on the distance from the predetermined part of the person U to the image generating device 40. In addition, when multiple values ​​are detected as distances, such as the distance between the reference position and each predetermined part, the predetermined condition may be a condition on a group of distances from each predetermined part of the person U to the image generating device 40. For example, the predetermined condition may be a condition that at least one of the detected distance group satisfies less than the fourth distance Th4, or at least one of the detected distance group satisfies the fifth distance Th5 or more. However, hereinafter, except for some explanations, an example will be described in which the distance detected by the detection unit 413 and the distance to be determined included in the predetermined condition are the distance between the reference position and the person U and are detected as one value.

[0098] By adopting the fourth condition as the predetermined condition, the detection unit 413 and the character image generation unit 414 carry out the following process. When the person U is at a distance less than a fourth distance Th4 on the horizontal plane from the position of the image generation device 40, the detection unit 413 notifies the user that the condition is met, and the character image generation unit 414 executes a reflection stop process.

[0099] For example, as shown in Fig. 20, when person U becomes less than the fourth distance Th4 from the position of image generation device 40, detection unit 413 issues a condition satisfaction notification, and character image generation unit 414 executes the reflection stop processing. When person U becomes less than the fourth distance Th4 from image generation device 40 in this way, the image of the virtual character stops in the image at that time, for example, in the state shown by virtual character Vc5 in image V5 in Fig. 21, by this reflection stop processing. Of course, as shown in Fig. 20, when at least a part of person U becomes less than the fourth distance Th4 from the position of image generation device 40, detection unit 413 may issue a condition satisfaction notification, and character image generation unit 414 may execute the reflection stop processing.

[0100] In fact, when the person U becomes less than the fourth distance Th4 from the image generating device 40, this occurs when the person U performs the following action from the image generating state. That is, a situation occurs in which the person U becomes less than the fourth distance Th4 by performing an action of operating the operation unit of the image generating device 40, for example, to stop the generation, storage, distribution, etc. of the image or to change the setting. However, in this situation, since the character image generating unit 414 executes the reflection stop processing, such an action of the person U does not need to be reflected in the movement of the virtual character. For example, as described above, the image V5 generated by this reflection stop processing is an image of the virtual character Vc5 that reflects the movement of the person U just before the person U becomes less than the fourth distance Th4 from the image generating device 40. Therefore, according to this embodiment, when operating the virtual character according to the movement of the person U, it is possible to prevent unnecessary actions from being reflected.

[0101] This effect will be described by taking a specific usage scene as an example. For example, when the person U uses the image generating device 40 to transmit the virtual character's movements to an external device, such as by live streaming the movements, the movements of the person U during the period when the person U is less than the fourth distance Th4 from the image generating device 40 are not transmitted. Therefore, even in such a case, the image generating device 40 can transmit appropriate movements of the virtual character. Also, for example, when the person U uses the image generating device 40 to store the movements of the virtual character, the movements of the person U during the period when the person U is less than the fourth distance Th4 from the image generating device 40 are not stored. Therefore, even in such a case, the image generating device 40 can reduce the number of editing steps after storage.

[0102] Furthermore, the reflection stop processing may be accompanied by the following correction processing. That is, when the reflection of human action information on the movement of a virtual character is stopped by the reflection stop processing, the character image generating unit 414 may perform a correction to transition the virtual character from the action at the time of the stop to a predetermined action. This transition may be an instantaneous switch from the virtual character before the correction to the virtual character after the correction, or a gradual change from the virtual character before the correction to the virtual character after the correction.

[0103] The image generated by this correction process transitions from image V5 in a stopped state to image V6 including virtual character Vc6, as shown in Fig. 21. Image V6 is an image in which virtual character Vc5 reflecting person movement information immediately before person U becomes less than a fourth distance Th4 from image generating device 40 in image V5 is stopped, and virtual character Vc5 is corrected to virtual character Vc6.

[0104] Here, the virtual character Vc6 is exemplified as a virtual character showing a form during calibration, which will be described later, and such modifications can be read out by storing information on the virtual character during calibration. However, the virtual character Vc6 does not have to be a virtual character showing a form during calibration, and may be, for example, a predetermined virtual character. In that case, a character indicating that an operation is being performed may be added to the virtual character.

[0105] In this embodiment, by employing such a correction process, when operating a virtual character in accordance with the movement of person U, it is possible to prevent unnecessary movements from being reflected as they are.

[0106] This effect will be described by taking a specific usage scene as an example. For example, when the person U uses the image generating device 40 to transmit the virtual character's movements to an external device, such as by live streaming the movements, the movements during the period when the person U is less than the fourth distance Th4 from the image generating device 40 are corrected and transmitted. Even in such a case, the image generating device 40 can transmit appropriate virtual character movements. Also, for example, when the person U uses the image generating device 40 to store the virtual character's movements, the movements during the period when the person U is less than the fourth distance Th4 from the image generating device 40 are corrected and stored. Even in such a case, the image generating device 40 can reduce the number of editing steps after storage.

[0107] Next, a case where the fifth or sixth condition is adopted as the predetermined condition will be described. By adopting the fifth condition as the predetermined condition, the detection unit 413 and the character image generation unit 414 carry out the following process. When the person U is at a distance equal to or greater than the fifth distance Th5 on the horizontal plane from the position of the image generation device 40, the detection unit 413 notifies the user that the condition is met, and the character image generation unit 414 executes the reflection stop process. In this case as well, the correction process described above may be adopted.

[0108] Furthermore, by adopting the sixth condition as the predetermined condition, the detection unit 413 and the character image generation unit 414 carry out the following process. When the person U is at a distance less than the fourth distance Th4 or equal to or greater than the fifth distance Th5 on the horizontal plane from the position of the image generation device 40, the detection unit 413 notifies the user that the condition is met, and the character image generation unit 414 executes the reflection stop process. In this case as well, the correction process described above may be adopted.

[0109] Next, an example will be given in which the fourth distance Th4 and the fifth distance Th5 are variable values. The fourth distance Th4 and the fifth distance Th5 may be changed according to attribute information indicating the attributes of the person U, for example. This attribute information may be information indicating whether the person U is a main user or a guest user of the image generating device 40. For this information, for example, information such as login information to the image generating device 40 may be used, or a main / guest designation operation or a user name designation operation may be received at the start of use.

[0110] The main user is as likely to perform an operation to stop image generation as the guest user, but is more likely to change some settings of the image generation device 40. Therefore, if the fourth distance Th4 is set larger for the main user than for the guest user, the reflection stop process will not be performed even if the guest user unintentionally approaches the image generation device 40 without intending to operate it. Also, the guest user is more likely to move around without being aware of the execution of the reflection stop process than the main user. Therefore, if the fifth distance Th5 is set larger for the guest user than for the main user, the reflection stop process will not be performed even if the guest user unintentionally moves far away from the image generation device 40.

[0111] Furthermore, the fourth distance Th4 and the fifth distance Th5 may be changed to smaller and larger values, respectively, depending on the intensity of the movement of the person U. By making such changes, it is possible to prevent the reflection stop process from being executed even if the person U unintentionally moves outside the range of the distances Th4 or more and less than Th5 due to intense movement.

[0112] Another example of the reference position will be described with reference to Fig. 22 and Fig. 23. Fig. 22 is a diagram for explaining another example of the detection process executed by the image generating device 40 of Fig. 14, and is a schematic diagram showing an example of the positional relationship between the image generating device 40 and person U as viewed from the right side. Fig. 23 is a schematic diagram showing the example of the positional relationship of Fig. 22 as viewed from above.

[0113] The reference position may be the position of the person U (hereinafter, the calibration position) when calibrating the process of acquiring human motion information in the image generating device 40. That is, the detection unit 413 may detect the distance between the reference position and the person U by using the calibration position as the reference position.

[0114] The calibration position as the reference position is the position of the person U when calibration is performed, and this position may be defined as follows. That is, the position of the person U when calibration is performed may be, for example, the position of the center of gravity or the center of the positions of multiple predetermined parts of the person U when calibration is performed, and the detection unit 413 may detect the distance from a reference position common to the multiple predetermined parts of the person U. Alternatively, the position of the person U when calibration is performed may be the positions of multiple predetermined parts of the person U when calibration is performed, and the detection unit 413 may detect the distance from a reference position corresponding to each part of the person U. Alternatively, the position of the person U when calibration is performed may be the position of a predetermined part, such as the head, when calibration is performed, and the detection unit 413 may detect the distance from a reference position common to the multiple predetermined parts of the person U.

[0115] In the calibration, for example, the person U to be photographed takes an upright state within the photographing range of the imaging unit 43 at the start of the calibration, as illustrated in FIG. 15 and FIG. 16. Then, as described above, a frame imitating a person in an upright state and a line indicating the bottom end are displayed on the display unit 421 by superimposing them on the photographed image, and the person U stands upright according to the frame and line, and the calibration is performed. This determines the position of the person U or each predetermined part of the person U relative to the image generating device 40 when the calibration is performed. The position of the person U or the position of each predetermined part of the person U acquired in this state is registered as the position of the virtual character standing upright in the center of the image. For example, for both hands and both feet of the person U, the position represented by the distance Ca shown in FIG. 22 in the depth direction of the horizontal plane is registered as the position of the virtual character standing upright in the center of the image, and the same is registered in the width direction and vertical direction of the horizontal plane, not shown. The same is true for other parts of the person U. In this way, a reference position may be set for each specified part of person U, but may also be set for only one specified part, in which case, when detecting the distance by detection unit 413, the distance to each specified part may be detected with respect to this one reference position.

[0116] In this way, the image corresponding to the position registered when calibration is performed becomes, for example, image V4 including virtual character Vc4. After calibration is completed, the movement of person U is reflected in the movement of the virtual character with the calibration position as the reference position. Note that the movement detection unit 412 detects the current position of each predetermined part for this reflection, and the calibration position that is the reference for this current position may be the position of each predetermined part at the time of calibration.

[0117] As described above, the detection unit 413 detects that the distance from the reference position satisfies a predetermined condition, and passes a condition satisfaction notification to the character image generation unit 414. In the example described here, the detection unit 413 detects the distance between the reference position and the person U, using the calibration position as the reference position.

[0118] In this case, the above-mentioned predetermined condition can be various conditions such as a condition of being equal to or greater than a sixth predetermined distance (hereinafter, the sixth distance) on a horizontal plane, as exemplified in Fig. 22 and Fig. 23. Note that Fig. 22 and Fig. 23 show an example in which the position of person U at the time of calibration is set as the reference position. Of course, the predetermined condition is not limited to an example defined only by the distance on the horizontal plane, and a condition such as being outside a rectangular parallelepiped area centered on the calibration position can also be used. Also, although the calibration position as the reference position has been described on the assumption that it is a position on the horizontal plane, the height of the reference position may also be taken into consideration.

[0119] The sixth distance Th6 may be preset as, for example, 2.0 m. By adopting a condition of the sixth distance Th6 or more as the predetermined condition, the detection unit 413 and the character image generation unit 414 perform the following process. When the person U is at a distance of the sixth distance Th6 or more on the horizontal plane from the calibration position, the detection unit 413 notifies the condition match, and the character image generation unit 414 executes the reflection stop process. For example, when the position of the person U is at a distance of the sixth distance Th6 or more on the horizontal plane from the calibration position, the image of the virtual character is stopped by this reflection stop process in the image at that time, for example, in the state shown by the virtual character Vc5 in the image V5 in FIG. 21. As a result, even when the calibration position is adopted as the reference position, the same effect as when the position of the image generation device 40 is adopted as the reference position is achieved. Moreover, even when the calibration position is adopted as the reference position, the above-mentioned correction process may be adopted.

[0120] Next, a video generation method according to this embodiment (hereinafter, this method) will be described. This method is executed by a video generation device such as video generation device 40, and will be described below as a method executed by video generation device 40. Fig. 24 is a flow diagram for explaining an example of the video generation method executed by video generation device 40 according to this embodiment.

[0121] First, with the person U standing upright at the calibration position, the image generating device 40 starts capturing images with the imaging unit 43 (step S41) and executes the calibration (step S42). The position where the person U stands upright when the calibration is executed may be determined in advance, but it may be simply determined as a rough position, facing the image generating device 40 and at a predetermined distance from the image generating device 40.

[0122] Next, the image generating device 40 sets a range for the predetermined condition based on the result of this calibration (step S43). As a result, the predetermined condition is set to a range with the position of the image generating device 40 as the reference position or a range with the calibration position as the reference position.

[0123] Next, the image generating device 40 starts capturing to reflect the movement of the person U in the virtual character (step S44). This capturing includes a process in which the shooting control unit 411 causes the imaging unit 43 to shoot and acquire an image, the movement detection unit 412 sequentially acquires person movement information indicating the movement of the person U, and the character image generating unit 414 generates an image in which the person movement information is reflected in the movement of the virtual character. The image in which the captured movement of the person U is reflected in the virtual character can be displayed on the display unit 421, stored in a built-in storage device, or distributed to an external device. The execution of one or more of the display, storage, and distribution of the image can be performed according to the setting made by the person U in the image generating device 40, for example.

[0124] After starting capture in this manner, image generation device 40 generates an image, and also causes detection unit 413 to detect the distance of person U from the reference position and detect whether the distance satisfies a predetermined condition (step S45).

[0125] If NO in step S45, that is, if the distance does not satisfy the predetermined condition, the image generating device 40 judges whether the capture has ended (step S50), and ends the process if it has ended, and continues the detection of step S45 if it has not ended. The end of the capture may be determined when an end operation is performed by the person U or the like from the operation unit 422. Note that, during the distribution of the image, the capture may be ended, for example, when communication with an external device, for example communication with a network, by the above-mentioned communication unit is interrupted.

[0126] If the answer is YES in step S45, that is, if the distance satisfies the predetermined condition, then character image generation unit 414 instructs motion detection unit 412 or motion detection unit 412 and photography control unit 411 to suspend capture (step S46). Note that, although reflection is halted by suspending capture in step S46, for example, acquisition of human motion information by motion detection unit 412 may continue without suspension, and only reflection of the human motion information to the virtual character may be suspended. Next, character image generation unit 414 generates an image in which the virtual character has been shifted to a reference position (step S47).

[0127] Even in this state, detection by detection unit 413 continues, and detection unit 413 detects whether the distance has become one that does not satisfy the predetermined condition, that is, whether the distance has returned to one at which the reflection stop process is not performed (step S48).

[0128] If the answer is YES in step S48, image generation device 40 resumes acquisition of human movement information in movement detection unit 412, and also resumes processing for generating an image in which the human movement information is reflected in the movement of the virtual character in character image generation unit 414 (step S49). After step S49, the process returns to step S45, and detection is performed as to whether the distance has reached a predetermined condition. On the other hand, if the answer is NO in step S48, image generation device 40 determines whether capture has ended as in step S50 (step S51), and ends the processing if the answer is YES in step S51, and continues the determination of step S48 if the answer is NO in step S51.

[0129] As illustrated in FIG. 24, in this method, an image generation device, exemplified by image generation device 40, executes an image acquisition step, a motion detection step, a character image generation step, a detection step, and a stopping step, as described below.

[0130] The video acquisition step, as exemplified in step S44, acquires a video of the person U. The movement detection step, as exemplified in step S44, detects person movement information, which is information indicating the movement of the person U being photographed, from the video acquired in the video acquisition step. The character video generation step, as exemplified in step S44, generates a video in which the person movement information detected in the movement detection step is reflected in the movement of a virtual character. The detection step, as exemplified in step S45, detects the distance between a preset reference position and the person U.

[0131] The stopping step is exemplified by the process of step S46 in the case where step S45 is YES. That is, the stopping step executes a reflection stopping process when it is detected in the detection step that the distance between the reference position and the person U has become a distance that satisfies a predetermined condition. Also, as exemplified in step S47, the stopping step may further include a correction step of transitioning the virtual character from the action at the time of stopping to a predetermined action after stopping.

[0132] Furthermore, the character image generating unit 414 may execute a reflection stop process when it is detected that at least one of a plurality of predetermined parts of the person U is at a distance from the reference position that satisfies a predetermined condition. The predetermined part may be, for example, a part that often determines the horizontal range of the body of the person U, such as the left hand, right hand, left foot, or right foot. However, the predetermined part may be another part, such as the head.

[0133] In this manner, in the stopping step, if the detection unit 413 detects that the distance between the reference position and the hand or foot of person U has become a distance that satisfies a predetermined condition, the reflection stopping process may be executed.

[0134] In this embodiment, the program described above as being stored in control device 41 may be a program such as the following. That is, the program according to this embodiment is a program in which a computer that operates the image generation device exemplified by image generation device 40, for example, control device 41, executes the image acquisition step, motion detection step, character image generation step, detection step, and stopping step described above. In addition, the various application examples exemplified for image generation device 40 and the image generation method can also be applied to this program.

[0135] As described above, according to this embodiment, when a virtual character is operated in accordance with the movement of a person, unnecessary movements can be prevented from being reflected. This embodiment can be suitably applied, for example, to a case where the movement of a person is tracked by photographing and reflected in the movement of a virtual character. For example, this embodiment can be suitably applied to a case where a virtual character is moved in real time by performing motion capture based on a video of a person.

[0136] <Other embodiments, etc.> In the first and second embodiments, the functions of each part of the image generating device have been described, but it is sufficient if the image generating device can realize these functions.

[0137] Furthermore, the motion capture technology is not limited to those described in the first and second embodiments, and various other types of technology may be applied. Although markers were mentioned in the second embodiment, for example, optical motion capture technology may be used to measure reflective markers or LED (Light Emitting Diode) markers with multiple dedicated cameras and transmit the measured data to an image generating device. This technology corresponds to a technology in which, for example, markers are placed at the positions of the motion capture units 20 in the first embodiment, and the markers are photographed by cameras (however, two or more cameras) as in the second embodiment, to obtain information indicating the movement of a person.

[0138] Also, for example, the movement of the target person may be determined based on both the user movement information described in the first embodiment and the person movement information described in the second embodiment, and the determined person movement may be reflected in the movement of the virtual character by the character image generating unit. The movement of the person may be determined in various ways, for example, by using the user movement information as a reference and correcting the user movement information with the person movement information.

[0139] Furthermore, not limited to the image generating devices 10 and 40 described in the first and second embodiments, if the image generating device can obtain information indicating the movement of a person and the distance between a preset reference position and the person, a similar reflection stop process, that is, the following reflection stop process, can be performed. That is, this reflection stop process is a process of stopping the reflection of information indicating the movement of a person to the movement of a virtual character when it is detected that the distance between the reference position and the person has become a distance that satisfies a predetermined condition. That is, the image generating device may be configured to include an information acquisition unit that acquires information indicating the movement of a person instead of the motion acquisition unit, or instead of the image acquisition unit and the motion detection unit. Also, the image generating method may include an information acquisition step that acquires information indicating the movement of a person instead of the motion acquisition step, or instead of the image acquisition step and the motion detection step.

[0140] A part or all of the above-described embodiments can be described as, but is not limited to, the following supplementary notes. (Appendix 1) a motion acquisition unit that acquires information indicating a user's motion from a plurality of motion capture units attached to the user; a character image generating unit that generates an image in which the information indicating the user's movement acquired by the action acquiring unit is reflected in a movement of a virtual character; A detection unit that detects distances between a preset reference position and the plurality of motion capture units; Equipped with when it is detected that a distance that satisfies a predetermined condition is reached from the reference position for at least one of the plurality of motion capture units detected by the detection unit, the character image generation unit stops reflecting information indicating the user's movement on the movement of the virtual character. Image generation device. (Appendix 2) an image acquisition unit that acquires an image of a person captured by an imaging device; a motion detection unit that detects information indicating a motion of a person being photographed from the video acquired by the video acquisition unit; a character image generating unit that generates an image in which information indicating the movement of the person detected by the movement detecting unit is reflected in a movement of a virtual character; A detection unit that detects a distance between a preset reference position and the person; Equipped with when the detection unit detects that the distance between the reference position and the person has become a distance that satisfies a predetermined condition, the character image generation unit stops reflecting information indicating a movement of the person in a movement of a virtual character. Image generation device. (Appendix 3) An information acquisition step of acquiring information indicating a person's movement; a character image generating step of generating an image in which the information indicating the movement of the person is reflected in the movement of a virtual character; a detection step of detecting a distance between a preset reference position and the person; a stopping step of stopping reflection of information indicating a movement of the person in a movement of a virtual character when it is detected in the detecting step that the distance between the reference position and the person has become a distance that satisfies a predetermined condition; The image generating method is executed by an image generating device. (Appendix 4) An information acquisition step of acquiring information indicating a person's movement; a character image generating step of generating an image in which the information indicating the movement of the person is reflected in the movement of a virtual character; a detection step of detecting a distance between a preset reference position and the person; a stopping step of stopping reflection of information indicating a movement of the person in a movement of a virtual character when it is detected in the detecting step that the distance between the reference position and the person has become a distance that satisfies a predetermined condition; A program executed by a computer to operate an image generating device. (Appendix 5) an information acquisition unit that acquires information indicating a person's movement; a character image generating unit that generates an image in which the information indicating the movement of the person is reflected in the movement of a virtual character; A detection unit that detects a distance between a preset reference position and the person; Equipped with and when the detection unit detects that the distance between the reference position and the person has become a distance that satisfies a predetermined condition, the character image generation unit stops reflecting information indicating a movement of the person in a movement of a virtual character. Image generation device.

[0141] Although the invention made by the inventor has been specifically described based on the embodiment, it goes without saying that the present invention is not limited to the above embodiment and can be modified in various ways without departing from the gist of the invention. Furthermore, the present invention may be implemented by appropriately combining the respective embodiments. [Explanation of symbols]

[0142] 1. Video Creation System 10, 40 Image generation device 10Ho, 40Ho mounting stand 11, 21, 41 Control device 12, 42 Touch panel 13, 23 Communications Department 20 Motion capture unit 22 Sensor section 43 Imaging unit 111, 211 Communication control unit 112 Operation acquisition section 113, 413 Detector 114, 414 Character image generation unit 115, 415 Display control unit 116, 416 Operation control section 121, 421 Display section 122, 422 Operation section 212 Sensor control unit 411 Shooting control section 412 Motion detection unit 1000 devices 1001 Processor 1002 Memory 1003 Interface

Claims

1. An information acquisition step of acquiring information indicating a person's movement; a character image generating step of generating an image in which the information indicating the movement of the person acquired in the information acquiring step is reflected in the movement of a virtual character; a detection step of detecting a distance between a preset reference position and the person; a stopping step of stopping reflection of information indicating the movement of the person in the movement of a virtual character when it is detected in the detecting step that the distance between the reference position and the person has become a distance that satisfies a predetermined condition; The image generation method is executed by an image generation device.

2. The information acquisition step acquires information indicating the movements of the person from a plurality of motion capture units attached to the person, In the detecting step, a distance between the reference position and the person is detected by detecting a distance between the reference position and the plurality of motion capture units; the character image generating step generates an image in which information indicating the movement of the person obtained by a plurality of motion capture units attached to the person is reflected in the movement of the virtual character; In the stopping step, when it is detected that a distance from the reference position of at least one of the plurality of motion capture units satisfies a predetermined condition, reflection of information indicating the user's movement on the movement of the virtual character is stopped. The image generation method according to claim 1 .

3. In the stopping step, when it is detected in the detecting step that a motion capture unit attached to a hand or a foot of the user or a position close to the hand or foot of the user among the plurality of motion capture units is at a distance from the reference position that satisfies a predetermined condition, reflection of information indicating the user's movement in the movement of the virtual character is stopped. The image generation method according to claim 2 .

4. In the information acquisition step, information indicating the movement of the person being photographed is detected from the video of the person, In the detecting step, a distance between a preset reference position and the person is detected from an image of the person. The image generation method according to claim 1 .

5. In the stopping step, when it is detected in the detecting step that the distance between the reference position and a hand or a foot of the person has become a distance that satisfies a predetermined condition, reflection of information indicating the movement of the person in the movement of the virtual character is stopped. The image generation method according to claim 2 .

6. An information acquisition step of acquiring information indicating a person's movement; a character image generating step of generating an image in which the information indicating the movement of the person acquired in the information acquiring step is reflected in the movement of a virtual character; a detection step of detecting a distance between a preset reference position and the person; a stopping step of stopping reflection of information indicating the movement of the person in the movement of a virtual character when it is detected in the detecting step that the distance between the reference position and the person has become a distance that satisfies a predetermined condition; A program executed by a computer that operates an image generation device.

Images