Image generation method and image generation program

Abstract

The present invention aims to provide an image generation method and an image generation program that can generate realistic avatar images. [Solution] In a configuration in which an external input image captured from external video data by a capture device 112 is used as a background image, and an avatar image created by an avatar image generation program is placed in front of the external input image, a composite image is created and output, and the color of the avatar image placed in front of the external input image is changed based on the external input image.

Description

[Technical Field]

[0001] The present invention relates to an image generation method and an image generation program that can generate an avatar image according to a detected movement and output the generated avatar image by combining it with a specific image. [Background technology]

[0002] Conventionally, a technology has been proposed that, for example, uses face tracking to capture a face photographed with a smartphone, allowing the facial expression of the captured face to be reflected in the expression of an avatar, such as a character created using CG (see, for example, Patent Document 1).

[0003] Furthermore, a technique has been proposed for generating and outputting an image in which a specific image is superimposed on the back of an avatar image (see, for example, Patent Document 2). [Prior art documents] [Patent documents]

[0004] [Patent Document 1] Japanese Patent Application Publication No. 2019-186797 [Patent Document 2] Patent No. 731455 Summary of the Invention [Problem to be solved by the invention]

[0005] However, as in Patent Document 1, simply generating an image by combining another specific image behind an avatar image has the problem that the relationship between the avatar image and the specific image is weak, resulting in an image that lacks realism.

[0006] An object of the present invention is to provide an image generation method and an image generation program that can generate realistic avatar images. [Means for solving the problem]

[0007] The image generation method of means 1 is as follows: a movement detection step of detecting a movement of a performer; an avatar image generating step of generating an avatar image according to the detected movement; a specific image input step of inputting a specific image; a composite image generating step of generating a composite image in which an avatar image is placed in front of the input specific image; an image output step of outputting the composite image; An image generation method comprising: The method further includes a color change step of changing the color of the avatar image based on the specific image. It is characterized by the following. According to this feature, the color of the avatar placed in front of the specific image is changed based on the specific image, so that a more realistic image can be generated by combining the specific image and the avatar.

[0008] The image generation method of means 2 is the image generation method according to means 1, The color changing step multiplies the color of the specific image by the avatar image. It is characterized by the following. According to this feature, the color of the specific image can be reflected in the avatar image.

[0009] The image generation method of means 3 is the image generation method according to means 2, The color changing step multiplies the avatar image by an average color of the specific image. It is characterized by the following. According to this feature, the average color of the specific image can be reflected in the color of the avatar image.

[0010] The image generating method of means 4 is the image generating method according to means 3, The color changing step multiplies the avatar image by an average color of a partial area of ​​the specific image. It is characterized by the following. According to this feature, the color of the avatar image reflects the average color of a part of the specific image, rather than the entire image. Therefore, for example, by reflecting the average color of the central or characteristic area of ​​the specific image in the color of the avatar image, the relevance to the specific image can be increased.

[0011] The image generating method of means 5 is the image generating method according to means 3 or 4, The color changing step multiplies the avatar image by an average value of colors of the specific image that are equal to or greater than a predetermined brightness. It is characterized by the following. According to this feature, the average value of the colors of the specific image that are equal to or greater than a predetermined brightness is reflected in the color of the avatar, so the avatar will not be too dark.

[0012] The image generating method of means 6 is the image generating method according to any one of means 2 to 5, In the color changing step, the color to be multiplied is changed for each of the plurality of regions according to the color scheme of the specific image. It is characterized by the following. According to this feature, the color of each of the multiple regions changes depending on the color scheme of the specific image, so that a more realistic image can be generated by combining the specific image and the avatar image.

[0013] The image generating method of means 7 is the image generating method according to means 2, In the color change step, the specific image is blurred and multiplied by the avatar image. It is characterized by the following. According to this feature, the blurred specific image is multiplied onto the avatar image, so that the specific image is not projected clearly onto the avatar image, and the color of the specific image can be reflected naturally in the avatar image.

[0014] The image generating method of means 8 is the image generating method according to any one of means 2 to 7, The color changing step changes the brightness of the color to be multiplied in the central region and the outer edge region of the avatar image. It is characterized by the following. According to this feature, by using different brightness levels for the color multiplied in the central area and the outer edge area of ​​the avatar, the color of the specific image can be reflected three-dimensionally in the avatar.

[0015] The image generating method of means 9 is the image generating method according to any one of means 1 to 8, the specific image input in the specific image input step is a moving image, further comprising a motion amount detection step of detecting a motion amount of the specific image; In the color changing step, the color of the avatar image is changed based on the amount of movement of the specific image detected in the amount of movement detection step. It is characterized by the following. According to this feature, the color of the avatar image placed in front of the specific image is changed based on the amount of movement of the specific image, so that a more realistic image can be generated by combining the specific image and the avatar image.

[0016] The image generating method of means 10 is the image generating method according to any one of means 1 to 9, the specific image input in the specific image input step is a moving image, the composite image is an image that constitutes a moving image, In the color changing step, a color of the avatar image is changed based on the specific image for each frame of the video, and in the composite image generating step, the composite image is generated using the specific image and the color-changed avatar image. It is characterized by the following. According to this feature, it is possible to generate a moving image of an avatar image in which a changing specific image is reflected in each frame.

[0017] The image generating program of means 11 includes: a movement detection step of detecting a movement of a performer; an avatar image generating step of generating an avatar image according to the detected movement; a specific image input step of inputting a specific image; a composite image generating step of generating a composite image in which an avatar image is placed in front of the input specific image; an image output step of outputting the composite image; In an image generation program that causes a computer to execute A color change step of changing the color of the avatar image based on the specific image is further performed. It is characterized by the following. According to this feature, the color of the avatar placed in front of the specific image is changed based on the specific image, so that a more realistic image can be generated by combining the specific image and the avatar.

[0018] The present invention may have only the invention-specifying matters set forth in the claims of the present invention, or may have the invention-specifying matters set forth in the claims of the present invention as well as configurations other than the invention-specifying matters. [Brief explanation of the drawings]

[0019] [Figure 1] FIG. 2 is a block diagram showing an example of the configuration of a computer terminal used in the first and second embodiments of the present invention. [Figure 2] FIG. 2 is a diagram showing the configuration of a program installed in a computer terminal used in the first embodiment of the present invention. [Figure 3] FIG. 2 is a diagram showing the relationship between programs installed in a computer terminal in the first embodiment of the present invention. [Figure 4] FIG. 2 is a diagram showing the relationship between programs installed in a computer terminal in the first embodiment of the present invention. [Figure 5] FIG. 10 is a diagram illustrating a process in which the computer terminal combines an external input image with an avatar image and outputs the combined image in the first embodiment of the present invention. [Figure 6] 10 is a flowchart showing the control content of a color correction program executed by a computer terminal in the first embodiment of the present invention. [Figure 7]10A to 10C are diagrams illustrating a specific example of color correction of an avatar image by a computer terminal according to the first embodiment of the present invention. [Figure 8] 10A to 10C are diagrams illustrating a specific example of color correction of an avatar image by a computer terminal according to the first embodiment of the present invention. [Figure 9] 10A and 10B illustrate a modified example of color correction of an avatar image by a computer terminal according to the first embodiment of the present invention. [Figure 10] FIG. 10 is a diagram showing the configuration of a program installed in a computer terminal used in Example 2 of the present invention. [Figure 11] FIG. 10 is a diagram showing the relationship between programs installed in a computer terminal in Example 2 of the present invention. [Figure 12] FIG. 10 is a diagram showing the relationship between programs installed in a computer terminal in Example 2 of the present invention. [Figure 13] FIG. 10 is a diagram illustrating a process in which a computer terminal combines an external input image with an avatar image and outputs the combined image in accordance with the second embodiment of the present invention. [Figure 14] 10 is a flowchart showing the control content of a 3D model color correction program executed by a computer terminal in Example 2 of the present invention. [Figure 15] 10A and 10B are diagrams illustrating a specific example of color correction of an avatar image by a computer terminal according to the second embodiment of the present invention. DETAILED DESCRIPTION OF THE INVENTION

[0020] [Form 1] The image generation method of form 1-1 is A motion detection step (such as a face tracking program or a motion tracking program) for detecting the movements of the performer; an avatar image generating step (avatar image generating program) for generating an avatar image according to the detected movement; a specific image input step (capturing an external input image) for inputting a specific image (external input image); a composite image generating step (image synthesis program) for generating a composite image in which an avatar image is placed in front of the input specific image (external input image); an image output step of outputting the composite image (output of composite image); An image generation method comprising: The image processing method further includes a color correction step (color correction program) for changing the color of the avatar image based on the specific image (external input image). It is characterized by the following. According to this feature, the color of the avatar placed in front of the specific image is changed based on the specific image, so that a more realistic image can be generated by combining the specific image and the avatar.

[0021] The image generation method of embodiment 1-2 is the image generation method according to embodiment 1-1, In the color change step (color correction program), the color of the specific image (external input image) is multiplied by the avatar image. It is characterized by the following. According to this feature, the color of the specific image can be reflected in the avatar image.

[0022] The image generation method of embodiment 1-3 is the image generation method according to embodiment 1-2, In the color change step (color correction program), the average value of the color of the specific image (external input image) is multiplied by the avatar image. It is characterized by the following. According to this feature, the average color of the specific image can be reflected in the color of the avatar image.

[0023] The image generation method of embodiment 1-4 is the image generation method according to embodiment 1-3, In the color change step (color correction program), the avatar image is multiplied by an average color of a partial area of ​​the specific image (external input image). It is characterized by the following. According to this feature, the color of the avatar image reflects the average color of a part of the specific image, rather than the entire image. Therefore, for example, by reflecting the average color of the central or characteristic area of ​​the specific image in the color of the avatar image, the relevance to the specific image can be increased.

[0024] The image generation method of embodiment 1-5 is the image generation method according to embodiment 1-3 or embodiment 1-4, In the color change step (color correction program), the avatar image is multiplied by an average value of colors of the specific image (external input image) that are equal to or higher than a predetermined brightness. It is characterized by the following. According to this feature, the average value of the colors of the specific image that are equal to or greater than a predetermined brightness is reflected in the color of the avatar, so the avatar will not be too dark.

[0025] The image generation method of embodiment 1-6 is the image generation method according to any one of embodiment 1-2 to embodiment 1-5, In the color change step (color correction program), the color to be multiplied for each of a plurality of regions is changed according to the color scheme of the specific image (external input image). It is characterized by the following. According to this feature, the color of each of the multiple regions changes depending on the color scheme of the specific image, so that a more realistic image can be generated by combining the specific image and the avatar image.

[0026] The image generation method of embodiment 1-7 is the image generation method according to embodiment 1-2, In the color change step (color correction program), the specific image (external input image) is blurred and multiplied by the avatar image. It is characterized by the following. According to this feature, the blurred specific image is multiplied onto the avatar image, so that the specific image is not projected clearly onto the avatar image, and the color of the specific image can be reflected naturally in the avatar image.

[0027] The image generation method of embodiment 1-8 is the image generation method according to any one of embodiment 1-2 to embodiment 1-7, The color change step (color correction program) changes the brightness of the color to be multiplied in the central area and the outer edge area of ​​the avatar image. It is characterized by the following. According to this feature, by using different brightness levels for the color multiplied in the central area and the outer edge area of ​​the avatar, the color of the specific image can be reflected three-dimensionally in the avatar.

[0028] The image generation method of embodiment 1-9 is the image generation method according to any one of embodiment 1-1 to embodiment 1-8, the specific image (external input image) input in the specific image input step (capturing the external input image) is a moving image, Further comprising a motion amount detection step (image analysis program) for detecting a motion amount of the specific image (external input image), In the color change step (color correction program), the color of the avatar image is changed based on the amount of movement of the specific image (external input image) detected in the movement amount detection step (image analysis program). It is characterized by the following. According to this feature, the color of the avatar image placed in front of the specific image is changed based on the amount of movement of the specific image, so that a more realistic image can be generated by combining the specific image and the avatar image.

[0029] The image generation method of embodiment 1-10 is the image generation method according to any one of embodiment 1-1 to embodiment 1-9, the composite image is an image that constitutes a moving image, In the composite image generating step (image synthesis program), a color of the avatar image is changed for each frame of the video based on the specific image (external input image), and the composite image is generated using the specific image (external input image) and the color-changed avatar image. It is characterized by the following. According to this feature, a moving image of an avatar can be generated in which a specific image is reflected in each frame.

[0030] The image generation program of form 1-11 is A motion detection step (such as a face tracking program or a motion tracking program) for detecting the movements of the performer; an avatar image generating step (avatar image generating program) for generating an avatar image according to the detected movement; a specific image input step (capturing an external input image) for inputting a specific image (external input image); a composite image generating step (image synthesis program) for generating a composite image in which an avatar image is placed in front of the input specific image (external input image); an image output step of outputting the composite image (output of composite image); In an image generation program that causes a computer to execute A color change step (color correction program) for changing the color of the avatar image based on the specific image (external input image) is further executed. It is characterized by the following. According to this feature, the color of the avatar placed in front of the specific image is changed based on the specific image, so that a more realistic image can be generated by combining the specific image and the avatar.

[0031] [Form 2] The image generation method of form 2-1 is as follows: A motion detection step (such as a face tracking program or a motion tracking program) for detecting the movements of the performer; a 3D model generation step (avatar 3D model generation program) for generating a 3D model of an avatar (avatar 3D model) according to the detected movement; a 3D model placement step (object placement process) of placing the generated 3D model (avatar 3D model) in a 3D space (virtual 3D space); an avatar image generation step (avatar image generation process) of generating an avatar image from the 3D model (avatar 3D model) placed in the 3D space (virtual 3D space); a specific image input step (capturing an external input image) for inputting a specific image (external input image); a composite image generating step (image synthesis program) for generating a composite image in which the avatar image is placed in front of the input specific image (external input image); an image output step of outputting the composite image (output of composite image); An image generation method comprising: an irradiation step (irradiation light specification process, irradiation light arrangement process) of irradiating the 3D model (avatar 3D model) arranged in the 3D space (virtual 3D space) with virtual irradiation light based on the specific image (external input image); The avatar image generating step (avatar image generating process) generates the avatar image from the 3D model (avatar 3D model) illuminated with the virtual illumination light in the illumination step (illumination light specifying process, illumination light positioning process). It is characterized by the following. According to this feature, an avatar image generated from a 3D model illuminated with virtual light based on the specific image is used as the avatar image placed in front of the specific image, and the color of the avatar image is changed based on the specific image, so that a realistic image can be generated by combining the specific image and the avatar image.

[0032] The image generation method of embodiment 2-2 is the image generation method according to embodiment 2-1, In the irradiation step (irradiation light specification process, irradiation light arrangement process), irradiation light of the color of the specific image (external input image) is irradiated. It is characterized by the following. According to this feature, the color of the specific image can be reflected in the avatar image.

[0033] The image generation method of aspect 2-3 is the image generation method according to aspect 2-2, In the irradiation step (irradiation light specification process, irradiation light arrangement process), irradiation light of a color that is the average value of the colors of the specific image (external input image) is irradiated. It is characterized by the following. According to this feature, the average color of the specific image can be reflected in the color of the avatar image.

[0034] The image generation method of aspect 2-4 is the image generation method according to aspect 2-3, In the irradiation step (irradiation light specification process, irradiation light arrangement process), irradiation light of a color that is the average color of a partial area of ​​the specific image (external input image) is irradiated. It is characterized by the following. According to this feature, the color of the avatar image reflects the average color of a part of the specific image, rather than the entire image. Therefore, for example, by reflecting the average color of the central or characteristic area of ​​the specific image in the color of the avatar image, the relevance to the specific image can be increased.

[0035] The image generation method of embodiment 2-5 is the image generation method according to embodiment 2-3 or embodiment 2-4, In the irradiation step (irradiation light specification process, irradiation light arrangement process), irradiation light of a color having an average value of colors of the specific image (external input image) having a brightness equal to or greater than a predetermined value is irradiated. It is characterized by the following. According to this feature, the average value of the colors of the specific image that are equal to or greater than a predetermined brightness is reflected in the color of the avatar image, so the avatar image will not be too dark.

[0036] The image generation method of embodiment 2-6 is the image generation method according to any one of embodiment 2-2 to embodiment 2-5, In the irradiation step (irradiation light specification process, irradiation light arrangement process), irradiation light of different colors can be irradiated from a plurality of directions according to the color scheme of the specific image (external input image). It is characterized by the following. According to this feature, the color of the avatar image changes for each of the multiple regions depending on the color scheme of the specific image, so that the specific image and the avatar image can be used to generate an even more realistic image.

[0037] The image generation method of embodiment 2-7 is the image generation method according to any one of embodiment 2-1 to embodiment 2-6, the specific image (external input image) input in the specific image input step (capturing the external input image) is a moving image, Further comprising a motion amount detection step (image analysis program) for detecting a motion amount of the specific image (external input image), In the irradiation step (irradiation light specification process, irradiation light placement process), virtual irradiation light is irradiated onto the 3D model (avatar 3D model) placed in the 3D space (virtual 3D space) based on the amount of movement of the specific image (external input image) detected in the movement amount detection step (image analysis program). It is characterized by the following. According to this feature, the color of the avatar image placed in front of the specific image is changed based on the amount of movement of the specific image, so that a more realistic image can be generated by combining the specific image and the avatar image.

[0038] The image generation method of embodiment 2-8 is the image generation method according to any one of embodiment 2-1 to embodiment 2-7, the composite image is an image that constitutes a moving image, In the irradiation step (irradiation light specification process, irradiation light placement process), a virtual irradiation light based on the specific image (external input image) is irradiated for each frame of the video; the avatar image generating step (avatar image generating process) generates the avatar image from the 3D model (avatar 3D model) illuminated with the virtual illumination light in the illumination step (illumination light specifying process, illumination light positioning process) for each frame of the video; In the composite image generating step (image synthesis program), the composite image is generated for each frame of a video using the specific image (external input image) and the avatar image. It is characterized by the following. According to this feature, a moving image of an avatar can be generated in which a specific image is reflected in each frame.

[0039] The image generation program of form 2-9 is A motion detection step (such as a face tracking program or a motion tracking program) for detecting the movements of the performer; a 3D model generation step (avatar 3D model generation program) for generating a 3D model of an avatar (avatar 3D model) according to the detected movement; a 3D model placement step (object placement process) of placing the generated 3D model (avatar 3D model) in a 3D space (virtual 3D space); an avatar image generation step (avatar image generation process) of generating an avatar image from the 3D model (avatar 3D model) placed in the 3D space (virtual 3D space); a specific image input step (capturing an external input image) for inputting a specific image (external input image); a composite image generating step (image synthesis program) for generating a composite image in which the avatar image is placed in front of the input specific image (external input image); an image output step of outputting the composite image (output of composite image); In an image generation program that causes a computer to execute further executing an irradiation step (irradiation light specification process, irradiation light placement process) of irradiating the 3D model (avatar 3D model) placed in the 3D space (virtual 3D space) with virtual irradiation light based on the specific image (external input image); In the avatar image generating step (avatar image generating process), the avatar image is generated from the 3D model (avatar 3D model) illuminated with the virtual illumination light in the illumination step (illumination light specifying process, illumination light positioning process). It is characterized by the following. According to this feature, an avatar image generated from a 3D model illuminated with virtual light based on the specific image is used as the avatar image placed in front of the specific image, and the color of the avatar image is changed based on the specific image, so that a realistic image can be generated by combining the specific image and the avatar image.

[0040] [Form 3] The image generation method of form 3-1 is A motion detection step (such as a face tracking program or a motion tracking program) for detecting the movements of the performer; an avatar image generating step (avatar image generating program) for generating an avatar image according to the detected movement; a specific image input step (capturing an external input image) for inputting a specific image (external input image); a composite image generating step (image synthesis program) for generating a composite image in which an avatar image is placed in front of the input specific image (external input image); an image output step of outputting the composite image (output of composite image); An image generation method comprising: a voice input step (voice analysis program) for inputting voice (external input voice) added to the specific image (external input image); a color change step (color correction program) of changing the color of the avatar image based on the voice (external input voice) input in the voice input step (voice analysis program); Further equipped It is characterized by the following. According to this feature, the color of the avatar image placed in front of the specific image is changed based on the audio attached to the specific image, so that a realistic image can be generated by combining the specific image and the avatar image.

[0041] The image generation method of embodiment 3-2 is the image generation method according to embodiment 3-1, In the color change step (color correction program), the color of the avatar image is changed when the volume of the voice (external input voice) input in the voice input step (voice analysis program) is equal to or greater than a predetermined threshold (volume setting value). It is characterized by the following. According to this feature, when a loud voice is input, the input voice can be reflected in the color of the avatar image.

[0042] The image generation method of embodiment 3-3 is the image generation method according to embodiment 3-1 or embodiment 3-2, In the color change step (color correction program), the color of the avatar image is changed when a specific voice (high-pitched voice, low-pitched voice) is input in the voice input step (voice analysis program). It is characterized by the following. According to this feature, when a specific sound is input, the color of the avatar image can be reflected.

[0043] The image generation method of embodiment 3-4 is the image generation method according to embodiment 3-3, The specific sound is a sound in a specific range (high-pitched sound, low-pitched sound). It is characterized by the following. According to this feature, when a voice in a specific range is input, the color of the avatar image can be reflected.

[0044] The image generation method of embodiment 3-5 is the image generation method according to embodiment 3-3 or embodiment 3-4, The specific sound includes a plurality of predetermined types of sound (high-pitched sound, low-pitched sound), In the color change step (color correction program), the color of the avatar image is changed to a color set in accordance with the type of voice input in the voice input step (voice analysis program). It is characterized by the following. According to this feature, the color of the avatar image can be changed to a color corresponding to the type of input voice.

[0045] The image generation method of embodiment 3-6 is the image generation method according to any one of embodiment 3-1 to embodiment 3-5, In the color change step (color correction program), the color of the avatar image is changed by multiplying the avatar image by a specific color or a specific image. It is characterized by the following. According to this feature, the color of the avatar image can be changed with a simple process.

[0046] The image generation method of embodiment 3-7 is the image generation method according to any one of embodiment 3-1 to embodiment 3-6, In the color change step (color correction program), the color of the avatar image is changed by changing the brightness. It is characterized by the following. According to this feature, the color of the avatar image can be changed with a simple process.

[0047] The image generation method of embodiment 3-8 is the image generation method according to any one of embodiment 3-1 to embodiment 3-6, The avatar image generating step includes: a 3D model generation step (avatar 3D model generation program) for generating a 3D model of an avatar (avatar 3D model) according to the detected movement; a 3D model placement step (object placement process) of placing the generated 3D model (avatar 3D model) in a 3D space (virtual 3D space); Including, generating the avatar image from the 3D model (avatar 3D model) placed in the 3D space (virtual 3D space); The color changing step includes: an illumination step (illumination light specification process, illumination light arrangement process) of illuminating the 3D model (avatar 3D model) arranged in the 3D space (virtual 3D space) with virtual illumination light based on the voice (external input voice) input in the voice input step (voice analysis program); In the irradiation step (irradiation light specifying process, irradiation light positioning process), the avatar image is generated from the 3D model (avatar 3D model) irradiated with the virtual irradiation light, thereby changing the color of the avatar image. It is characterized by the following. This feature allows the color of the avatar image to be changed in a natural way.

[0048] The image generation method of embodiment 3-9 is the image generation method according to any one of embodiment 3-1 to embodiment 3-8, In the color change step (color correction program), the color of the avatar image is changed based on the specific image (external input image). It is characterized by the following. According to this feature, the color of the avatar placed in front of the specific image is changed based on the specific image, so that a more realistic image can be generated by combining the specific image and the avatar.

[0049] The image generation method of embodiment 3-10 is the image generation method according to embodiment 3-9, In the color change step (color correction program), the color of the avatar image is changed based on the color of the specific image (external input image). It is characterized by the following. According to this feature, the color of the specific image can be reflected in the avatar image.

[0050] The image generation method of embodiment 3-11 is the image generation method according to embodiment 3-9 or embodiment 3-10, the specific image (external input image) input in the specific image input step (capturing the external input image) is a moving image, Further comprising a motion amount detection step (image analysis program) for detecting a motion amount of the specific image (external input image), In the color change step (color correction program), the color of the avatar image is changed based on the amount of movement of the specific image (external input image) detected in the movement amount detection step (image analysis program). It is characterized by the following. According to this feature, the color of the avatar image placed in front of the specific image is changed based on the amount of movement of the specific image, so that a more realistic image can be generated by combining the specific image and the avatar image.

[0051] The image generation method of embodiment 3-12 is the image generation method according to any one of embodiment 3-1 to embodiment 3-11, the composite image is an image that constitutes a moving image, In the color change step (color correction program), a color of the avatar image is changed for each frame of the video based on the audio (external input audio) input in the audio input step (audio analysis program), In the composite image generating step (image synthesis program), the composite image is generated for each frame of a video using the specific image (external input image) and the avatar image with a changed color. It is characterized by the following. According to this feature, a moving image of an avatar can be generated in which the audio attached to the specific image is reflected in each frame.

[0052] The image generation program of form 3-13 is A motion detection step (such as a face tracking program or a motion tracking program) for detecting the movements of the performer; an avatar image generating step (avatar image generating program) for generating an avatar image according to the detected movement; a specific image input step (capturing an external input image) for inputting a specific image (external input image); a composite image generating step (image synthesis program) for generating a composite image in which an avatar image is placed in front of the input specific image (external input image); an image output step of outputting the composite image (output of composite image); In an image generation program that causes a computer to execute a voice input step (voice analysis program) for inputting voice (external input voice) added to the specific image (external input image); a color change step (color correction program) of changing the color of the avatar image based on the voice (external input voice) input in the voice input step (voice analysis program); Run the It is characterized by the following. According to this feature, the color of the avatar image placed in front of the specific image is changed based on the audio attached to the specific image, so that a realistic image can be generated by combining the specific image and the avatar image.

[0053] DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS The present invention will be described below with reference to the accompanying drawings based on the embodiments. [Example]

[0054] [Computer Terminal] Fig. 1 is a block diagram showing an example of the configuration of a computer terminal 1 used in a first embodiment of the present invention. A computer terminal 1 having the same configuration is also used in a second embodiment, which will be described later. The computer terminal 1 shown in Fig. 1 may be a desktop PC or a workstation having separate display devices and input devices such as a keyboard and a mouse, a notebook PC having an integrated display device and input device, or a mobile terminal such as a tablet terminal or smartphone having a display device equipped with a touch panel or the like. However, the following description will be given assuming a terminal such as a desktop PC or a notebook PC.

[0055] As shown in FIG. 1, the computer terminal 1 is equipped with a processor 101, a memory 102, and a storage 103 such as a hard disk or SSD, and the processor 101, memory 102, and storage 103 are connected via a data bus 111, and the processor 101 can execute various processes according to programs stored in the storage 103.

[0056] Furthermore, a display device 105 and a speaker 106 are connected to the data bus 111 via an output interface (not shown), and images and sounds can be output based on the processing executed by the processor 101 .

[0057] In addition, input devices such as an input device 107, a camera 108, a microphone 109, a vital sign measuring instrument 110, a capture device 112, and an audio input device 113 are connected to the data bus 111 via an input interface (not shown), and information can be input via these input devices. The input device 107 is an input device such as a keyboard or a mouse, and various commands can be input. Note that commands can also be input via an input device such as a touch panel formed integrally with the display device 105. The camera 108 is a 3D camera, and can input data including depth data as captured image data. The microphone 109 may be a stereo microphone or a monaural microphone, and can input ambient sounds including the subject's voice. The vital sign measuring instrument 110 is a wearable device such as a smart band, and can input vital sign data (heart rate, blood pressure, body temperature, etc.). The capture device 112 can input external video data and capture and import images constituting the input external video data at a predetermined frame rate. The audio input device 113 is capable of inputting audio that is added to external video data and output together with the external video data.

[0058] Furthermore, a communication interface 104 is connected to the data bus 111, and is configured to be able to communicate with other computer terminals and server computers via a local area network, the Internet, or other public network, either wired or wirelessly.

[0059] While this embodiment illustrates a configuration in which the image generation method of the present invention is implemented using one computer terminal, it may also be implemented using multiple computer terminals. Furthermore, while the computer terminal 1 of this embodiment is configured to include a display device 105 and a speaker 106 as output devices, it may also be configured to not include these output devices and output images and audio from the output device of another computer terminal. Furthermore, the computer terminal 1 of this embodiment is configured to include input devices such as an input device 107 (e.g., a keyboard and a mouse), a camera 108, and a microphone 109, but it is sufficient if it is configured to include at least a touch panel, a keyboard, a mouse, a camera capable of capturing images, and a capture device capable of capturing images from external video data.

[0060] 2, various programs executed by the computer terminal 1 are stored in the storage 103 of the computer terminal 1, in addition to an operating system (OS) not shown. Specifically, a face tracking program, a motion tracking program, a voice detection program, an input detection program, a vital sign detection program, a parameter correction program, an avatar image generation program, an image analysis program, a voice analysis program, a color correction program, an image synthesis program, and a distribution program are stored, and, although not particularly shown, configuration settings, material data used by the avatar image generation program, and the like are also stored.

[0061] As shown in Figures 3 to 5, computer terminal 1 is configured so that an avatar image generation program generates an avatar image with an expression based on the expression parameters and a form based on the motion parameters based on the expression parameters and the motion parameters, based on the image data output from camera 108; an image synthesis program generates a composite image by combining an external input image captured by capture device 112 as a background image with the avatar image generated by the avatar image generation program in front of it; and a distribution program can distribute a video using the composite image generated by the image synthesis program.

[0062] Furthermore, the facial expression parameters output by the face tracking program are not output directly to the avatar image generation program, but are output to the avatar image generation program via a parameter correction program, as shown in Figure 3. The parameter correction program corrects at least some of the facial expression parameters using some of the facial expression parameters and parameters other than the facial expression parameters (voice parameters, input parameters, vital parameters, and configuration parameters), and outputs the corrected facial expression parameters to the image generation program.

[0063] Furthermore, the avatar image generated by the avatar image generation program is not output directly to the image synthesis program; rather, as shown in Figures 4 and 5, the avatar image is color-corrected by a color correction program and then output to the image synthesis program. The color correction program performs color correction on the avatar image based on the color of the external input image to be synthesized by the image synthesis program, the amount of movement in the external input image, and the external input audio input together with the external input image, and then outputs the color-corrected avatar image to the image synthesis program.

[0064] Note that color correction, or correcting a color, means correcting the elements that make up a color, namely, hue, recolor, and brightness. It may be a configuration in which all of the hue, recolor, and brightness are corrected, or a configuration in which only part of the hue, recolor, and brightness are corrected.

[0065] Here, the computer terminal 1 can input video content such as game play videos and movies as external video data from an external device and import images of this video data as external input images, and by distributing videos that combine the input video content such as game play videos and movies with avatar images, it is possible to distribute game commentary videos and movie commentary videos as new videos.

[0066] [program] Next, a program executed by the computer terminal 1 of this embodiment will be described.

[0067] As shown in FIG. 3, the face tracking program is a program that uses image data including depth data input from the camera 108 to detect the state of multiple facial features constituting a facial image of a subject, and outputs facial parameters that quantify the degree of movement of each feature from the detected state. The facial parameters are parameters that can identify the facial expression of the subject. In this embodiment, the facial parameters are parameters that quantify, in the range of 0.0 to 1.0, six types of movement degrees related to left eye movement, six types of movement degrees related to right eye movement, 27 types of movement degrees related to mouth and jaw movement, 10 types of movement degrees related to eyebrow, cheek, and nose movement, and one type of movement degree related to tongue movement. Note that the facial parameters are not limited to these exemplified ones, and more detailed types of facial parameters may be used, or fewer types of facial parameters may be used.

[0068] 3, the motion tracking program is a program that detects the body movement of the subject using image data including depth data input from the camera 108, and outputs motion parameters that identify the positions and directions of parts that make up the body. The motion parameters are parameters that can identify the body movement of the subject, and in this embodiment include a head pose that indicates the position coordinates and direction of the head, all finger joints that indicate the position coordinates and directions of all finger joints, and a hand pose that indicates the position coordinates and directions of the hand. Note that the motion parameters in this embodiment are configured not to include parameters regarding the movement of the entire body, but may be configured to include parameters regarding the movement of the entire body.

[0069] 3, the voice detection program is a program that detects the voice of the target person using voice data input from microphone 109 and outputs voice parameters that digitize the components of the detected voice. The voice parameters are parameters that can identify the voice of the target person, and in this embodiment, are parameters that digitize the voice volume and voice pitch, each in the range of 0.0 to 1.0.

[0070] 3, the input detection program is a program that converts input data from the input device 107 into input parameters and outputs them. The input parameters are parameters that can identify the input status from the input device 107, and in this embodiment, are parameters that indicate the type of command input via a keyboard or the like. Command inputs include, for example, command inputs that specify emotions such as joy, anger, sadness, and pleasure in stages, command inputs that specify decorative images, and the like.

[0071] 3, the vital sign detection program is a program that converts vital data (heart rate, blood pressure, body temperature, etc.) input from the vital sign measuring device 110 and outputs vital parameters. The vital parameters are parameters that can identify the subject's vital signs, and in this embodiment, the heart rate, blood pressure, and body temperature are each converted into a numerical value between 0.0 and 1.0.

[0072] As shown in FIG. 3, the parameter correction program is a program that primarily corrects facial expression parameters created by the face tracking program, and includes a parameter correction process that corrects at least some of the facial expression parameters created by the face tracking program using some of the facial expression parameters, voice parameters from the voice detection program, input parameters from the input detection program, vital parameters from the vital detection program, and configuration parameters based on preset configurations (setting values), and a parameter creation process that creates new facial expression parameters based on input parameters. The parameter correction program is executed for each frame at intervals based on the frame rate of the video distributed by the distribution program. For example, if the frame rate is 60 fps, the parameter correction program is executed 60 times per second.

[0073] The config (setting value) is a value that can set the amount of change in the facial expression parameter, and can be set by displaying a configuration setting screen on the display device 105 and using the input device 107. The config (setting value) can be set for each type of facial expression parameter, and in this embodiment, a numerical value between 0.0 and 1.0 can be set individually for each of the facial expression parameters related to eye movement, the facial expression parameters related to mouth and jaw movement, the facial expression parameters related to eyebrow movement, the facial expression parameters related to cheek and nose movement, and the facial expression parameters related to tongue movement, and the set numerical value is used as the config parameter. In this embodiment, the larger the numerical value set as the config, the greater the amount of change in the corresponding facial expression parameter and the greater the movement of the corresponding part.

[0074] As shown in FIG. 3, the avatar image generation program executes a process to generate an avatar image based on the facial expression parameter N and extension parameters from the parameter correction program, the motion parameters from the motion tracking program, the audio parameters from the audio parameters, and the input parameters from the input detection program, and outputs the generated avatar image. The program includes a facial image generation process to generate a facial image, a facial image correction process to correct the facial image, a costume image generation process to generate a costume image, and an ornamental image generation process to generate an ornamental image. The avatar image generation program is executed for each frame at intervals corresponding to the frame rate of the video distributed by the distribution program. For example, if the frame rate is 60 fps, the avatar image generation program is executed 60 times per second.

[0075] As shown in FIG. 4, the image analysis program is a program that, based on an external input image captured by the capture device 112 from external video data, determines the amount of movement in the external input image from the difference between the external input image captured last time and outputs the amount of movement detection data. The image analysis program determines the ratio of pixels whose data differs between the currently captured external input image and the previously captured external input image as the amount of movement, and outputs a numerical value between 0.0 and 1.0. Note that it is also possible to compare only a partial area of ​​the external input image rather than the entire area of ​​the external input image and determine the amount of movement from the difference. The capture device 112 captures the external input image from the external video data frame by frame at intervals corresponding to the frame rate of the video distributed by the distribution program, and the image analysis program is also executed frame by frame at intervals corresponding to the frame rate of the video distributed by the distribution program. For example, if the frame rate is 60 fps, the external input image is captured and the image analysis program is executed 60 times per second.

[0076] As shown in Fig. 4, the audio analysis program is a program that determines the volume and pitch of audio input by audio input device 113, i.e., audio added to external video data and output together with the external video data, and outputs volume data indicating the determined volume and pitch data indicating the determined pitch as numerical values ​​between 0.0 and 1.0. The audio analysis program is executed for each frame at intervals according to the frame rate of the video distributed by the distribution program. For example, if the frame rate is 60 fps, the audio analysis program is executed 60 times per second.

[0077] As shown in FIG. 4, the color correction program is a program that performs color correction on the avatar images generated by the avatar image generation program based on the color of the external input image captured from external video data by capture device 112, the movement amount detection data output from the image analysis program, and the volume and pitch data output from the audio analysis program, and outputs the corrected avatar images. The color correction program includes a multiplication image generation process that generates an image to be multiplied by the avatar image, a multiplication process that multiplies the avatar image by the multiplication image, and a brightness change process that changes the brightness of the avatar image. The color correction program is executed for each frame at intervals that correspond to the frame rate of the video distributed by the distribution program. For example, if the frame rate is 60 fps, the color correction program is executed 60 times per second.

[0078] As shown in Fig. 4, the image synthesis program is a program that generates and outputs a synthesized image (see Fig. 5) in which an external input image captured by capture device 112 from external video data is used as a background image and an avatar image generated by the avatar image generation program and color corrected by the color correction program is placed in front of the external input image. The image synthesis program is executed for each frame at intervals corresponding to the frame rate of the video distributed by the distribution program. For example, if the frame rate is 60 fps, the image synthesis program is executed 60 times per second.

[0079] 4, the distribution program is a program that distributes video data consisting of a video composed of a composite image in which an avatar image is placed in front of an external input image output from the image synthesis program, audio input from microphone 109, and audio input from audio input device 113. The frame rate of the video data distributed by the distribution program can be set from among several types, and at intervals based on the set frame rate, the external input image is captured by capture device 112 as described above for each frame of the video data distributed by the distribution program, and the parameter correction program, avatar image generation program, image analysis program, audio analysis program, color correction program, and image synthesis program are executed.

[0080] [Color correction program] FIG. 6 is a flowchart showing the control content of the color correction program.

[0081] As shown in Fig. 6, the color correction program first acquires an external input image captured by the capture device 112 from external video data (Sa1). Next, a set area of ​​the acquired external input image is clipped (Sa2). The area to be clipped can be set arbitrarily, and in this embodiment, an example is shown in which the central area of ​​the external input image is clipped, as shown in Fig. 7(a).

[0082] Next, the cut-out set area is divided into a plurality of areas (Sa3). The number and size of the divided areas can be set arbitrarily, and in this embodiment, as shown in Figure 7(a), an example is shown in which the area is divided into three areas vertically and horizontally, resulting in nine areas.

[0083] Next, colors with a brightness of 50% or more are extracted for each region, and their average value is calculated. As shown in FIG. 7(b), the calculated average value is set as the color of each region (Sa4).

[0084] Next, as shown in FIG. 7(c), the boundaries between the multiple regions are blurred (Sa5), and then the image is cut out to the outline shape of the avatar image (Sa6), as shown in FIG. 7(d).

[0085] Next, as shown in FIG. 8(a), an outer edge region of the image cut out to the outline shape is set (Sa7), and as shown in FIG. 8(b), the brightness is gradually decreased from the inside to the outside of the outer edge region (Sa8).

[0086] Next, an avatar image output from the avatar image generation program is acquired (Sa9), and as shown in FIG. 8(c), the acquired avatar image is multiplied by the multiplied image generated in Sa1 to Sa8 (Sa10).

[0087] Next, based on the movement amount detection data output from the image analysis program, it is determined whether the movement amount of the external input image is equal to or greater than a preset movement setting value (Sa11). If the movement amount of the external input image is not equal to or greater than the movement setting value, the process proceeds to Sa13. If the movement amount of the external input image is equal to or greater than the movement setting value, the brightness of the avatar image multiplied by the multiplication image in Sa10 is increased by 20% (Sa12). The movement setting value referenced in Sa11 can be set arbitrarily, and the brightness to be varied in Sa12 can also be set arbitrarily.

[0088] Next, based on the volume data and pitch data output from the voice analysis program, it is determined whether the volume of the external input voice is equal to or greater than the volume setting value and the pitch is equal to or greater than the treble setting value (Sa13). If the volume of the external input voice is less than the volume setting value or the pitch is less than the treble setting value, the process proceeds to Sa15. If the volume of the external input voice is equal to or greater than the volume setting value and the pitch is equal to or greater than the treble setting value, the brightness of the avatar image multiplied by the multiplication image in Sa10 is increased by 20% (Sa14). The volume setting value and treble setting value referenced in Sa13 can be set arbitrarily, and the brightness to be varied in Sa14 can also be set arbitrarily.

[0089] Next, based on the volume data and pitch data output from the audio analysis program, it is determined whether the volume of the external input audio is equal to or greater than the volume setting value and the pitch is equal to or less than the bass setting value (Sa15). If the volume of the external input audio is less than the volume setting value or the pitch is greater than the bass setting value, the process proceeds to Sa17. If the volume of the external input audio is equal to or greater than the volume setting value and the pitch is equal to or less than the bass setting value, the brightness of the avatar image multiplied by the multiplication image in Sa10 is reduced by 20% (Sa16). The volume setting value and bass setting value referenced in Sa15 can be set arbitrarily, and the brightness to be varied in Sa16 can also be set arbitrarily.

[0090] Next, the color-corrected avatar image is output to an image synthesis program (Sa17).

[0091] [Effect 1] In this embodiment, an external input image captured by capture device 112 from external video data is used as a background image, and a composite image is generated and output in which an avatar image generated by an avatar image generation program is placed in front of the external input image. Since the color of the avatar image placed in front of the external input image is changed based on the external input image, a realistic image can be generated by combining the external input image and the avatar image.

[0092] In addition, in this embodiment, the color of the avatar image is changed based on the external input image each time a frame of video is generated, so that a video of the avatar image can be generated that reflects the external input image based on the external video data for each frame.

[0093] Note that the color of the avatar image may be changed based on the external input image every time multiple frames are generated.

[0094] In this embodiment, the avatar image is multiplied by a multiplication image based on the color of the external input image, allowing the color of the external input image to be reflected in the avatar image, thereby creating a sense of realism as if the avatar image were being viewed through a monitor displaying the external input image.

[0095] In addition, in this embodiment, the avatar image is multiplied by the average value of the colors that make up the external input image and whose brightness is equal to or higher than a predetermined brightness (50% or higher in this embodiment). This allows the average value of the colors in the external input image to be reflected in the color of the avatar image, and prevents the avatar image from becoming too dark.

[0096] In this embodiment, the avatar image is multiplied by the average value of the colors that make up the external input image and whose brightness is equal to or greater than a predetermined brightness. However, the avatar image may also be multiplied by the average value of all the colors that make up the external input image. Even with this configuration, the average value of the colors in the external input image can be reflected in the color of the avatar image.

[0097] Furthermore, in this embodiment, the average color value of a set area of ​​the external input image is multiplied by the avatar image, and the average color value of that area, rather than the entire external input image, is reflected in the color of the avatar image.Therefore, for example, by reflecting the average color value of a central area or a characteristic area of ​​the external input image in the color of the avatar image, the relevance to the external input image can be increased.

[0098] In addition, the avatar image may be multiplied by the average color value of the entire external input image. By using this configuration, the processing load can be reduced while the average color value of the external input image is reflected in the avatar image.

[0099] In addition, in this embodiment, the external input image is divided into multiple regions, and the average color value of each region is set as the color of that region and multiplied by the avatar image.Since the color of each of the multiple regions changes depending on the color scheme of the external input image, it is possible to generate an even more realistic image by combining the external input image and the avatar image.

[0100] Furthermore, in a multiplied image in which different colors are set for each of a plurality of regions, the boundaries of the regions are blurred, so that the color boundaries are not clearly projected onto the avatar image, and the different colors for each of the plurality of regions can be naturally reflected on the avatar image.

[0101] In this embodiment, the avatar image is multiplied by a multiplication image in which different colors are set for each of multiple regions, but it may also be configured so that the avatar image is multiplied by a single-color image, such as the average color value based on the entire or part of the external input image, and such a configuration can reduce the processing load.

[0102] In addition, in this embodiment, the brightness of the outer edge region of the multiplied image based on the external input image is reduced, and the brightness of the color multiplied in the central region and the outer edge region of the avatar image is different, so the color of the external input image can be reflected in the avatar image in a three-dimensional manner.

[0103] In particular, in this embodiment, by gradually decreasing the brightness from the inside to the outside of the outer edge region of the multiplied image, the colors of the external input image can be reflected in the avatar image in a natural and three-dimensional manner.

[0104] It is also possible to adopt a configuration in which the brightness is not changed depending on the area of ​​the multiplied image, thereby reducing the processing load.

[0105] In this embodiment, the avatar image is multiplied by the average color value based on the external input image. However, as shown in FIG. 9, the avatar image may also be multiplied by the external input image itself. Even in this configuration, the color of the external input image can be reflected in the avatar image.

[0106] In this case, instead of simply multiplying the external input image onto the avatar image, as shown in Figure 9, the external input image is blurred and then multiplied onto the avatar image. This prevents the external input image from being projected too clearly onto the avatar image, allowing the color of the external input image to be reflected naturally in the avatar image.

[0107] Furthermore, in this embodiment, when it is detected that the amount of movement in the external input image is equal to or greater than a predetermined amount of movement, the brightness of the multiplied image by which the avatar image is multiplied is added, and the color of the avatar image placed in front of the external input image is changed based on the amount of movement in the external input image, thereby enabling the generation of a realistic image from the external input image and the avatar image.

[0108] In this embodiment, when it is detected that the amount of movement in the external input image is equal to or greater than a predetermined amount of movement, the color of the avatar image is changed by changing the brightness of the multiplied image by which the avatar image is multiplied. However, it is also possible to change the color of the avatar image by multiplying or adding a specific color or image to the avatar image when it is detected that the amount of movement in the external input image is equal to or greater than a predetermined amount of movement. Even in this configuration, the color of the avatar image placed in front of the external input image is changed based on the amount of movement in the external input image, making it possible to generate a realistic image using the external input image and the avatar image.

[0109] Furthermore, in this embodiment, the color of the avatar image is changed based on the amount of movement in the external input image each time a frame of video is generated, so that a video of the avatar image can be generated that reflects the external input image based on the amount of movement in the external input image for each frame.

[0110] Note that the color of the avatar image may be changed based on the amount of movement in the external input image every time multiple frames are generated.

[0111] Furthermore, in this embodiment, the brightness of the multiplied image multiplied onto the avatar image is changed based on the external input audio added to the external video data, and the color of the avatar image placed in front of the external input image is changed based on the audio added to the external video data, so that a realistic image can be generated by combining the external input image and the avatar image.

[0112] In this embodiment, the color of the avatar image is changed by changing the brightness of the multiplied image multiplied by the avatar image based on the external input audio added to the external video data. However, the color of the avatar image may also be changed by multiplying or adding a specific color or image to the avatar image based on the external input audio added to the external video data. Even in such a configuration, the color of the avatar image placed in front of the external input image is changed based on the audio added to the external video data, so that a realistic image can be generated by combining the external input image and the avatar image.

[0113] In addition, in this embodiment, if the external input audio added to the external video data is equal to or greater than a preset volume setting value, the color of the avatar image is changed, so that if loud audio is input, the color of the avatar image can be reflected.

[0114] In addition, in this embodiment, when a predetermined high-pitched sound or a predetermined low-pitched sound is input as the external input sound added to the external video data, that is, when a specific sound is input, the color of the avatar image is changed, so that the color of the avatar image can reflect that a specific sound has been input.

[0115] In addition, in this embodiment, when audio in a specific range is input as external input audio added to external video data, the color of the avatar image is changed, so that the color of the avatar image can reflect the fact that audio in a specific range has been input.

[0116] In addition, in this embodiment, the color of the avatar image is changed to a color that is set depending on whether a predetermined high-pitched sound or a predetermined low-pitched sound is input as the external input sound added to the external video data, i.e., depending on the type of input sound, and the color of the avatar image can be changed to a color that corresponds to the type of input sound.

[0117] In this embodiment, a specific range of sound is identified and the color of the avatar image is changed depending on the type of sound. However, it is also possible to identify the type of sound, such as the genre of background music, and change the color of the avatar image depending on the type of sound identified. Even in such a configuration, the color of the avatar image can be changed to a color that corresponds to the type of sound input.

[0118] Furthermore, in this embodiment, the color of the avatar image is changed based on the external input audio added to the external video data each time a frame of video is generated, so that a video of the avatar image can be generated that reflects the audio based on the external video data for each frame.

[0119] Note that the color of the avatar image may be changed based on the external input sound added to the external input image every time multiple frames are generated.

[0120] Furthermore, in this embodiment, color correction based on the color of the external input image, color correction based on the amount of movement of the external input image, and color correction based on the external input audio are all configured to be performed each time one frame of video is generated, but color correction based on the color of the external input image, color correction based on the amount of movement of the external input image, and color correction based on the external input audio may be configured to be performed at different frequencies. For example, color correction based on the color of the external input image may be configured to be performed each time one frame of video is generated, while color correction based on the amount of movement of the external input image and color correction based on the external input audio may be configured to be performed each time multiple frames are generated.

[0121] Although the first embodiment of the present invention has been described above with reference to the drawings, the present invention is not limited to this first embodiment, and it goes without saying that the present invention also includes modifications and additions that do not deviate from the spirit of the present invention.

[0122] For example, in the above-mentioned first embodiment, an example was described in which a composite image generated by an image synthesis program is used for video distribution, but the use of a composite image generated by an image synthesis program is arbitrary, and the composite image may be recorded as an archive, or may be used for producing animation, etc.

[0123] In addition, in the above-mentioned first embodiment, an example has been described in which facial expression parameters output from a face tracking program are corrected by a parameter correction program, and an image generation program generates an avatar image using the corrected facial expression parameters. However, a configuration may also be adopted in which an avatar image is generated by an image generation program directly using parameters output from a face tracking program without using a parameter correction program.

[0124] Furthermore, in the above-described first embodiment, a composite image is generated and output in which an external input image captured by capture device 112 from external video data is used as a background image and an avatar image generated by an avatar image generation program is placed in front of the external input image, and the color of the avatar image is corrected based on each of the color of the external input image, the amount of movement of the external input image, and the external input audio added to the external input image. However, any other configuration may be used in which the color of the avatar image is corrected based on at least one element of the color of the external input image, the amount of movement of the external input image, and the external input audio added to the external input image.

[0125] In addition, in the above-mentioned Example 1, an avatar image generation program generates an avatar image, and a color correction program separate from the avatar image generation program changes the color of the avatar image generated by the avatar image generation program based on an external input image. However, it is also possible to configure the avatar image generation program to generate an avatar image and change its color based on an external input image. [Example]

[0126] A second embodiment of the present invention will be described below. The computer terminal 1 used in the second embodiment has the same configuration as the computer terminal 1 used in the first embodiment, and so the same reference numerals are used and detailed description thereof will be omitted. Furthermore, the configurations and variations thereof that are applicable to the first embodiment can also be applied to this embodiment.

[0127] 10, the storage 103 of the computer terminal 1 of this embodiment stores various programs executed by the computer terminal 1, in addition to an operating system (OS) not shown. Specifically, a face tracking program, a motion tracking program, a voice detection program, an input detection program, a vital sign detection program, a parameter correction program, an avatar 3D model generation program, an image analysis program, a voice analysis program, a 3D model color correction program, an image synthesis program, and a distribution program, and further stores configuration settings, material data used by the image generation program, and the like, which are not particularly shown.

[0128] As shown in Figures 11 to 13, computer terminal 1 is configured so that an avatar 3D model generation program generates an avatar 3D model with an expression based on the expression parameters and a form based on the motion parameters based on the expression parameters and motion parameters based on image data output from camera 108, an image synthesis program uses an external input image captured by capture device 112 as a background image and generates a composite image by combining an avatar image based on the avatar 3D model in front of it, and a distribution program can distribute a video using the composite image generated by the image synthesis program.

[0129] Furthermore, the facial expression parameters output by the face tracking program are not output directly to the avatar 3D model generation program, but are output to the avatar 3D model generation program via a parameter correction program, as shown in FIG. 11. The parameter correction program corrects at least some of the facial expression parameters using some of the facial expression parameters and parameters other than the facial expression parameters (voice parameters, input parameters, vital parameters, and configuration parameters), and outputs the corrected facial expression parameters to the avatar 3D model generation program.

[0130] Furthermore, the avatar 3D model generated by the avatar 3D model generation program is not an avatar image based on that avatar 3D model that is output as is to the image synthesis program; rather, as shown in Figures 12 and 13, a 3D model color correction program illuminates the avatar 3D model with virtual light, thereby correcting the color of the avatar image based on the avatar 3D model, and outputs the color-corrected avatar image to the image synthesis program. The 3D model color correction program illuminates the avatar 3D model with virtual light based on the color of the external input image to be synthesized by the image synthesis program, the amount of movement in the external input image, and the external input audio input together with the external input image, thereby correcting the color of the avatar image based on the avatar 3D, and outputs the color-corrected avatar image to the image synthesis program.

[0131] Here, similar to Example 1, the computer terminal 1 of this example is capable of inputting video content such as gameplay videos and movies as external video data from an external device and capturing images of this video data as external input images, and is configured to be able to distribute game commentary videos and movie commentary videos as new videos by distributing videos that combine the input video content such as gameplay videos and movies with avatar images.

[0132] [program] Next, a description will be given of the programs executed by the computer terminal 1 of this embodiment. Note that a description of programs common to the first embodiment will be omitted.

[0133] As shown in FIG. 11, the avatar 3D model generation program executes a process to generate an avatar 3D model based on the facial expression parameter N and extension parameters from the parameter correction program, the motion parameters from the motion tracking program, the audio parameters from the audio parameters, and the input parameters from the input detection program, and outputs the generated avatar 3D model. The program includes a face model generation process to generate a face model, a face model correction process to correct the face model, a clothing model generation process to generate a clothing model, and an ornament model generation process to generate an ornament model. The avatar 3D model generation program is executed for each frame at intervals corresponding to the frame rate of the video distributed by the distribution program. For example, if the frame rate is 60 fps, the avatar 3D model generation program is executed 60 times per second.

[0134] As shown in FIG. 12 , the 3D model color correction program is a program that performs color correction on an avatar image based on an avatar 3D model generated by an avatar 3D model generation program by irradiating a virtual light onto the avatar 3D model based on the color of an external input image captured from external video data by capture device 112, movement amount detection data output from the image analysis program, and volume and pitch data output from the audio analysis program, and generates and outputs the corrected avatar image. The program includes an object placement process that places the avatar 3D model in virtual 3D space, an irradiation light identification process that identifies the irradiation light to be irradiated onto the avatar 3D model, an irradiation light placement process that places the irradiation light identified by the irradiation light identification process in virtual 3D space, and an avatar image generation process that generates an avatar image based on the avatar 3D model. The 3D model color correction program is executed for each frame at intervals corresponding to the frame rate of the video distributed by the distribution program. For example, if the frame rate is 60 fps, the 3D model color correction program is executed 60 times per second.

[0135] As shown in Fig. 12, the image synthesis program is a program that generates and outputs a synthesized image (see Fig. 13) in which an external input image captured by capture device 112 from external video data is used as a background image, and an avatar image based on a 3D avatar generated by the avatar 3D model generation program and color-corrected by the 3D model color correction program is placed in front of the external input image. The image synthesis program is executed for each frame at intervals corresponding to the frame rate of the video distributed by the distribution program. For example, if the frame rate is 60 fps, the image synthesis program is executed 60 times per second.

[0136] 12, the distribution program is a program that distributes video data consisting of a video composed of a composite image in which an avatar image is placed in front of an external input image output from the image synthesis program, audio input from microphone 109, and audio input from audio input device 113. The frame rate of the video data distributed by the distribution program can be set from among several types, and as described above, the external input image is captured by capture device 112 for each frame of the video data distributed by the distribution program at intervals based on the set frame rate, and the parameter correction program, avatar 3D model generation program, image analysis program, audio analysis program, 3D model color correction program, and image synthesis program are executed.

[0137] [3D model color correction program] FIG. 14 is a flowchart showing the control contents of the 3D model color correction program.

[0138] As shown in FIG. 14, in the 3D model color correction program, first, the avatar 3D model generated by the avatar 3D model generation process is placed in the virtual 3D space (Sb1).

[0139] Next, an external input image is acquired by capturing it from the external video data using the capture device 112 (Sb2). Next, a set area of ​​the acquired external input image is cut out (Sb3). The area to be cut out can be set arbitrarily, and in this embodiment, an example is shown in which the central area of ​​the external input image is cut out, as shown in Figure 15(a).

[0140] Next, the cut-out set area is divided into a plurality of areas (Sb4). The number and size of the divided areas can be set arbitrarily, and in this embodiment, as shown in Figure 15(a), the area is divided into three areas vertically and horizontally, resulting in nine areas.

[0141] Next, colors with a brightness of 50% or more are extracted for each region, and their average value is calculated. As shown in FIG. 15(b), the calculated average value is set as the color of each region (Sb5).

[0142] Next, as shown in Figures 15(c) to 15(e), virtual light sources of illumination light are placed in the virtual 3D space at coordinates corresponding to the regions divided in Sb4, facing the direction of the 3D avatar (Sb6). The virtual illumination light sources are light sources that irradiate the respective regions with the colors set for each region in Sb5. As a result, illumination light of the colors set for each region based on the external input image is irradiated onto the avatar 3D model from the directions set for each region, changing the color of the avatar 3D model.

[0143] Next, based on the motion amount detection data output from the image analysis program, it is determined whether the motion amount of the external input image is equal to or greater than a preset motion setting value (Sb7). If the motion amount of the external input image is not equal to or greater than the motion setting value, the process proceeds to Sb9. If the motion amount of the external input image is equal to or greater than the motion setting value, the brightness of all the light sources of the illumination light arranged in Sb6 is increased by 20% (Sb8). The motion setting value referenced in Sb7 can be set arbitrarily, and the brightness of the light source varied in Sb8 can also be set arbitrarily.

[0144] Next, based on the volume data and pitch data output from the audio analysis program, it is determined whether the volume of the external input audio is equal to or greater than the volume setting value and the pitch is equal to or greater than the treble setting value (Sb9), and if the volume of the external input audio is less than the volume setting value or the pitch is less than the treble setting value, the process proceeds to Sb11, and if the volume of the external input audio is equal to or greater than the volume setting value and the pitch is equal to or greater than the treble setting value, the brightness of all the light sources of the illumination light arranged in Sb6 is increased by 20% (Sb10). The volume setting value and treble setting value referenced in Sb9 can be set arbitrarily, and the brightness of the light source varied in Sb10 can also be set arbitrarily.

[0145] Next, based on the volume data and pitch data output from the audio analysis program, it is determined whether the volume of the external input audio is equal to or greater than the volume setting value and the pitch is equal to or less than the bass setting value (Sb11), and if the volume of the external input audio is less than the volume setting value or the pitch is greater than the bass setting value, the process proceeds to Sb13, and if the volume of the external input audio is equal to or greater than the volume setting value and the pitch is equal to or less than the bass setting value, the brightness of all the light sources of the illumination light arranged in Sb6 is reduced by 20% (Sb12). The volume setting value and bass setting value referenced in Sb11 can be set arbitrarily, and the brightness of the light source varied in Sb12 can also be set arbitrarily.

[0146] Next, as shown in Figure 15(d), an avatar image is generated from the avatar 3D model placed in the virtual 3D space (Sb13). Specifically, an image of the avatar 3D model as seen from a specified direction in the virtual 3D space is written out as the avatar image.

[0147] Next, the generated avatar image is output to an image synthesis program (Sb14).

[0148] [Effect 2] In this embodiment, the external input image captured by capture device 112 from external video data is used as a background image, and a composite image is generated and output in which an avatar image based on an avatar 3D model generated by an avatar 3D model generation program is placed in front of the external input image.The avatar image placed in front of the external input image is an avatar image generated from an avatar 3D model illuminated with virtual illumination light based on the external input image, and the color of the avatar image is changed based on the external input image, making it possible to generate a realistic image using the external input image and the avatar image.

[0149] Furthermore, in this embodiment, each time a frame of a video is generated, a virtual illumination light based on an external input image is irradiated onto an avatar 3D model, and each time a frame of a video is generated, an avatar image is generated from the avatar 3D model illuminated with the virtual illumination light, thereby changing the color of the avatar image. As a result, a video of an avatar image that reflects an external input image based on external video data for each frame can be generated.

[0150] Note that the color of the avatar image based on the avatar 3D model may be changed by changing the virtual illumination light based on the external input image each time multiple frames are generated.

[0151] In this embodiment, the 3D avatar model is illuminated with light of the color of the external input image, and the color of the avatar image based on the 3D avatar model is changed, so that the color of the external input image can be reflected in the avatar image, creating a sense of realism as if the avatar image were being viewed on a monitor displaying the external input image.

[0152] Furthermore, in this embodiment, the avatar 3D model is illuminated with light of the average color of the colors that make up the external input image and whose brightness is equal to or higher than a predetermined brightness (50% or higher in this embodiment), and the color of the avatar image based on the avatar 3D model is changed, so that the average color of the external input image can be reflected in the color of the avatar image and the avatar image does not become too dark.

[0153] In this embodiment, the avatar 3D model is illuminated with light of the average color of the colors that make up the external input image and whose brightness is equal to or greater than a predetermined brightness. However, the avatar 3D model may also be illuminated with light of the average color of all the colors that make up the external input image. Even with this configuration, the average color of the external input image can be reflected in the color of the avatar image.

[0154] Furthermore, in this embodiment, the avatar 3D model is illuminated with light of a color that is the average color of a set area of ​​the external input image, and the average color of that area, rather than the entire external input image, is reflected in the color of the avatar image.Therefore, for example, by reflecting the average color of a central area or a characteristic area of ​​the external input image in the color of the avatar image, it is possible to increase the relevance to the external input image.

[0155] In addition, the avatar 3D model may be illuminated with light of a color that is the average color of the entire external input image.By using such a configuration, the processing load can be reduced while the average color of the external input image can be reflected in the avatar image.

[0156] In addition, in this embodiment, the external input image is divided into multiple regions, the average color value of each region is set as the color of that region, and light of the set color corresponding to each region is emitted from the coordinates corresponding to each region toward the avatar 3D model.Since the color changes for each of the multiple regions depending on the color scheme of the external input image, it is possible to generate an even more realistic image by combining the external input image and the avatar image.

[0157] In this embodiment, the avatar 3D model is illuminated with multiple light sources, each with a different color set for each of the multiple regions. However, the avatar 3D model may be illuminated with a single color light, such as an average color based on the entire external input image or a portion of the image, which can reduce the processing load. In this case, the illumination light may be emitted from multiple or single light sources.

[0158] Furthermore, in this embodiment, if it is detected that the amount of movement in the external input image is equal to or greater than a predetermined amount of movement, the brightness of the light source that illuminates the avatar 3D model is increased, and the color of the avatar image placed in front of the external input image is changed based on the amount of movement in the external input image, thereby enabling the generation of a realistic image from the external input image and the avatar image.

[0159] In this embodiment, if it is detected that the amount of movement in the external input image is equal to or greater than a predetermined amount of movement, the color of the avatar image is changed by changing the brightness of the light source of the light that illuminates the avatar 3D model. However, it is also possible to change the color of the avatar image by changing the color of the light source of the light that illuminates the avatar 3D model when it is detected that the amount of movement in the external input image is equal to or greater than a predetermined amount of movement. Even in this configuration, the color of the avatar image placed in front of the external input image is changed based on the amount of movement in the external input image, making it possible to generate a realistic image using the external input image and the avatar image.

[0160] Furthermore, in this embodiment, each time a frame of a video is generated, the brightness of the light source that illuminates the avatar 3D model is changed based on the amount of movement in the external input image, and the color of the avatar image is changed, so that a video of an avatar image can be generated that reflects the external input image based on the amount of movement in the external input image for each frame.

[0161] Note that the brightness of the light source that illuminates the avatar 3D model may be changed based on the amount of movement in the external input image each time multiple frames are generated, thereby changing the color of the avatar image.

[0162] In addition, in this embodiment, the brightness of the light source that illuminates the avatar 3D model is changed based on the external input audio added to the external video data, and the color of the avatar image placed in front of the external input image is changed based on the audio added to the external video data, so that a realistic image can be generated by combining the external input image and the avatar image.

[0163] In this embodiment, the color of the avatar image is changed by changing the brightness of the light source of the illumination light that illuminates the avatar 3D model based on the external input audio added to the external video data. However, the color of the avatar image may also be changed by changing the color of the light source of the illumination light that illuminates the avatar 3D model based on the external input audio added to the external video data. Even in this configuration, the color of the avatar image placed in front of the external input image is changed based on the audio added to the external video data, so that a realistic image can be generated by combining the external input image and the avatar image.

[0164] In addition, in this embodiment, if the external input audio added to the external video data is equal to or greater than a preset volume setting value, the brightness of the light source that illuminates the avatar 3D model is changed, and the color of the avatar image is changed, so that if loud audio is input, it can be reflected in the color of the avatar image.

[0165] In addition, in this embodiment, when a preset high-pitched sound or a preset low-pitched sound is input as the external input sound added to the external video data, i.e., when a specific sound is input, the brightness of the light source that illuminates the avatar 3D model is changed and the color of the avatar image is changed, so that the input of a specific sound can be reflected in the color of the avatar image.

[0166] In addition, in this embodiment, when a sound in a specific range is input as external input sound added to external video data, the brightness of the light source that illuminates the avatar 3D model is changed, and the color of the avatar image is changed, so that the color of the avatar image can reflect the fact that a sound in a specific range has been input.

[0167] In addition, in this embodiment, the brightness of the light source of the light that illuminates the avatar 3D model is changed so that the brightness is set according to whether a preset high-pitched sound or a preset low-pitched sound is input as the external input sound added to the external video data, i.e., according to the type of input sound, and the color of the avatar image is changed to a color that is set according to the type of input sound, so that the color of the avatar image can be changed to a color that is according to the type of input sound.

[0168] In this embodiment, a specific range of sound is identified and the color of the avatar image is changed depending on the type of sound. However, it is also possible to identify the type of sound, such as the genre of background music, and change the color of the avatar image depending on the type of sound identified. Even in such a configuration, the color of the avatar image can be changed to a color that corresponds to the type of sound input.

[0169] Furthermore, in this embodiment, each time a frame of a video is generated, the brightness of the light source that illuminates the avatar 3D model is changed based on the external input audio added to the external video data, and the color of the avatar image is changed, so that a video of an avatar image that reflects the audio based on the external video data for each frame can be generated.

[0170] In addition, each time multiple frames are generated, the brightness of the light source that illuminates the avatar 3D model may be changed based on the external input audio added to the external input image, and the color of the avatar image may be changed.

[0171] In this embodiment, the change of the illumination light based on the color of the external input image, the change of the illumination light based on the amount of movement of the external input image, and the change of the illumination light based on the external input sound are all configured to be performed each time one frame of video is generated, but the change of the illumination light based on the color of the external input image, the change of the illumination light based on the amount of movement of the external input image, and the change of the illumination light based on the external input sound may be configured to be performed at different frequencies. For example, the change of the illumination light based on the color of the external input image may be configured to be performed each time one frame of video is generated, while the change of the illumination light based on the amount of movement of the external input image and the change of the illumination light based on the external input sound may be configured to be performed each time multiple frames are generated.

[0172] Although the second embodiment of the present invention has been described above with reference to the drawings, it goes without saying that the present invention is not limited to this second embodiment, and that modifications and additions that do not deviate from the spirit of the present invention are also included in the present invention. Furthermore, anything that can be applied to the configuration of the first embodiment and its modifications may also be applied to the second embodiment.

[0173] For example, in the above-mentioned second embodiment, an example was described in which facial expression parameters output from a face tracking program are corrected by a parameter correction program, and the corrected facial expression parameters are used by an avatar 3D model generation program to generate an avatar 3D model. However, it is also possible to configure the avatar 3D model generation program to generate an avatar 3D model by directly using parameters output from the face tracking program without using a parameter correction program.

[0174] Furthermore, in the above-mentioned second embodiment, a composite image is generated and output in which an external input image captured by the capture device 112 from external video data is used as a background image, and an avatar image based on an avatar 3D model generated by an avatar 3D model generation program is placed in front of the external input image, and the light illuminating the avatar 3D model is changed based on the color of the external input image, the amount of movement of the external input image, and the external input audio added to the external input image, and the color of the avatar image based on the avatar 3D model is corrected. However, any other configuration may be used as long as the light illuminating the avatar 3D model is changed based on at least one element of the color of the external input image, the amount of movement of the external input image, and the external input audio added to the external input image, and the color of the avatar image based on the avatar 3D model is corrected.

[0175] In the second embodiment, an avatar 3D model generation program generates an avatar 3D model, and a 3D model color correction program separate from the avatar 3D model generation program irradiates the avatar 3D model generated by the avatar 3D model generation program with virtual light based on an external input image to generate an avatar image, thereby changing the color of the avatar image based on the external input image. However, the avatar 3D model generation program may also generate an avatar 3D model, and change the color of the avatar image by irradiating it with virtual light based on an external input image to generate an avatar image. [Explanation of symbols]

[0176] 1 computer terminal 101 processors 102 memory 103 Storage 104 Communication Interface 105 Display device 106 Speaker 107 Input Device 108 Camera 109 Mike 110 Vital Signs Meter 111 Data Bus 112 Capture Device 113 Voice Input Device

Claims

1. a movement detection step of detecting a movement of a performer; an avatar image generating step of generating an avatar image according to the detected movement; a specific image input step of inputting a specific image; a composite image generating step of generating a composite image in which an avatar image is placed in front of the input specific image; an image output step of outputting the composite image; An image generation method comprising: a color change step of changing a color of the avatar image based on the specific image; The color changing step multiplies the avatar image by an average value of colors of the specific image that are equal to or greater than a predetermined brightness. An image generating method comprising:

2. a movement detection step of detecting a movement of a performer; an avatar image generating step of generating an avatar image according to the detected movement; a specific image input step of inputting a specific image; a composite image generating step of generating a composite image in which an avatar image is placed in front of the input specific image; an image output step of outputting the composite image; An image generation method comprising: a color change step of changing a color of the avatar image based on the specific image; In the color changing step, the specific image is multiplied by the avatar image, and the brightness of the color to be multiplied is changed between a central region and an outer edge region of the avatar image. An image generating method comprising:

3. a movement detection step of detecting a movement of a performer; an avatar image generating step of generating an avatar image according to the detected movement; a specific image input step of inputting a specific image; a composite image generating step of generating a composite image in which an avatar image is placed in front of the input specific image; an image output step of outputting the composite image; An image generation method comprising: a color change step of changing a color of the avatar image based on the specific image; the specific image input in the specific image input step is a moving image, further comprising a motion amount detection step of detecting a motion amount of the specific image; In the color changing step, the color of the avatar image is changed based on the amount of movement of the specific image detected in the amount of movement detection step. An image generating method comprising:

4. the composite image is an image that constitutes a moving image, In the color changing step, a color of the avatar image is changed based on the specific image for each frame of the video, and in the composite image generating step, the composite image is generated using the specific image and the color-changed avatar image. The image generating method according to any one of claims 1 to 3.

5. a movement detection step of detecting a movement of a performer; an avatar image generating step of generating an avatar image according to the detected movement; a specific image input step of inputting a specific image; a composite image generating step of generating a composite image in which an avatar image is placed in front of the input specific image; an image output step of outputting the composite image; In an image generation program that causes a computer to execute a color change step of changing a color of the avatar image based on the specific image; The color changing step multiplies the avatar image by an average value of colors of the specific image that are equal to or greater than a predetermined brightness. An image generation program comprising:

6. a movement detection step of detecting a movement of a performer; an avatar image generating step of generating an avatar image according to the detected movement; a specific image input step of inputting a specific image; a composite image generating step of generating a composite image in which an avatar image is placed in front of the input specific image; an image output step of outputting the composite image; In an image generation program that causes a computer to execute a color change step of changing a color of the avatar image based on the specific image; In the color changing step, the specific image is multiplied by the avatar image, and the brightness of the color to be multiplied is changed between a central region and an outer edge region of the avatar image. An image generation program comprising:

7. a movement detection step of detecting a movement of a performer; an avatar image generating step of generating an avatar image according to the detected movement; a specific image input step of inputting a specific image; a composite image generating step of generating a composite image in which an avatar image is placed in front of the input specific image; an image output step of outputting the composite image; In an image generation program that causes a computer to execute a color change step of changing a color of the avatar image based on the specific image; the specific image input in the specific image input step is a moving image, further executing a motion amount detection step of detecting a motion amount of the specific image; In the color changing step, the color of the avatar image is changed based on the amount of movement of the specific image detected in the amount of movement detection step. An image generation program comprising:

Images