Composite image generation system, composite image generation method, and composite image generation program

Abstract

This invention provides a composite video generation system and method for easily creating composite videos by inserting an avatar based on a person's image into a target video generated using multiple video materials. [Solution] The composite video generation system comprises a background video database 1 for storing background videos, a content video database 2 for storing content videos used as video materials, a person video input unit 3 for inputting person videos, a background video selection unit 4 for selecting background videos to be used to generate the composite video, a content video selection unit 5 for selecting content videos to be used to generate the composite video, an insertion target video generation unit 7 for generating insertion target videos consisting of background videos and content videos, an avatar generation unit 8 for generating avatars related to a person based on the person video, an avatar insertion unit 9 for inserting the generated avatars into the insertion target videos to generate the composite video, and a composite video output unit 10 for outputting the composite video.

Description

Technical Field

[0001] The present invention relates to a technique for generating a composite video by inserting an avatar generated based on a person video into an insertion target video composed of one or more video materials.

Background Art

[0002] In recent years, with the improvement of processing capabilities in electronic computers such as computers, many computer graphics of human figures, so-called avatars, that reflect the characteristics of real people have been utilized. For example, an avatar is used as one's own icon when using SNS (Social Networking Service), or one's own avatar is used as the character of the protagonist in an online game or the like. In addition, in contents such as still images and moving images, a service has been proposed in which some of the characters appearing are replaced with one's own avatar for viewing and the like.

[0003] By using an avatar that reflects one's own characteristics in this way, for example, when a protagonist consisting of an avatar expressing the characteristics of a user in a game battles against an enemy character, an effect of improving the user's sense of immersion in the game world occurs. By using an avatar that abstractly represents the user himself / herself as an icon indicating the user in SNS, it is expected that an effect such as promoting communication between users in a virtual space in the same sense as the real world will occur.

[0004] Both Patent Documents 1 and 2 disclose techniques for using avatars that respectively imitate the actual appearance of the player himself / herself or a co-player in a computer game that expresses a virtual space using a head-mounted display.

Prior Art Documents

Patent Documents

[0005]

Patent Document 1

[0006] However, since the appearances of real people vary greatly, using an avatar that reflects the characteristics of a real person requires creating an avatar individually, which presents a problem due to the complexity of avatar generation. Furthermore, if existing still images or videos are insufficient as the target video for inserting one's avatar, it becomes necessary to prepare original still images or videos. However, for ordinary users who do not possess specialized skills in video generation, generating still images or videos to insert an avatar into is not easy. Moreover, even if one possesses the technology to generate original still images or videos using existing videos, legal issues may arise depending on the terms of use of the existing videos. Despite these problems, neither Patent Documents 1 nor 2 disclose any technology to solve these problems.

[0007] The present invention has been made in view of the above problems, and aims to provide a technology for easily generating a composite video, which is generated by inserting an avatar based on a person's image into a target video generated using multiple video materials. [Means for solving the problem]

[0008] To achieve the above objective, the composite video generation system according to claim 1 is a composite video generation system that generates a composite video by inserting an avatar generated based on a predetermined person video into a part of a video to be inserted, and is characterized by comprising: background video selection means for selecting a background video which is a video material relating to the background of the video to be inserted; content video selection means for selecting a content video which is a video material that is arranged separately from the background video in the video to be inserted; content video processing means for performing necessary processing on the content video selected by the content video selection means according to the arrangement in the video to be inserted; video to be inserted generation means for generating the video to be inserted based on the background video selected by the background video selection means and the content video selected by the content video selection means; avatar generation means for generating the avatar based on the person video; avatar insertion means for inserting the avatar generated by the avatar generation means into all or part of the content video in the video to be inserted and / or part of the background video in the video to be inserted; and composite video output means for outputting a composite video generated by inserting the avatar into the video to be inserted by the avatar insertion means.

[0009] Furthermore, in order to achieve the above objective, the composite video generation system according to claim 2, in the above invention, the background video selection means comprises a first usage mode information generation means that generates first usage mode information which is information relating to the usage mode of the generated composite video and / or the usage mode of the background video in the composite video, and a first usability determination means that determines whether the usage mode of the background video indicated by the first usage mode information satisfies predetermined usage conditions for the background video, the content video selection means comprises a second usage mode information generation means that generates second usage mode information which is information relating to the usage mode of the content video in the video to be inserted, and a second usability determination means that determines whether the usage mode of the content video indicated by the second usage mode information satisfies predetermined usage conditions for the content video, and the video to be inserted generation means generates the video to be inserted using the background video which the first usability determination means has determined satisfies the usage conditions and the content video which the second usability determination means has determined satisfies the usage conditions.

[0010] Furthermore, in order to achieve the above objective, the composite video generation system according to claim 3, in the above invention, the background video selection means comprises: a first usage mode information generation means that generates first usage mode information which is information relating to the usage mode of the composite video including the background video and / or the usage mode of the background video in the composite video; a first candidate video extraction means that extracts one or more candidate background videos from one or more background videos for which usage conditions that allow use in the usage mode indicated by the first usage mode information are set; and a first video determination means that determines the background video to be used in the composite video from among the one or more candidate background videos extracted by the first candidate video extraction means, and the content video selection means is the usage of the composite video including the content video The insertion target video generation means comprises: a second usage mode information generation means for generating second usage mode information which is information relating to the manner and / or manner of use of the content video in the composite video; a second candidate video extraction means for extracting one or more candidate content videos from one or more of the content videos for which usage conditions that permit use in the manner of use indicated in the second usage mode information are set; and a second video determination means for determining the content video to be used in the composite video from the one or more candidate content videos extracted by the second candidate video extraction means, wherein the insertion target video generation means generates the insertion target video using the background video determined by the first video determination means and the content video determined by the second video determination means.

[0011] Furthermore, in order to achieve the above objective, the composite video generation method according to claim 4 is a composite video generation method that generates a composite video by inserting an avatar generated based on a predetermined person video into a part of the video to be inserted, and is characterized by including: a video material selection step of selecting one or more video materials to form the video to be inserted; a video material processing step of performing necessary processing on the video materials selected in the video material selection step according to their arrangement in the video to be inserted; an insertion target video generation step of generating the video to be inserted based on the video materials selected in the video material selection step; an avatar generation step of generating the avatar based on the person video; an avatar insertion step of inserting the avatar generated in the avatar generation step into all or part of the video materials in the video to be inserted; and a composite video output step of outputting the composite video generated by inserting the avatar into the video to be inserted in the avatar insertion step.

[0012] Furthermore, in order to achieve the above objective, the composite video generation method according to claim 5 is characterized in that, in the above invention, the video material selection step further includes a usage pattern information generation step which generates usage pattern information which is information relating to the usage pattern of the generated composite video and / or the usage pattern of the video material in the composite video, and a usability determination step which determines whether the usage pattern of the video material indicated by the usage pattern information satisfies predetermined usage conditions for the video material, and the insertion target video generation step generates the insertion target video using the video material which the usability determination step has determined satisfies the usage conditions.

[0013] Furthermore, in order to achieve the above objective, the composite video generation program according to claim 6 is a composite video generation program that causes a computer to generate a composite video by inserting an avatar generated based on a predetermined person image into a part of the video to be inserted, and is characterized in that it causes the computer to execute: a video material selection function that, when determining one or more video materials to form the video to be inserted, selects video materials that have usage conditions that allow the manner in which the composite video including the video materials is used and / or the manner in which the video materials are used in the composite video; a video material processing function that performs necessary processing on the one or more video materials selected by the video material selection function according to the arrangement in the video to be inserted; an insertion target video generation function that generates the video to be inserted based on the video materials selected by the video material selection function; an avatar generation function that generates the avatar based on the person image; an avatar insertion function that inserts the avatar generated by the avatar generation function into all or part of the video material in the video to be inserted; and a composite video output function that outputs the composite video generated by inserting the avatar into the video to be inserted by the avatar insertion function. [Effects of the Invention]

[0014] According to the present invention, the generation of a composite video, which is created by inserting an avatar based on a person's image into a target video generated using multiple video materials, is made easier. [Brief explanation of the drawing]

[0015] [Figure 1] This is a schematic diagram showing the configuration of the composite video generation system according to Embodiment 1. [Figure 2] This is a schematic diagram showing the configuration of the composite image generation system according to Embodiment 2. [Modes for carrying out the invention]

[0016] The embodiments of the present invention will be described in detail below with reference to the drawings. The following embodiments describe the most appropriate examples of the present invention, and naturally, the content of the present invention should not be limited to the specific examples shown in these embodiments. It goes without saying that any configuration other than the specific configurations shown in the embodiments that produces similar functions and effects is also included in the technical scope of the present invention.

[0017] (Embodiment 1) First, the composite video generation system according to Embodiment 1 will be described. As shown in Figure 1, the composite video generation system according to Embodiment 1 includes a background video database 1 that stores background videos, which are video materials used as background videos in the videos to be inserted that constitute the composite video; a content video database 2 that stores content videos, which are video materials that are placed together with the background videos in the videos to be inserted that constitute the composite video; a person video input unit 3 that inputs person videos that will be the basis for avatars to be inserted into the composite video; a background video selection unit 4 for selecting background videos to be used to generate the composite video from the videos stored in the background video database 1; a content video selection unit 5 for selecting content videos to be used to generate the composite video from the videos stored in the content video database 2; a content video processing unit 6 that performs necessary processing such as enlarging and reducing on the selected content videos according to their placement in the videos to be inserted that constitute the composite video; an insertion target video generation unit 7 that generates an insertion target video consisting of the selected and processed background videos and content videos; and a person video input via the person video input unit 3 that processes the person... Avatar generation unit 8 generates avatars related to the background video, avatar insertion unit 9 inserts the generated avatars into the target video to generate a composite video, composite video output unit 10 outputs the generated composite video, background token generation unit 11 generates background tokens which are non-fungible tokens that have a one-to-one correspondence with each background video, content token generation unit 12 generates content tokens which are non-fungible tokens that have a one-to-one correspondence with each content video, avatar token generation unit 13 generates avatar tokens which are non-fungible tokens that have a one-to-one correspondence with avatars, video token generation unit 14 which are non-fungible tokens that have a one-to-one correspondence with the generated composite video, background transaction generation unit 15 generates background transactions which are information stored in blocks in a distributed ledger (described later) that have a one-to-one correspondence with each background token, and content transaction generation unit 16 generates content transactions which are information stored in blocks in a distributed ledger that have a one-to-one correspondence with each content token.The system comprises: an avatar transaction generation unit 17 that generates avatar transactions, which are information stored in blocks in a distributed ledger that have a one-to-one correspondence with individual avatar tokens; a video transaction generation unit 18 that generates video transactions, which are information stored in blocks in a distributed ledger that have a one-to-one correspondence with individual synthesized videos; an electronic signature generation unit 19 that generates electronic signatures, which are data that proves that the generation and output of the information contained in each transaction conforms to the intentions of the avatar token holder; and an output unit 20 that outputs each transaction and the electronic signature corresponding to each transaction to the corresponding distributed ledger.

[0018] Background video database 1 is one of the video materials used to generate the composite video, and is for storing background videos used as background videos in the videos to be inserted that constitute the composite video. Specifically, the background database has the function of storing data on multiple background videos in a manner in which background videos and identification information, which is information for identifying the background video, are associated with each other. In this embodiment 1, each background video, background token, and background transaction (and furthermore, the distributed ledger in which the background transaction is stored) has a configuration in which the correspondence between each other is defined using the identification information of the background video. In addition, information regarding the usage conditions and usage history of each background video is generated by the background transaction generation unit 15, which will be described later, and this information is recorded in the distributed ledger corresponding to the background video.

[0019] The content video database 2 is one of the video materials used to generate the composite video, and is for storing content videos that are each component placed together with the background video in the insertion target video that constitutes the composite video, such as artificial objects such as individual buildings that make up a landscape, natural objects such as rocks and plants, and living things such as animals and people. Specifically, the content video database 2 has the function of storing data related to multiple content videos in a manner in which the content video and identification information, which is information for identifying the content video, are related to each other. In this embodiment 1, each content video, content token and content transaction (and furthermore, the distributed ledger in which the content transaction is stored) have a configuration in which the correspondence between them is defined using the identification information of the content video. In addition, information regarding the usage conditions and usage history of each content video is generated by the content transaction generation unit 16, which will be described later, and this information is recorded in the distributed ledger corresponding to the content video.

[0020] Note that both the background video and the content video function as video materials that constitute the composite video. Although there is a difference in that the former is displayed as a background for the entire composite video, while the latter is displayed so as to be superimposed on the former in a partial area of the composite video, essentially, they have the same identity as video materials that constitute the composite video. Therefore, in the first embodiment, it is possible to configure the video to be inserted before avatar insertion (as will be described later, a composite video is generated by inserting an avatar into the video to be inserted) with only the background video or only the content video, or to uniformly handle them as video materials without distinguishing between the background video and the content video, and it is also possible to use a video database that combines the background video database 1 and the content video database 2 (this also applies to other components such as the background token generation unit 11 and the content token generation unit 12). Further, the specific configuration of the background video and the content video may be either a 2D video or a 3D video, and there is no problem whether it is a still image or a moving image. Furthermore, in the content video, similar to the case of the person video described later, it may be configured in the form of an avatar.

[0021] The person video input unit 3 is for inputting a person video, which is a material used for generating an avatar that constitutes a composite video. Specifically, the person video input unit 3 has a function of inputting a person video, which is a video related to the whole or a part of the person who is the insertion target for the video to be inserted, and may be configured to input a video from the outside, or may be configured to include an imaging mechanism for acquiring a video. The specific configuration of the person video may be a full-body video of the target person or a partial video such as a face image. Also, it may be either a still image or a moving image, and may be either 2D or 3D. In the first embodiment, an explanation is given using a 2D and still image video related to the face of the target person as the person video, but it is of course possible to use a person video other than such a configuration in the composite video generation system according to the first embodiment.

[0022] The background video selection unit 4 is for selecting a background video to be used for generating a composite video. Specifically, the background video selection unit 4 includes a video identification unit 21 for identifying the background video to be used, a usage mode information generation unit 22 for generating usage mode information which is information regarding the usage mode of the identified background video in the composite video, a usage condition information acquisition unit 23 for acquiring usage condition information which is information regarding the content of the usage conditions of the background video identified by the video identification unit 21, a usage determination unit 24 for determining whether the usage mode of the background video included in the usage mode information generated by the usage mode information generation unit 22 satisfies the usage conditions included in the usage condition information acquired by the usage condition information acquisition unit 23, and a video extraction unit 25 for extracting the target background video from the background video database 1 when it is determined by the usage determination unit 24 that the usage conditions are satisfied.

[0023] The video identification unit 21 is for identifying the background video to be the selection target in the background video selection unit 4. Specifically, the video identification unit 21 has a function of searching the background video database for the background video to be selected and acquiring the identification information of the searched background video. Using the identification information of the background video acquired by the video identification unit 21, information acquisition processing and the like by the usage condition information acquisition unit 23 are performed.

[0024] The usage information generation unit 22 generates usage information (corresponding to the first usage information in the claims) for the background video to be selected, which is information regarding the usage of the background video in relation to the composite video to be generated, and corresponds to the first usage information generation means in the claims. The content of the usage information includes, but is not limited to, one or more pieces of information from the following: the usage of the composite video generated using the background video as video material (whether or not the composite video is made public, the scope of public disclosure, the purpose (video distribution, output to paper media such as glossy paper or postcards of still images, etc.), whether or not it is for commercial use at the time of use, the size of the composite video to be used, etc.), the usage of the background video in the composite video (the content of the processing if processing is performed when using it in the composite video, the relationship with the content video used if content video is also used in the composite video, etc.), attribute information of the person who generates and / or uses the composite video (gender, age, nationality, beliefs, identification information, eligibility to use this system (e.g., paid user or unpaid user), etc.), and the costs that can be borne regarding the use of the background video. The method of generating usage information by the usage information generation unit 22 may be to input character information corresponding to each piece of information, or it may be to directly generate the information by performing image analysis processing on the specific data of the composite video to be generated. The usage information is output to the usability determination unit 24 and used by the usability determination unit 24 as material for determining whether the background video can be used.

[0025] The usage conditions information acquisition unit 23 is for acquiring usage conditions information, which is information regarding the usage conditions of the background video identified by the video identification unit 21. Specifically, the usage conditions information acquisition unit 23 has the function of accessing a distributed ledger that corresponds one-to-one with the background video, and acquiring usage conditions information, which is information regarding the usage conditions of the background video, from the background transactions stored in the blocks of the distributed ledger. The usage conditions information may include, but is not limited to, one or more of the following: usage conditions for composite videos that use background video as video material (how far the scope of publication of the composite video is permitted, how far the uses of the composite video are permitted (video distribution, output to paper media such as glossy paper or postcards as still images, etc.), how far commercial use is permitted, etc.); usage conditions for background video in composite videos (how far processing is permitted when processing is performed, which content videos from the same composite video are permitted to be used, etc.); the range of attributes that may be permitted for a person who generates and / or uses composite videos using background video (gender, age, nationality, beliefs, identification information, eligibility to use this system (e.g., paid user or unpaid user), etc.); and usage fees for background video. The usage conditions information acquired by the usage conditions information acquisition unit 23 is output to the usability determination unit 24 and used as material for determining whether the background video can be used by the usability determination unit 24.

[0026] The Usability Determination Unit 24 determines whether or not a background video can be used by comparing the usage pattern information and usage condition information related to the background video identified by the Video Identification Unit 21, and corresponds to the first Usability Determination Means in the claims. Specifically, the Usability Determination Unit 24 extracts information from the usage pattern information items (for example, usage patterns of the composite video to be generated) that correspond to each item related to usage conditions included in the usage condition information (for example, usage conditions of the composite video to be generated), and determines whether or not the information in the usage pattern information items satisfies the conditions specified in the usage condition information items. If the content of the usage pattern information for all items is within the acceptable range of the usage condition information, the Usability Determination Unit 24 determines that the background video can be used, and otherwise determines that the background video cannot be used.

[0027] The video extraction unit 25 is for extracting image data of background videos that have been determined to be usable by the usability determination unit 24. Specifically, when the usability determination unit 24 determines that a background video is usable, the video extraction unit 25 has the function of extracting the background video corresponding to the identification information acquired by the video identification unit 21 from the background video database 1 and outputting it to the video generation unit 7.

[0028] The content video selection unit 5 is for selecting content videos to be used to generate a composite video. Specifically, the content video selection unit 5 includes a video identification unit 26 for identifying the content video to be used, a usage mode information generation unit 27 for generating usage mode information, which is information regarding the usage mode of the identified content video in the composite video, a usage mode information acquisition unit 28 for acquiring usage mode information, which is information regarding the content of the usage conditions for the content video identified by the video identification unit 26, a usability determination unit 29 for determining whether the usage mode of the content video included in the usage mode information generated by the usage mode information generation unit 27 satisfies the usage conditions included in the usage mode information acquired by the usage mode information acquisition unit 28, and a video extraction unit 30 for extracting the target content video from the content video database 2 if the usability determination unit 29 determines that the usage conditions are met.

[0029] The video identification unit 26 is for identifying the content video to be selected by the content video selection unit 5. Specifically, the video identification unit 26 has the function of searching the content video database 2 for the content video to be selected and obtaining identification information of the retrieved content video. Using the identification information of the content video obtained by the video identification unit 26, the usage condition information acquisition unit 28 performs information acquisition processing, etc. Note that the function of the video identification unit 26 is basically the same as the function of the video identification unit 21, and both may be formed by a single component to perform processing related to both background video and content video.

[0030] The usage information generation unit 27 generates usage information (corresponding to the second usage information in the claims) for the selected content video, which is information regarding the usage of the content video in relation to the composite video to be generated, and corresponds to the second usage information generation means in the claims. The content of the usage information includes, but is not limited to, one or more pieces of information from the following: the usage of the composite image generated using the content video (whether or not the composite video is made public, the scope of public disclosure, the purpose (video distribution, output of still images to paper media such as glossy paper or postcards, etc.), whether or not it is for commercial use at the time of use, the size of the composite video to be used, etc.), the usage of the content video in the composite video (position and size of the content video in the composite video, relationship with other content videos and background videos, processing details if the content video is processed, etc.), attribute information of the person who generates and / or uses the composite video (gender, age, nationality, beliefs, identification information, eligibility to use this system (e.g., paid user or unpaid user), etc.), and the costs that can be borne regarding the use of the content video. The usage information generation method of the usage information generation unit 27 may be to input character information corresponding to each piece of information, or it may be to directly acquire information by performing image analysis processing on the specific data of the composite video to be generated. The usage information generated by the usage information generation unit 27 is output to the usability determination unit 29 and used by the usability determination unit 29 as material for determining whether the content video can be used. The function of the usage information generation unit 27 is basically the same as that of the usage information generation unit 22, and both may be formed by a single component to generate usage information for both the background video and the content video.

[0031] The usage conditions information acquisition unit 28 is for acquiring usage conditions information, which is information regarding the usage conditions of the content video identified by the video identification unit 26. Specifically, the usage conditions information acquisition unit 28 has the function of accessing a distributed ledger that corresponds one-to-one with the target content video, and acquiring usage conditions information, which is information regarding the usage conditions of the content video, from the content transactions stored in the blocks of the distributed ledger. The usage conditions information includes, for example, one or more pieces of information such as: usage conditions for composite videos that use content video as video material (to what extent is the scope of public release of the composite video permitted, to what extent is the use of the composite video permitted (video distribution, output to paper media such as glossy paper or postcards as still images, etc.), to what extent is commercial use permitted, etc.); usage conditions for content video in composite videos (to what extent is processing permitted when processing is performed, which background videos and content videos placed in the same composite video are permitted to be used for the content video, etc.); the range of attributes that may be permitted for a person who generates and / or uses a composite video using content video (gender, age, nationality, beliefs, identification information, eligibility to use this system (e.g., paid user or unpaid user), etc.); and usage fees for content video. The usage conditions information acquired by the usage conditions information acquisition unit 28 is output to the usability determination unit 29 and used as material for determining whether the content video can be used by the usability determination unit 29. The function of the usage conditions information acquisition unit 28 is basically the same as that of the usage conditions information acquisition unit 23, and both may be formed by a single component, and the information may be acquired by accessing a distributed ledger that corresponds to the background video and content video, respectively.

[0032] The Usability Determination Unit 29 determines whether or not a content video can be used by comparing the usage pattern information and usage condition information related to the content video identified by the Video Identification Unit 26, and corresponds to the second Usability Determination Means in the claims. Specifically, the Usability Determination Unit 29 extracts information from the usage pattern information items (for example, the usage pattern of the composite video to be generated) that correspond to each item included in the usage condition information (for example, the usage conditions of the composite video to be generated), and determines whether or not the information in the usage pattern information items satisfies the conditions specified in the usage condition information items. If the content of the usage pattern information for all items is within the acceptable range of the usage condition information, the Usability Determination Unit 29 determines that the content video can be used, and otherwise determines that the content video cannot be used. The function of the Usability Determination Unit 29 is basically the same as that of the Usability Determination Unit 24, and both may be formed by a single component to determine the usability of background video and content video intended for use in composite video.

[0033] The video extraction unit 30 is for extracting image data of content video that has been determined to be usable by the usability determination unit 29. Specifically, when the usability determination unit 29 determines that it is usable, the video extraction unit 30 has the function of extracting background video corresponding to the identification information acquired by the video identification unit 26 from the content video database 2 and outputting it to the insertion target video generation unit 7.

[0034] The content video processing unit 6 is for processing content videos used as video material for generating composite videos, and is necessary when generating the video to be inserted (composite video before avatar insertion). While it is possible to incorporate the content video directly into the video to be inserted (and even the composite video) as stored in the content video database 2, it is usually necessary to process the content video by scaling it up or down, changing its orientation, fine-tuning its shape, or changing its movement, depending on its position within the composite video, its relationship with other content videos, and the arrangement of the content video within the video to be inserted (position, posture, direction, specific shape, etc.), which is determined based on the concept of the composite video. The content video processing unit 6 is for processing content videos in such cases. For example, if the content video is of a specific person, it can scale it up or down to balance it with other people, or change a static, upright image to one with arms outstretched to match the concept of the composite video. Furthermore, the processing performed by the content video processing unit 6 can be carried out using previously known techniques. For example, if the content video is formed by an avatar, processing will be performed to change the posture, etc., by changing the positional relationship of the bones based on the skeletal information. In this embodiment 1, the content video processing unit 6 is based on the premise that it performs processing on the content video, but it may also be provided with a function to perform necessary processing on the background video, or a separate background video processing unit may be newly provided for processing the background video, separate from the content video processing unit 6. In addition, in this embodiment 1, the content video processing unit 6 performs processing on the content video after the selection process by the content video selection unit 5 has been completed, that is, the content video after it has been determined to be usable by the usability determination unit 29, but it may also be configured to perform processing before the determination. After performing processing on the content video before the determination, the actual processing content may be incorporated as part of the usage information generated by the usage information generation unit 27, and the determination process in the usability determination unit 29 may be performed including the processing content.

[0035] The Insertion Target Video Generation Unit 7 generates an insertion target video by combining the background video selected by the background video selection unit 4 and the content video selected by the content video selection unit 5. The insertion target video is the video that will be used as the basis for the composite video, and more specifically, the composite video is the insertion target video to which avatar insertion processing has been applied. The Insertion Target Video Generation Unit 7 generates the insertion target video by arranging the background video and the content video, which has undergone the necessary processing by the content video processing unit 6, in predetermined positions. As a specific example of the Insertion Target Video Generation Unit 7, it is possible to have a configuration that has image processing functions using conventional technology, and there are no particular restrictions on the configuration of the generated insertion target video, such as video / still images, color video / black and white video, 2D video / 3D video, etc., according to the form of the finished video.

[0036] The avatar generation unit 8 is for generating an avatar to replace all or part of the content video in the video to be inserted, based on a person video input via the person video input unit 3. Specifically, the avatar generation unit 8 may consist of skeletal information (bones), surface information (skin), and weight information defining the relationship between the two, but it is also preferable to have a configuration consisting only of surface information including three-dimensional shape and information on the color tone on the surface, and in an even simpler configuration, it may consist of the image data itself. Furthermore, the avatar generated by the avatar generation unit 8 may be an avatar corresponding to the full body image of a person, but it may also be an avatar consisting only of a part of a person, for example, only a part of the face. In this embodiment 1, an avatar consisting only of the head from the neck up is generated. The avatar generation unit 4 extracts feature points based on the human video input via the human video input unit 3, according to the position and shape of the surface features (eyes, eyebrows, nose, mouth, ears, hairstyle, etc.) and internal features (joints, etc.) of the human video. By reflecting the positional relationships between the extracted feature points onto the avatar, it generates a realistic avatar that reflects the physical characteristics of the model. However, if a simpler configuration is adopted, for example, the avatar may be generated from the human video as is, or from the human video with only the minimum necessary modifications such as adjustments to size and orientation.

[0037] The avatar insertion unit 9 is for generating a composite image by inserting the generated avatar into the target video. Specifically, the avatar insertion unit 9 has the function of inserting the avatar generated by the avatar generation unit 8 into at least one of the content video area and / or a part of the background video area in the target video, thereby completing the composite image. If the target video is a still image, the avatar insertion unit 9 acquires information regarding the size and direction of the replacement target in the target video, and appropriately changes the size and direction of the avatar based on that information, and then inserts the avatar into the target video in a manner that replaces the replacement target. If the target video is a video, the avatar insertion unit 9 acquires information regarding the operation of the replacement target in the target video, for example by detecting changes in the position of feature points, and appropriately changes the size, direction and operation of the avatar based on that information, and then inserts the avatar into the target video in a manner that replaces the replacement target. While it is generally preferable to use avatars to replace content footage related to people within the video being inserted, this is not limited to that. For example, if content footage of a locomotive is present, an avatar consisting of a portion of a face image could be inserted in front of the locomotive. Furthermore, the replacement target is not limited to content footage. For example, a composite image could be generated by replacing a portion of the background footage with an avatar.

[0038] The composite video output unit 10 is for outputting a composite video generated by inserting an avatar into the target video by the avatar insertion unit 9. The composite video output method by the composite video output unit 10 may be to output the generated composite video as video data as is, to output it after converting the data format, to output it after printing it on glossy paper to make it like a photograph, or to output it printed on a postcard.

[0039] The background token generation unit 11 is for generating non-fungible tokens that have a one-to-one correspondence with each individual background image. A "non-fungible token" is a so-called NFT (Non-Fungible Token), that is, a token that has the property of being infungible with other tokens by possessing unique data, and is issued based on the Ethereum® standard ERC721, for example. The non-fungible token in this embodiment 1 is issued based on ERC721 or other predetermined standards, and is configured so that the transaction history of the object, information about the owner, and information such as the establishment, transfer, and modification of rights such as the right to use the object are recorded in a distributed ledger on the blockchain that corresponds to the non-fungible token. The information to be recorded regarding the background token is generated by the background transaction generation unit 15, which will be described later, and is output to the distributed ledger by the output unit 20 along with the electronic signature generated by the electronic signature generation unit 19.

[0040] Blockchain is a technology that synchronizes data between multiple computers constituting a decentralized network using cryptographic techniques. Specifically, each block is composed of a collection of token information, such as agreed-upon transaction records, and information for connecting to other blocks (information from the previous block). A blockchain is formed by linking multiple such blocks. Even if data is tampered with on some of the computers, the correct data is selected by majority vote among the other computers, making it extremely difficult to destroy or tamper with the data. Blockchains can be classified into public blockchains, which do not restrict who can participate in the majority vote, meaning that an unspecified number of people can participate in the recording process on the distributed ledger; and consortium blockchains and private blockchains, where only a select number of specific individuals can participate in the majority vote, meaning that only predetermined individuals can participate in the recording process on the distributed ledger. The former is a concept included in the open distributed ledger in this invention, while the latter is a concept included in the closed distributed ledger in this invention.

[0041] Specific methods for linking background tokens and background images include associating the identifier of the background token with the identification information of the background image. More simply, the identifier of the background token may be matched with the identification information of the background image. Furthermore, the generation of non-fungible tokens may be performed by the background token generation unit 11 itself, or by an external system generating the tokens by issuing a predetermined command to an external system directly or indirectly connected to the background token generation unit 11. In addition, the specific format of the non-fungible token is not limited to that conforming to the Ethereum® standard ERC721, but may be any format as long as it has the property of being non-fungible and information such as transaction history can be stored in an open distributed ledger or a closed distributed ledger.

[0042] The content token generation unit 12 is for generating content tokens, which are non-fungible tokens that have a one-to-one correspondence with each individual content video. The content tokens generated by the content token generation unit 12 are similar to the background tokens generated by the background token generation unit 11, except that the target of the correspondence is the content video rather than the background video, and the tokens are generated by so-called NFTs. Therefore, the content token generation unit 12 may be formed integrally with the background token generation unit 11, and both background tokens and content tokens may be generated with a single component. The correspondence between content tokens and content videos is set in a format in which the identifier of the content token and the identification information of the content video are mutually associated.

[0043] The avatar token generation unit 13 is for generating avatar tokens, which are non-fungible tokens that have a one-to-one correspondence with each avatar. The avatar tokens generated by the avatar token generation unit 13 are similar to the background tokens generated by the background token generation unit 11 and the content tokens generated by the content token generation unit 12, except that the object with which the correspondence is established is an avatar, not a background video or content video, and the tokens are generated by so-called NFTs. Therefore, the avatar token generation unit 13 may be formed integrally with the background token generation unit 11 and the content token generation unit 12, and background tokens, content tokens, and avatar tokens may be generated with a single component. When setting the correspondence between avatar tokens and avatars, it is preferable to set identification information for the avatar and then set it in a form in which the identification information and the identifier of the avatar token are mutually associated.

[0044] The video token generation unit 14 is for generating video tokens, which are non-fungible tokens that have a one-to-one correspondence with the generated composite video. The video tokens generated by the video token generation unit 14 are similar to other tokens in that the tokens are generated by so-called NFTs, except that the object with which the correspondence is established is the composite video. For this reason, the video token generation unit 14 may be formed integrally with the background token generation unit 11, etc., and each token, including the video token, may be generated by a single component. When setting the correspondence between the video token and the composite video, it is preferable to set identification information in the composite video and then associate that identification information with the identifier of the video token.

[0045] The background transaction generation unit 15 is for generating background transactions, which are information stored in a distributed ledger that corresponds to a background token. The background transaction generation unit 15 has the function of generating a background transaction that includes information about the holder of the background video, usage condition information for the corresponding background video, and usage pattern information when the background video is used in a composite video (i.e., when its use is permitted by the usage approval unit 24), and outputting it to the electronic signature generation unit 19 and the output unit 20. Regarding the generation of usage condition information, it is preferable to generate it based on regulations predetermined by laws and regulations (for example, prohibiting the use of stimulating videos in environments where minors can view them), as well as based on usage conditions set by the creator or holder of the background video. Furthermore, the background transaction generation unit 15 has the function of generating a new background transaction that includes this information when the holder changes, when the content of the usage condition information is updated due to a change in usage conditions, or when usage pattern information is generated when a composite video using the background video is newly created, and outputting it to the electronic signature generation unit 19 and the output unit 20.

[0046] The content transaction generation unit 16 is for generating content transactions, which are information stored in a distributed ledger that corresponds to a content token. The content transaction generation unit 16 has the function of generating a content transaction that includes information about the owner of the content video, usage condition information for the corresponding content video, and usage pattern information when the content video is used in a composite video (i.e., when its use is permitted by the usage approval unit 29), and outputting it to the electronic signature generation unit 19 and the output unit 20. Regarding the generation of usage condition information, it is preferable to generate it based on regulations predetermined by laws and regulations (for example, prohibiting the use of stimulating videos in environments where minors can view them), as well as based on usage conditions set by the creator or owner of the content video. Furthermore, the content transaction generation unit 16 has the function of generating a new content transaction that includes this information when the owner changes, when the content of the usage condition information is updated due to a change in usage conditions, or when usage pattern information is generated when a composite video using the content video is newly created, and outputting it to the electronic signature generation unit 19 and the output unit 20.

[0047] The avatar transaction generation unit 17 is for generating avatar transactions, which are information stored in a distributed ledger that corresponds to an avatar token. The avatar transaction generation unit 17 has the function of generating an avatar transaction that includes information about the avatar owner, information about the usage conditions of the corresponding avatar, and information about the content video and / or background video that was inserted when the avatar was used in a composite video, and outputting it to the electronic signature generation unit 19 and the output unit 20. Regarding the generation of usage conditions information, it is preferable to generate it based on regulations predetermined by laws and regulations, as well as based on usage conditions set by the avatar owner. Furthermore, the avatar transaction generation unit 17 has the function of generating a new content transaction that includes the information when the avatar owner is changed, when the content of the usage conditions information is updated due to a change in usage conditions, or when a new composite video using the avatar is generated and a new insertion target is set, and outputting it to the electronic signature generation unit 19 and the output unit 20.

[0048] The video transaction generation unit 18 is for generating video transactions, which are information stored in a distributed ledger that corresponds to a video token. The video transaction generation unit 18 has the function of generating a video transaction that includes information about the holder of the corresponding composite video, as well as identification information of the background video, content video, and avatar that constitute the composite video, and usage information regarding the background video and content video, and outputting it to the electronic signature generation unit 19 and the output unit 20.

[0049] The electronic signature generation unit 19 is for generating electronic signatures, which are data that proves that the information contained in each of the background transaction, content transaction, avatar transaction, and video transaction is genuine. Specifically, the electronic signature generation unit 19 has the function of generating a hash value for each of the background transaction, content transaction, avatar transaction, and video transaction, and then encrypting the hash value with a private key to generate an electronic signature. The "hash value" is a fixed-length value obtained by applying a certain calculation procedure to the original data. Since this calculation procedure is irreversible, it is considered impossible to recover the original data from the hash value. The "private key" is a sequence of numbers used in the encryption process and is configured to be decryptable with a corresponding "public key". Normally, the "private key" is managed by the holders of the background token, content token, avatar token, and video token, so the electronic signature generation unit 19 may be provided with a separate electronic signature generation unit corresponding to each token.

[0050] The output unit 20 outputs the background transactions, content transactions, avatar transactions, and video transactions generated by the background transaction generation unit 15, content transaction generation unit 16, and avatar transaction generation unit 17, along with their corresponding digital signatures, to a distributed ledger associated with each of the background tokens, content tokens, avatar tokens, and video tokens. The output unit 20 functions as one aspect of the output means within the claims, and in this embodiment, it is configured to directly output each transaction and its corresponding digital signature, but it may also be configured to simply instruct other components (including those provided outside the system) to output. In this embodiment 1, the output unit 20 is directly or indirectly connected to a network where the distributed ledger is installed, and is configured to output predetermined data to the distributed ledger. For the data output from the output unit 20, a hash value is generated for each transaction. This hash value is then compared with the data obtained by decrypting the digital signature corresponding to each transaction using the public key. If the two values ​​are different, the transaction is determined not to be legitimate and registration to the distributed ledger is rejected. If the two values ​​match, the transaction is determined to be legitimate, and the information composed of the transaction is stored in a block that constitutes the distributed ledger. The correspondence between each transaction and the distributed ledger can be established, for example, by linking the identification information of the distributed ledger with the identification information of each transaction. More simply, the same information as the identification information of each transaction may be used as the identification information of the distributed ledger. The identification information of the distributed ledger is used when outputting information by the output unit 20, as well as when acquiring information in the usage condition information acquisition units 23 and 28.

[0051] Next, the advantages of the composite video generation system according to this embodiment 1 will be described. First, in generating the video to be inserted, which is the basis for the composite video, the composite video generation system according to this embodiment 1 does not create the background video and content video used as video materials each time, but rather selects them as appropriate from those stored in a database in advance, making it possible to generate the video to be inserted quickly and easily. Furthermore, the composite video generation system according to this embodiment 1 has the advantage that, by adopting a configuration in which usage condition information, which is information about the usage conditions set in advance for the video materials stored in the database, it is possible to easily and quickly determine whether there are any problems regarding the specific way in which the video materials are used in the composite video and the specific way in which the video materials are used in the composite video, before the composite video is generated. Moreover, the composite video generation system according to this embodiment 1 has a configuration in which usage condition information and usage condition information regarding video materials are stored in corresponding distributed ledgers. With this configuration, it has the advantage that accurate and genuine information can be used for both the usage condition information and usage condition information, which is not subject to tampering by third parties.

[0052] (Embodiment 2) Next, a composite image generation system according to Embodiment 2 will be described. In Embodiment 2, components that have the same name and reference numerals as those in Embodiment 1 will perform the same functions as those in Embodiment 1 unless otherwise specified.

[0053] Figure 2 is a schematic diagram showing the configuration of the composite video generation system according to Embodiment 2. As shown in Figure 2, the composite video generation system according to Embodiment 2 includes: a video candidate extraction unit 31 that extracts one or more background video candidates with usage conditions that allow the said usage based on the content of the usage information about the background video generated by the usage information generation unit 22; a video determination unit 32 that determines the background video to be actually used from among the background video candidates extracted by the video candidate extraction unit 31; and a video candidate extraction unit 33 that extracts one or more content video candidates with usage conditions that allow the said usage based on the content of the usage information about the content video generated by the usage information generation unit 27. The system now includes a video selection unit 34 that determines which content video to actually use from among the content video candidates extracted by the video candidate extraction unit 33.

[0054] The video candidate extraction unit 31 (corresponding to the first video candidate extraction means in the claims) is for extracting candidate background videos to be used in the generation of a composite video based on the content of the usage mode information of the background videos during the generation of a composite video. Specifically, the video candidate extraction unit 31 has the function of extracting one or more background videos for which usage conditions are set that permit use based on the usage mode indicated in the presented usage mode information. The video candidate extraction unit 31 has the function of accessing the corresponding distributed ledger for each background video stored in the background video database 1 and confirming the content of the usage condition information stored in the distributed ledger, thereby extracting background videos for which usage conditions are set that permit the usage mode indicated in the usage mode information newly generated by the usage mode information generation unit 22.

[0055] The video determination unit 32 (corresponding to the first video determination means in the claims) is for determining which of the one or more background video candidates extracted by the video candidate extraction unit 31 will actually be used as the background video to be used in the composite video. Specifically, the video determination unit 32 may determine which of the one or more candidates extracted by the video candidate extraction unit 31 is to be used according to a predetermined algorithm based on the preferences of the person who instructed the generation of the composite video, the person who intends to possess the composite video, the person who intends to use it, and other components, such as the relationship with avatars and content videos (for example, commonality in shape, color tone, etc.), or it may be configured to visually display the candidate background videos and allow the person who instructed the generation of the composite video to make a selection.

[0056] The video candidate extraction unit 33 (corresponding to the second video candidate extraction means in the claims) is for extracting candidate content videos to be used for generating a composite video based on the content video usage information during the generation of a composite video. Specifically, the video candidate extraction unit 33 has the function of extracting one or more content videos for which usage conditions are set that permit use based on the usage pattern indicated in the presented usage pattern information. The video candidate extraction unit 31 has the function of accessing the corresponding distributed ledger for each content video stored in the content video database 2 and confirming the content of the usage condition information stored in the distributed ledger, thereby extracting content videos for which usage conditions are set that permit the usage pattern indicated in the usage pattern information newly generated by the usage pattern information generation unit 27.

[0057] The video determination unit 34 (corresponding to the second video determination means in the claims) is for determining which of the one or more content video candidates extracted by the video candidate extraction unit 33 will actually be used as the content video to be used in the composite video. Specifically, the video determination unit 34 may determine which of the one or more candidates extracted by the video candidate extraction unit 33 is to be used according to a predetermined algorithm based on the preferences of the person who instructed the generation of the composite video, the person who intends to possess the composite video, the person who intends to use it, and other components, such as avatars, background videos, and relationships with other content videos (for example, commonality in shape, color tone, etc.), or it may be configured to visually display the candidate content videos and allow the person who instructed the generation of the composite video to make a selection.

[0058] Next, the advantages of the composite video generation system according to this second embodiment will be described. In addition to the advantages shown in the first embodiment, the composite video generation system according to this second embodiment has the advantage that, when generating the video to be inserted, which is the source of the composite video, even if no specific candidate video materials for background video and content video have been set, the system automatically acquires video candidates that allow the usage patterns indicated in the usage pattern information of the background video and content video, based on the usage pattern information of said usage pattern information. This eliminates the need to consider the content of the video materials from scratch, and makes it easy to acquire options for the video materials to be used. Furthermore, in this second embodiment, the video candidate extraction unit has the function of extracting video materials for which usage conditions that allow the usage patterns have been set as video candidates. This eliminates the problem of having to redo the selection of video materials due to usage condition issues after the design has been formulated, and has the advantage that the video to be inserted can be generated simply and quickly. Note that the video candidate extraction units 31 and 33 may extract video materials (background video and content video) that have been used in the past in the usage patterns in question, rather than extracting video materials for which usage conditions that satisfy the specific usage patterns used when generating the composite video have been set. In this case, new advantages arise, such as being able to consider the specific configuration of new composite images while referring to past usage examples. [Industrial applicability]

[0059] This invention can be used as a technique to generate a composite image by inserting an avatar, which is generated based on a person's image, into a target image composed of one or more video materials. [Explanation of Symbols]

[0060] 1. Background video database 2. Content Video Database 3. Human Video Input Section 4.35 Background video selection section 5.36 Content Video Selection Section 6. Content Video Processing Department 7. Insertion target video generation unit 8. Avatar Generation Section 9 Avatar insertion section 10. Composite video output section 11 Background Token Generation Unit 12. Content Token Generation Unit 13 Avatar Token Generation Unit 14. Video Token Generation Unit 15 Background Transaction Generation Unit 16. Content Transaction Generation Unit 17 Avatar Transaction Generation Unit 18. Video Transaction Generation Unit 19 Electronic signature generation section 20 Output section 21, 26 Video Identification Department 22, 27 Usage information generation unit 23, 28 Usage condition information acquisition section 24, 29 Usability determination section 25, 30 Video extraction section 31, 33 Video Candidate Extraction Unit 32, 34 Image Determination Section

Claims

1. A composite video generation system that generates a composite video by inserting an avatar, generated based on a predetermined person's video, into a portion of the target video, A background video selection means for selecting a background video, which is video material related to the background of the video to be inserted, A content video selection means for selecting a content video, which is a video material to be placed together with a background video in the aforementioned video to be inserted, Content video processing means that performs necessary processing on the content video selected by the content video selection means according to its arrangement in the video to be inserted, An insertion target video generation means generates the insertion target video based on the background video selected by the background video selection means and the content video selected by the content video selection means, An avatar generation means for generating the avatar based on the aforementioned person video, An avatar insertion means for inserting the avatar generated by the avatar generation means into all or part of the content video in the video to be inserted and / or a part of the background video in the video to be inserted. A composite video output means that outputs a composite video generated by inserting the avatar into the target video by the avatar insertion means, A composite image generation system characterized by having the following features.

2. The background image selection means is A first usage information generation means generates first usage information which is information relating to the usage of the synthesized video that is generated and / or the usage of the background video in the synthesized video, A first usability determination means for determining whether the usage of the background video indicated in the first usage information satisfies the usage conditions predetermined for the background video, Equipped with, The aforementioned content video selection means is A second usage pattern information generation means generates a second usage pattern information which is information relating to the usage pattern of the content video in the video to be inserted, A second means for determining whether the usage of the content video indicated in the second usage information satisfies the usage conditions predetermined for the content video, Equipped with, The composite video generation system according to claim 1, characterized in that the insertion target video generation means generates the insertion target video using a background video which the first usability determination means has determined to satisfy the usage conditions and a content video which the second usability determination means has determined to satisfy the usage conditions.

3. The background image selection means is A first usage mode information generating means that generates first usage mode information which is information relating to the usage mode of the composite video including the background video and / or the usage mode of the background video in the composite video, A first candidate video extraction means for extracting one or more candidate background videos from among the one or more background videos for which usage conditions are set that permit use in the usage mode indicated in the first usage mode information, A first video determination means for determining the background video to be used in the composite video from among one or more candidate background videos extracted by the first candidate video extraction means, Equipped with, The aforementioned content video selection means is A second usage information generation means generates a second usage information which is information relating to the usage of the composite video including the content video and / or the usage of the content video in the composite video, A second candidate video extraction means for extracting one or more candidate content videos from among the one or more content videos for which usage conditions are set that permit use in the usage mode indicated in the second usage mode information, A second video determination means for determining the content video to be used in the composite video from among the one or more candidate content videos extracted by the second candidate video extraction means, Equipped with, The composite video generation system according to claim 1, characterized in that the insertion target video generation means generates the insertion target video using the background video determined by the first video determination means and the content video determined by the second video determination means.

4. A composite video generation method that generates a composite video by inserting an avatar, generated based on a predetermined person's video, into a portion of the target video, A video material selection step of selecting one or more video materials that form the video to be inserted, A video material processing step in which the video material selected in the video material selection step performs necessary processing on the video material according to its arrangement in the video to be inserted, An insertion target video generation step in which the insertion target video is generated based on the video material selected in the video material selection step, An avatar generation step of generating the avatar based on the aforementioned person video, An avatar insertion step in which the avatar generated in the avatar generation step is inserted into all or part of the video material in the video to be inserted, A composite video output step which outputs a composite video generated by inserting the avatar into the video to be inserted in the avatar insertion step, A method for generating composite images, characterized by including [a specific component].

5. The aforementioned video material selection step is, A usage mode information generation step generates usage mode information which is information relating to the manner in which the synthesized video is used and / or the manner in which the video material is used in the synthesized video, A usability determination step that determines whether the usage of the video material indicated in the usage information satisfies the usage conditions predetermined for the video material, It further includes, The composite video generation method according to claim 4, characterized in that the insertion target video generation step generates the insertion target video using video material that the usability determination step has determined to satisfy the usage conditions.

6. A composite video generation program that causes a computer to generate a composite video by inserting an avatar, generated based on a predetermined person's image, into a portion of the target video, To the aforementioned computer, A video material selection function that, in determining one or more video materials to form the aforementioned video to be inserted, selects a video material for which usage conditions are set that permit the manner in which the composite video including the video material is used and / or the manner in which the video material is used in the composite video. A video material processing function that performs necessary processing on one or more video materials selected by the video material selection function according to their arrangement in the video to be inserted, An insertion target video generation function that generates the insertion target video based on the video material selected by the video material selection function, An avatar generation function that generates the avatar based on the aforementioned person video, An avatar insertion function that inserts the avatar generated by the avatar generation function into all or part of the video material in the video to be inserted, A composite video output function that outputs a composite video generated by inserting the avatar into the target video using the avatar insertion function, A composite video generation system characterized by performing the following action.

Images