Generative Artificial Intelligence-Based Method and System for Video Generation for Theater Screening

Abstract

An interactive video generation method implemented by at least one artificial intelligence module according to a disclosed embodiment comprises: receiving a user's response by a first module, generating avatar data appearing in the video, and controlling the emotional expression of the avatar based on the user's response; generating video data including virtual space information that serves as the background of the video by a second module; generating story data of the video based on the user's response by a third module; generating interactive video data based on the data generated by the first module, the second module, and the third module by a fourth module; and converting the interactive video data generated by the fourth module to display the interactive video on a display, wherein receiving a user's response includes receiving responses from a plurality of users in a movie theater, and the step of displaying the interactive video may include displaying the generated video on a screen of the movie theater.

Description

Technology Field

[0001] The present invention relates to an interactive image generation device and an interactive image generation method, and more specifically, to a method and device for producing a generative image using a user's response as a prompt. Background Technology

[0002] Conventional methods for generating user-responsive video involved receiving a user's response at a preset point in time and transmitting a subsequent video based on the received result. Consequently, conventional responsive video generation methods had the problem of inevitably limiting user choices.

[0003] Specifically, conventional responsive video was a method in which the next video was determined based on the user's alternative choice of a pre-recorded video. The problem to be solved

[0004] To solve the aforementioned problems, the present invention aims to provide an interactive video generation method and apparatus that generate an avatar expressing emotions based on a user's response and generate multiple stories based on the user's response. means of solving the problem

[0005] An interactive video generation method implemented by at least one artificial intelligence module according to a disclosed embodiment comprises: receiving a user's response by a first module, generating avatar data appearing in the video, and controlling the emotional expression of the avatar based on the user's response; generating video data including virtual space information that serves as the background of the video by a second module; generating story data of the video based on the user's response by a third module; generating interactive video data based on the data generated by the first module, the second module, and the third module by a fourth module; and converting the interactive video data generated by the fourth module to display the interactive video on a display, wherein receiving a user's response includes receiving responses from a plurality of users in a movie theater, and the step of displaying the interactive video may include displaying the generated video on a screen of the movie theater.

[0006] An interactive video generation device that generates video based on a user's response in a disclosed embodiment may include at least one artificial intelligence module, and the at least one artificial intelligence module may include a first module that generates avatar data appearing in the video and controls the emotional expression of the avatar based on the user's response, a second module that generates video data including a virtual space that serves as the background of the video, a third module that generates story data of the video based on the user's response, a fourth module that generates interactive video data based on the data generated by the first module, the second module, and the third module, and a video generation processor configured to convert the interactive video data generated by the fourth module and display the interactive video on a display. Effects of the invention

[0007] By including all of the above-described configurations, the disclosed embodiment can provide an interactive video generation method and apparatus that generates an avatar expressing emotions based on a user's response and generates multiple stories based on the user's response. Brief explanation of the drawing

[0008] FIGS. 1 and FIGS. 2 are intended to illustrate an interactive image generation system according to a disclosed embodiment. FIG. 3 is a block diagram of an interactive image generating device according to a disclosed embodiment. FIG. 4a is a block diagram of an interactive image generation system according to a disclosed embodiment. FIG. 4b is a conceptual diagram of an interactive image generation system according to a disclosed embodiment. FIG. 5 is a block diagram for explaining a third module according to a disclosed embodiment. FIG. 6 is a block diagram for explaining a fourth module according to a disclosed embodiment. FIG. 7 is a flowchart of an interactive image generation method according to a disclosed embodiment. FIG. 8 is a flowchart illustrating the generation of a story of an image in an interactive image generation method according to a disclosed embodiment. FIG. 9 is a flowchart illustrating the control of emotions of an avatar appearing in an image in an interactive image generation method according to a disclosed embodiment. FIG. 10 is a flowchart for explaining the generation of an image in an interactive image generation method according to a disclosed embodiment. Specific details for implementing the invention

[0009] Hereinafter, preferred embodiments according to the present invention will be described in detail with reference to the accompanying drawings. The detailed description disclosed below, together with the accompanying drawings, is intended to describe exemplary embodiments of the present invention and is not intended to represent the only embodiment in which the present invention can be practiced.

[0010] These embodiments are provided merely to make the disclosure of the invention complete and to fully inform those skilled in the art of the scope of the invention, and the invention is defined only by the scope of the claims.

[0011] In some cases, to avoid obscuring the concept of the present invention, known structures and devices may be omitted or illustrated in the form of block diagrams focusing on the core functions of each structure and device. Additionally, throughout this specification, the same reference numerals are used to describe identical components.

[0012] Throughout the specification, when a part is described as "comprising" or "including" a component, this means that, unless specifically stated otherwise, it does not exclude other components but may include additional components.

[0013] Additionally, the term "processor" as used in the specification refers to a unit that processes at least one function or operation, which may be implemented in hardware, software, or a combination of hardware and software. Furthermore, "one (a or an)," "one," and similar related terms may be used in the context describing the invention in both singular and plural forms, unless otherwise indicated in the specification or clearly contradicted by the context.

[0014] Furthermore, specific terms used in the embodiments of the present invention are provided to aid in understanding the invention, and unless otherwise defined, all terms used herein, including technical or scientific terms, have the same meaning as generally understood by those skilled in the art to which the present invention pertains. The use of such specific terms may be modified in other forms without departing from the technical spirit of the present invention.

[0015] FIGS. 1 and FIGS. 2 are intended to illustrate an interactive image generation system according to a disclosed embodiment.

[0016] Referring to FIG. 1, an interactive video generation system (10) according to one embodiment may include an interactive video generation device (100) and a user terminal (200). An interactive video generation device (100) according to one embodiment may receive a user's response from a user terminal (200) and generate a video corresponding to the received response. For example, the interactive video generation device (100) may include an artificial intelligence module that generates a video using the user's response as a prompt and generate a video corresponding to the user's response. An interactive video according to one embodiment may be a video generated by a generative artificial intelligence module using the received user's response as a prompt. The generative artificial intelligence module according to one embodiment may be a game engine. That is, the interactive video may be a video that can be generated in various scenarios using the user's response itself as a prompt.

[0017] An interactive video generation device (100) according to one embodiment includes at least one artificial intelligence module and can generate an interactive video. The at least one artificial intelligence module according to one embodiment receives a user's response, generates avatar data appearing in the video, and can adjust the emotional expression of the avatar based on the user's response. An avatar according to one embodiment may be an actor or character appearing in the video. For example, the avatar may be a virtual human appearing in the video.

[0018] According to another embodiment, at least one artificial intelligence module can generate image data including virtual space information that serves as the background of the image. According to one embodiment, the virtual space may be the shooting location or background of the virtually generated image. For example, at least one artificial intelligence module can generate a virtual background based on a user's response.

[0019] According to another example, at least one artificial intelligence module can generate video story data based on user responses. For example, at least one artificial intelligence module can analyze user responses using natural language processing functions and generate video story data corresponding to user responses.

[0020] According to another example, at least one artificial intelligence module can generate video story data based on user reactions. For example, at least one artificial intelligence module can analyze user reactions using natural language processing functions and generate video story data corresponding to user reactions. Additionally, at least one artificial intelligence module can generate interactive video data based on data generated by at least one artificial intelligence module included in the interactive video generation device (100). For example, at least one artificial intelligence module can generate interactive video data by combining avatar data, virtual space data, and story data generated by other artificial intelligence modules.

[0021] An interactive video generating device (100) according to one embodiment may convert interactive video data generated by at least one artificial intelligence module and display an interactive video on a display. The display according to one embodiment may be a screen or a user terminal existing outside the interactive video generating device (100), but is not limited thereto and may further include an external display. The user terminal according to one embodiment may include a smartphone, a tablet PC, a personal computer, an AR / XR device, or a metaverse display, but is not limited thereto. For example, the interactive video generating device (100) may generate interactive video data using an artificial intelligence module and display the interactive video data on a movie theater screen by at least one processor.

[0022] Referring to FIG. 1 and FIG. 2 together, an interactive image generation system (10) according to one embodiment may include a server (300) that stores a plurality of data. An interactive image generation device (100) according to one embodiment may transmit and receive a plurality of data to and from the server (300). For example, the interactive image generation device (100) may transmit information received from a user terminal (200) to the server (300). A server (300) according to one embodiment may collect and store a plurality of data by communicating with various external user terminals (not shown). An interactive image generation device (100) according to one embodiment may access the data of the server (300) and learn from the data of the server (300). An interactive image generation device (100) according to one embodiment may generate an image by utilizing the data of the server (300).

[0023] FIG. 3 is a block diagram of an interactive image generating device (100) according to a disclosed embodiment, and FIG. 4a is a block diagram of an interactive image generating system (10) according to a disclosed embodiment.

[0024] Referring to FIG. 3, the interactive video generating device (100) according to the disclosed embodiment may include a plurality of artificial intelligence modules. For example, the interactive video generating device (100) may include a first module (110) that generates avatar data appearing in the video and controls the emotional expression of the avatar based on the user's reaction, a second module (120) that generates video data including a virtual space that serves as the background of the video, a third module (130) that generates story data of the video based on the user's reaction, a fourth module (140) that generates interactive video data based on the data generated by the first module (110), the second module (120), and the third module (130), and a video generating processor (150) configured to convert the interactive video data generated by the fourth module (140) and display the interactive video on a display.

[0025] A first module (110) according to one embodiment may receive a user's response, generate avatar data appearing in a video, and adjust the emotional expression of the avatar based on the user's response. An avatar according to one embodiment may be an entity that communicates with a user in virtual reality. For example, the avatar may be a character, a virtual human, or a virtual animal, but is not limited thereto.

[0026] According to one embodiment, the first module (110) can generate an avatar in virtual reality. According to one embodiment, the first module (110) communicates with a user terminal (not shown) to which a user inputs a response and can generate interaction information between the avatar and the user. For example, the first module (110) can receive voice data from an external user terminal (not shown), convert the received voice data into text data, and analyze the converted text data. According to one embodiment, the first module (110) can analyze the user's facial expression fingerprint to generate an interaction scenario and generate an avatar whose facial expression fingerprint and facial fingerprint pattern match.

[0027] A first module (110) according to one embodiment can generate an interaction scenario with a user using analyzed text data. An interaction scenario according to one embodiment may be a scenario that controls the emotions expressed by an avatar. For example, the first module (110) can generate facial expression data and behavior data of the avatar from the generated interaction scenario. A first module (110) according to one embodiment can generate facial expression data and behavior data of the avatar by analyzing data received from a user terminal (not shown), and can generate an interaction scenario with a user based on the generated facial expression data and behavior data.

[0028] According to one embodiment, the first module (110) can generate an interaction scenario with a user by recognizing a face region and an action region using the user's facial expression fingerprint and action fingerprint, generating vector information from feature points extracted from the user's face region, and converting the generated vector information into data for an avatar. The user's response received by the first module (110) according to one embodiment may include first position information indicating the location of feature points in 3D space and second information indicating the amount of rotation and movement of the face region in 3D space. Each of the first information and second information according to one embodiment may include vector information for the user's body, and by controlling the vector information, an interaction scenario between the avatar and the user can be generated in which the facial expression of the avatar's face region is determined.

[0029] A first module (110) according to one embodiment can generate behavioral data for creating an avatar. The behavioral data according to one embodiment may include a person indicator representing the location of a behavioral area within a three-dimensional space and an emotional expression value representing voice information. A first module (100) according to one embodiment can determine the avatar's behavioral area and voice information by controlling the emotional expression value. For example, the voice information may be the pitch, tone, or speed of the avatar's voice.

[0030] A first module (110) according to one embodiment may include a deepfake module, a double actor filming module, a synthesis blending module, and a Text-to-Speech (TTS) module. For example, the first module (110) may process story data generated by the third module (130) described later to generate facial expression data and behavior data of the avatar, and may generate interaction information between the user and the avatar using the generated facial expression data and behavior data. A deepfake module according to one embodiment may generate a fake image by replacing a non-existent source image generated by computer graphics or artificial intelligence technology with a target image. A double actor filming module according to one embodiment may capture the face and body images of a double actor. For example, the double actor filming module may capture each of the double actions of the avatar that appear multiple times in the video. A synthesis blending module according to one embodiment may synthesize the face of a replacement actor with the face of a pre-recorded actor. A TTS module according to one embodiment may convert text data into voice data.

[0031] Additionally, the first module (110) according to one embodiment may include a STT (Speech-to-Text) module, a text mining module, and a DB (Data Base) loading / analysis module. For example, the first module (110) may be a data processing module.

[0032] A first module (110) according to one embodiment receives voice data provided from a user terminal (not shown), converts the voice data into text data, extracts meaningful text, emotions, etc. from the text data, and loads and / or analyzes the text data. A STT module according to one embodiment can generate text data corresponding to the voice data. A text mining module according to one embodiment can extract meaningful text data from the text data. A first module (110) according to one embodiment can extract meaningful text data from the extracted text data. The meaningful text data according to one embodiment may be text data capable of changing the facial expressions, emotions, and / or behaviors of a virtual human. A text mining module according to one embodiment can extract text data containing or revealing emotions from the text data. A DB load / analysis module according to one embodiment can store and / or analyze the text data. For example, the DB load / analysis module may store text data converted and / or generated by the STT module and text data extracted from the text mining module.

[0033] A first module (110) according to one embodiment can map and manage the location of the avatar in virtual space and the 3D modeling data of the avatar. For example, the first module (110) can change the location of the avatar over time.

[0034] A second module (120) according to one embodiment can generate video data including a virtual space that serves as the background of the video. For example, the background of the video may be the background of a pre-recorded movie. A second module (120) according to one embodiment can generate video data by creating a virtual space set for an avatar corresponding to a user to be active, and by synthesizing the virtual space and the avatar.

[0035] A third module (130) according to one embodiment can generate video story data based on user reactions. A third module (130) according to one embodiment may include a scenario generation module, a tag generation module, and a sequence branching point management module.

[0036] A tag generation module according to one embodiment can generate story data including facial expression fingerprints and behavior fingerprints of an avatar. A sequence branching point management module according to one embodiment can generate a sighting point for each sequence, which is a time unit of the video, to distinguish between the previous sequence and the current sequence when generating an interactive video.

[0037] A third module (130) according to one embodiment can generate a story based on a user's response received at a reference point of a pre-recorded video. A sequence branching point management module according to one embodiment can manage the reference point of the pre-recorded video. For example, the third module (130) may request a response from the user at a reference point of the pre-recorded video, receive the user's response when the user inputs a response into a user terminal (not shown), and generate a story of the video to be unfolded based on the received user's response information. The user's response according to one embodiment may be a prompt for the third module (130). For example, the third module (130) may ask the user to decide whether or not to open a door appearing in the video, and based on the received user's response, generate story data after the door is opened or generate story data when the door is not opened.

[0038] A third module (130) according to one embodiment can generate a story that can communicate with the user based on text data and emotion value data extracted from the user's response. Emotion value data according to one embodiment may be a numerical classification of the user's emotions. For example, the emotion value data may be learned by analyzing the user's voice in real time, and the user's emotion value may be extracted based on the learning result. Emotion values ​​according to one embodiment may be classified to have a numerical value from 1 to 100. A third module (130) according to one embodiment may analyze the pitch, speed, and tone of the user's voice and classify the user's emotions. Emotion values ​​according to one embodiment may be expressed as a value close to 1 in the case of anxious and sad emotions, and as a value close to 100 in the case of confident and happy emotions.

[0039] When text data and emotion value data are extracted from the user's response, the third module (130) according to one embodiment can generate story data of the video. For example, the third module (130) can generate story data using the analyzed text data provided by the first module (110), and can also generate avatar data including facial expression fingerprints and action fingerprints of the avatar from the generated story data. The third module (130) according to one embodiment can distinguish facial expression fingerprints and action fingerprints based on the data generated by the first module (110). For example, the first module (110) can generate facial expression fingerprints and action fingerprint data of the avatar for each scene, each situation, and each emotion from the story data generated by the third module (130).

[0040] A third module (130) according to one embodiment may generate tags for each fingerprint of the avatar's facial expression data, emotion data, and behavior data received from the first module (110), assign them to each of the facial expression, emotion, and behavior fingerprints, and generate story data by assigning a plurality of sequences. For example, the generated story data may be generated differently based on the user's emotion information determined by the first module (110). An avatar according to one embodiment may act or perform actions based on the generated story data.

[0041] A fourth module (140) according to one embodiment can generate interactive video data based on data generated by the first module (110), the second module (120), and the third module (130). For example, the fourth module (140) can generate interactive video data by utilizing avatar data and virtual space data. A fourth module (140) according to one embodiment may also generate video data by additionally synthesizing a plurality of avatars.

[0042] According to one embodiment, the fourth module (140) can edit the data received from the first module (110), the second module (120), and the third module (130) respectively based on the user's real-time viewing response. For example, the fourth module (140) can adjust the sound of the video or change the structure of the virtual space. Changing the structure of the virtual space according to one embodiment may involve adjusting the physical size of the virtual space, changing the 3D shape of the virtual space, changing the changes in installations appearing in the virtual space, or changing the texture projected onto the walls of the virtual space. According to one embodiment, the fourth module (140) can adjust the sound of the video and change the texture of the virtual space according to the avatar's facial expressions, actions of the avatar, voice of the avatar, etc. For example, the fourth module (140) can generate a video that interacts with the user by controlling the sound of the video based on statistically analyzed audience reactions.

[0043] A fourth module (140) according to one embodiment can acquire multi-view performance video data captured according to a plurality of viewpoints in a virtual space. For example, the fourth module (140) can configure the point of view in which an avatar looks at the virtual space into a plurality of viewpoints.

[0044] An image generation processor (150) according to one embodiment can convert interactive image data generated by the fourth module (140) to display an interactive image on a display (not shown). For example, the image generation processor (150) can receive image data generated by the fourth module (140) and encode the received image data into a plurality of different types including at least one of VR data, AR data, or normal video data. For example, the image generation processor (150) may include a plurality of encoders.

[0045] A processor (150) according to one embodiment can convert video data in different ways depending on the conversion mode. The conversion mode according to one embodiment may be one of VR mode, AR mode, and general viewing mode. When encoding video data into VR data, the video generation processor (150) according to one embodiment can encode video data in a VR format that is viewed from a first-person perspective of an avatar and generate VR data. When encoding video data into AR data, the video generation processor (150) according to one embodiment can encode video data in AR mode and generate AR data. For example, the video generation processor (150) can extract data about an avatar excluding virtual space from the video data and encode video data in a third-person perspective according to the AR format. When encoding video data into normal video data, the video generation processor (150) according to one embodiment can generate video viewed from a third-person perspective. For example, the video generation processor (150) according to one embodiment can encode data about virtual space and avatar.

[0046] A video generation processor (150) according to one embodiment may receive a viewing mode from a user. For example, the viewing mode may include a general viewing mode, a group viewing mode, and a private viewing mode. A general viewing mode according to one embodiment may be a mode in which the avatars of all viewers, including the viewer, are rendered in a virtual space. A group viewing mode according to one embodiment refers to a mode in which virtual avatars agreed upon by the user are displayed. A private viewing mode according to one embodiment may be a mode in which a video is viewed in a state where only one's own avatar is rendered in a virtual space.

[0047] Referring to FIGS. 3 and FIGS. 4a, an interactive video generation system (10) according to one embodiment may include an interactive video generation device (100), a user response detection unit (400), and a display (500). The user response detection unit (400) according to one embodiment may be a microphone or camera provided on a user terminal or externally. The interactive video generation device (100) according to one embodiment may receive a user's response from the user response detection unit (400) and convert the received user response information into data. For example, the interactive video generation device (100) may extract voice information, emotion information, or behavior information from the user's response information and convert it into data.

[0048] An interactive video generating device (100) according to one embodiment may convert interactive video data generated by at least one artificial intelligence module and display an interactive video on a display (500). The display (500) according to one embodiment may be a screen existing outside the interactive video generating device (100) or a screen installed on a user terminal, but is not limited thereto. A user terminal according to one embodiment may include a smartphone, a tablet PC, a personal computer, an AR / XR device, or a metaverse display, but is not limited thereto. For example, the interactive video generating device (100) may generate interactive video data using an artificial intelligence module and display the interactive video on a screen in a movie theater.

[0049] According to one embodiment, an interactive video generating device (100) can receive real-time viewing reactions from an audience watching a performance and can provide the audience's real-time viewing reactions to the video visually and audibly. For example, the interactive video generating device (100) can statistically analyze the user's response (e.g., real-time comments from the user, gaze of the user's virtual avatar, real-time number of users, location of the user's virtual avatar, facial expression of the user's virtual avatar, row of the user's virtual avatar, etc.), voice of the user's virtual avatar, etc. According to one embodiment, the interactive video generating device (100) can generate a performance video based on the user's virtual avatar. According to one embodiment, the performance video may be a video generated using the user's response as a prompt.

[0050] FIG. 4b is a conceptual diagram of an interactive image generation system (10) according to a disclosed embodiment.

[0051] Referring to FIGS. 4a and 4b, an interactive video generation system (10) according to one embodiment may include a user response detection unit (400), an interactive video generation device (100), and a display (500). In the embodiment of FIG. 4b, the case where the user (1000) is a plurality of audience members of a movie theater and the display (500) is a screen of the movie theater is described, but it is not limited thereto, and a response may be detected from a single or plurality of users (1000), and a video may be generated by using a display (500) with a configuration such as a screen of a user terminal.

[0052] According to one embodiment, the user (1000) may be multiple. For example, the user (1000) may be an audience watching a movie in a group at a movie theater or theater. According to one embodiment, the user response detection unit (400) may include a voice receiving unit (401) and a video receiving unit (402).

[0053] A voice receiving unit (401) according to one embodiment can detect the voice of a user (1000) and transmit the detected voice to an interactive video generating device (100). An interactive video generating device (100) according to one embodiment can generate a video using the received voice of the user (1000) as a prompt.

[0054] A video receiving unit (402) according to one embodiment can detect the actions of a user (1000) and transmit the detected actions to an interactive video generating device (100). An interactive video generating device (100) according to one embodiment can generate a video using the received actions of the user (1000) as a prompt.

[0055] That is, an interactive video generating device (100) according to one embodiment can detect the voice and actions of a user (1000) even if the user (1000) has not directly entered a prompt, and can generate video data using the detected voice or actions of the user as a prompt. The video data according to one embodiment may relate to a video in which an avatar represented as a virtual human appears, but is not limited thereto.

[0056] FIG. 5 is a block diagram for explaining a third module according to a disclosed embodiment.

[0057] Referring to FIG. 5, a third module (130) according to one embodiment may include a natural language recognition unit (131), a natural language processing unit (132), a script management unit (133), and a data collection unit (134).

[0058] A natural language recognition unit (131) according to one embodiment can recognize and receive language among a user's responses. A natural language recognition unit (131) according to one embodiment may include a Speech-to-Text (STT) module, a text mining module, and a Database (DB) load / analysis module. A natural language recognition unit (131) according to one embodiment can receive voice data provided from a user terminal (not shown) and convert the voice data into text data. An STT module according to one embodiment can generate text data corresponding to the voice data. A text mining module according to one embodiment can extract meaningful text data from the text data.

[0059] A natural language processing module (132) according to one embodiment can process received natural languages. For example, the natural language processing module (132) can analyze a user's response and generate story data of a video corresponding to the user's response. A natural language processing module according to one embodiment may be an artificial intelligence module including a Large Language Model (LM).

[0060] A script management unit (133) according to one embodiment can generate video story data based on a user's response. For example, a natural language processing module (132) can analyze a user's response using a natural language processing function, and the script management unit (133) can generate video story data corresponding to the user's response.

[0061] FIG. 6 is a block diagram for explaining a fourth module according to a disclosed embodiment.

[0062] Referring to FIGS. 1 and FIGS. 6, a fourth module (140) according to one embodiment can receive data from a first module (110), a second module (120), and a third module (130) in real time and process the received data.

[0063] For example, the fourth module (140) can generate interactive video data based on data generated by the first module (110), the second module (120), and the third module (130) included in the interactive video generation device (100). For example, the fourth module (140) can generate interactive video data by combining avatar data, virtual space data, and story data generated by the first module (110), the second module (120), and the third module (130), respectively.

[0064] FIG. 7 is a flowchart of an interactive image generation method according to a disclosed embodiment.

[0065] Referring to FIGS. 3 and FIGS. 7, an interactive image generating device (100) according to one embodiment can receive a user's response (S710).

[0066] An interactive video generating device (100) according to the disclosed embodiment may include a plurality of artificial intelligence modules. For example, the interactive video generating device (100) may include a first module (110) that generates avatar data appearing in a video and controls the emotional expression of the avatar based on the user's reaction; a second module (120) that generates video data including a virtual space that serves as the background of the video; a third module (130) that generates story data of the video based on the user's reaction; a fourth module (140) that generates interactive video data based on the data generated by the first module (110), the second module (120), and the third module (130); and a video generating processor (150) configured to convert the interactive video data generated by the fourth module (140) and display an interactive video on a display.

[0067] An interactive video generating device (100) according to one embodiment receives a user's response, generates avatar data appearing in the video, and can adjust the emotional expression of the avatar based on the user's response. An avatar according to one embodiment may be a being that communicates with a user in virtual reality. For example, the avatar may be a character, a virtual human, or a virtual animal, but is not limited thereto.

[0068] When a user's response is received, the interactive video generating device (100) according to one embodiment can generate avatar data appearing in the video (S720).

[0069] According to one embodiment, an interactive video generating device (100) can generate an avatar in virtual reality. According to one embodiment, an interactive video generating device (100) can communicate with a user terminal (not shown) to which a user inputs a response and generate interaction information between the avatar and the user. For example, the interactive video generating device (100) can receive voice data from an external user terminal (not shown), convert the received voice data into text data, and analyze the converted text data. According to one embodiment, an interactive video generating device (100) can analyze the user's facial expression fingerprint to generate an interaction scenario and generate an avatar whose facial expression fingerprint and facial fingerprint pattern match.

[0070] An interactive video generating device (100) according to one embodiment can control the emotional expression of an avatar based on the user's reaction (S730).

[0071] An interactive video generating device (100) according to one embodiment can generate an interaction scenario with a user using analyzed text data. An interaction scenario according to one embodiment may be a scenario that controls the emotions expressed by an avatar. For example, the interactive video generating device (100) can generate facial expression data and behavior data of an avatar from the generated interaction scenario. An interactive video generating device (100) according to one embodiment can generate facial expression data and behavior data of an avatar by analyzing data received from a user terminal (not shown), and can generate an interaction scenario with a user based on the generated facial expression data and behavior data.

[0072] According to one embodiment, an interactive image generating device (100) can generate an interaction scenario with a user by recognizing a face region and an action region using the user's facial expression fingerprint and action fingerprint, generating vector information from feature points extracted from the user's face region, and converting the generated vector information into data for an avatar. The user's response received by the interactive image generating device (100) according to one embodiment may include first position information indicating the location of feature points in 3D space and second information indicating the amount of rotation and movement of the face region in 3D space. Each of the first information and the second information according to one embodiment may include vector information for the user's body, and by controlling the vector information, an interaction scenario between an avatar and a user in which the facial expression of the avatar's face region is determined may be generated.

[0073] An interactive video generating device (100) according to one embodiment can generate behavior data for generating an avatar. The behavior data according to one embodiment may include a person indicator representing the location of a behavior area within a three-dimensional space and an emotional expression value representing voice information. The interactive video generating device (100) according to one embodiment can determine the behavior area and voice information of the avatar by controlling the emotional expression value. For example, the voice information may be the pitch, tone, or speed of the avatar's voice.

[0074] An interactive video generation device (100) according to one embodiment may include a deepfake module, a double actor shooting module, a synthesis blending module, and a TTS (Text-to-Speech) module. For example, the interactive video generation device (100) may process story data to generate facial expression data and behavior data of an avatar, and may generate interaction information between a user and an avatar using the generated facial expression data and behavior data. A deepfake module according to one embodiment may generate a fake image by replacing a non-existent source image generated by computer graphics or artificial intelligence technology with a target image. A double actor shooting module according to one embodiment may capture the face and body images of a double actor. For example, a double actor shooting module may capture each of the double actions of an avatar that appears multiple times in the video. A synthesis blending module according to one embodiment may synthesize the face of a replacement actor with the face of a pre-shot actor. A TTS module according to one embodiment may convert text data into voice data.

[0075] Additionally, an interactive video generating device (100) according to one embodiment may include a Speech-to-Text (STT) module, a text mining module, and a Database (DB) load / analysis module. An interactive video generating device (100) according to one embodiment may receive voice data provided from a user terminal (not shown), convert the voice data into text data, extract meaningful text, emotions, etc. from the text data, and load and / or analyze the text data. An STT module according to one embodiment may generate text data corresponding to the voice data. A text mining module according to one embodiment may extract meaningful text data from the text data. An interactive video generating device (100) according to one embodiment may extract meaningful text data from the extracted text data. The meaningful text data according to one embodiment may be text data capable of changing the facial expressions, emotions, and / or behaviors of a virtual human. A text mining module according to one embodiment may extract text data containing or revealing emotions from the text data. According to one embodiment, the DB load / analysis module can store and / or analyze text data. For example, the DB load / analysis module can store text data converted and / or generated by the STT module and text data extracted from the text mining module.

[0076] An interactive image generating device (100) according to one embodiment can map and manage the location of an avatar in a virtual space and 3D modeling data of the avatar. For example, the interactive image generating device (100) can change the location of the avatar over time.

[0077] An interactive image generating device (100) according to one embodiment can generate image data including a virtual space that serves as the background of the image (S740).

[0078] An interactive video generating device (100) according to one embodiment can generate video data including a virtual space that serves as the background of the video. For example, the background of the video may be the background of a pre-recorded movie. An interactive video generating device (100) according to one embodiment can generate video data by creating a virtual space set for an avatar corresponding to a user to be active, and by synthesizing the virtual space and the avatar.

[0079] An interactive video generating device (100) according to one embodiment can generate story data of a video based on a user's response (S750).

[0080] An interactive video generating device (100) according to one embodiment can generate story data of a video based on a user's response. An interactive video generating device (100) according to one embodiment may include a scenario generating module, a tag generating module, and a sequence branching point management module.

[0081] A tag generation module according to one embodiment can generate story data including facial expression fingerprints and behavior fingerprints of an avatar. A sequence branching point management module according to one embodiment can generate a sighting point for each sequence, which is a time unit of the video, to distinguish between the previous sequence and the current sequence when generating an interactive video.

[0082] An interactive video generation device (100) according to one embodiment can generate a story based on a user's response received at a reference point of a pre-recorded video. A sequence branching point management module according to one embodiment can manage the reference point of the pre-recorded video. For example, the interactive video generation device (100) may request a response from a user at a reference point of a pre-recorded video, and when the user inputs a response to a user terminal (not shown), receive the user's response and generate a story of the video to be unfolded based on the received user response information. The user's response according to one embodiment may be a prompt regarding the generated video. For example, the interactive video generation device (100) may ask the user to decide whether or not to open a door appearing in the video, and based on the received user response, generate story data after the door is opened or generate story data when the door is not opened.

[0083] An interactive video generating device (100) according to one embodiment can generate a story that can communicate with a user based on text data and emotion value data extracted from a user's reaction. For example, the interactive video generating device (100) can generate story data using text data, and can also generate avatar data including facial expression fingerprints and action fingerprints of an avatar from the generated story data. An interactive video generating device (100) according to one embodiment can distinguish between facial expression fingerprints and action fingerprints. For example, the interactive video generating device (100) can generate facial expression fingerprints and action fingerprint data of an avatar for each scene, each situation, and each emotion from the story data.

[0084] An interactive video generating device (100) according to one embodiment can generate tags for each fingerprint of facial expression data, emotion data, and behavior data of an avatar, assign them to each of the facial expression, emotion, and behavior fingerprints, and generate story data by assigning a plurality of sequences. For example, the interactive video generating device (100) can generate story data differently based on the user's emotion information. An avatar according to one embodiment can act or perform actions based on the generated story data.

[0085] An interactive image generating device (100) according to one embodiment can generate interactive image data (S760).

[0086] An interactive video generating device (100) according to one embodiment can generate interactive video data based on data generated by at least one artificial intelligence module. For example, the interactive video generating device (100) can generate interactive video data by utilizing avatar data and virtual space data. An interactive video generating device (100) according to one embodiment may also generate video data by additionally synthesizing a plurality of avatars.

[0087] An interactive video generating device (100) according to one embodiment can edit data received from at least one artificial intelligence module based on the user's real-time viewing response. For example, the interactive video generating device (100) can adjust the sound of the video or change the structure of the virtual space. Changing the structure of the virtual space according to one embodiment may involve adjusting the physical size of the virtual space, changing the 3D shape of the virtual space, changing the changes in installations appearing in the virtual space, or changing the texture projected onto the walls of the virtual space. An interactive video generating device (100) according to one embodiment can adjust the sound of the video and change the texture of the virtual space according to the avatar's facial expressions, actions of the avatar, voice of the avatar, etc. For example, the interactive video generating device (100) can generate a video that interacts with the user by controlling the sound of the video based on statistically analyzed audience reactions.

[0088] An interactive video generating device (100) according to one embodiment can acquire multi-view performance video data captured according to a plurality of viewpoints in a virtual space. For example, the interactive video generating device (100) can configure the point of view in which an avatar looks at the virtual space into a plurality of viewpoints.

[0089] When interactive video data is generated, the interactive video generating device (100) according to one embodiment can convert the generated interactive video data and display the interactive video on a display (S770).

[0090] An interactive video generating device (100) according to one embodiment can convert interactive video data generated by at least one artificial intelligence module and display an interactive video on a display (not shown). For example, the interactive video generating device (100) can receive video data generated by at least one artificial intelligence module and encode the received video data into a plurality of different types including at least one of VR data, AR data, or normal video data. For example, the interactive video generating device (100) may include a plurality of encoders.

[0091] An interactive video generating device (100) according to one embodiment can convert video data in different ways depending on the conversion mode. The conversion mode according to one embodiment may be one of VR mode, AR mode, and general viewing mode. When encoding video data as VR data, the interactive video generating device (100) according to one embodiment can encode video data in a VR format that is viewed from a first-person perspective of an avatar and generate VR data. When encoding video data as AR data, the interactive video generating device (100) according to one embodiment can encode video data in AR mode and generate AR data. For example, the interactive video generating device (100) can extract data about an avatar excluding virtual space from video data and encode video data in a third-person perspective according to the AR format. When encoding video data as normal video data, the interactive video generating device (100) according to one embodiment can generate video viewed from a third-person perspective. For example, an interactive image generating device (100) according to one embodiment can encode data for a virtual space and an avatar.

[0092] An interactive video generating device (100) according to one embodiment may receive a viewing mode from a user. For example, the viewing mode may include a general viewing mode, a group viewing mode, and a private viewing mode. A general viewing mode according to one embodiment may be a mode in which the avatars of all viewers, including the viewer, are rendered in a virtual space. A group viewing mode according to one embodiment refers to a mode in which virtual avatars agreed upon by the user are displayed. A private viewing mode according to one embodiment may be a mode in which a video is viewed in a state where only one's own avatar is rendered in a virtual space.

[0093] FIG. 8 is a flowchart illustrating the generation of a story of an image in an interactive image generation method according to a disclosed embodiment.

[0094] Referring to FIGS. 3 and FIGS. 8, an interactive image generating device (100) according to one embodiment can extract text data and emotion value data from a user's response (S810).

[0095] An interactive video generating device (100) according to one embodiment can generate a story that can communicate with a user based on text data and emotion value data extracted from a user's response. The emotion value data according to one embodiment may be a numerical classification of the user's emotions. For example, the emotion value data may be learned by analyzing the user's voice in real time, and the user's emotion value may be extracted based on the learning result. The emotion value according to one embodiment may be classified to have a numerical value of 1 to 100. The interactive video generating device (100) according to one embodiment may analyze the pitch, speed, and tone of the user's voice and classify the user's emotions. The emotion value according to one embodiment may be expressed as a value close to 1 in the case of anxious and sad emotions, and as a value close to 100 in the case of confident and happy emotions.

[0096] When text data and emotion value data are extracted from the user's response, the interactive video generating device (100) according to one embodiment can generate story data of the video (S820).

[0097] An interactive video generating device (100) according to one embodiment can generate a story based on a user's response received at a reference point of a pre-recorded video. A sequence branching point management module according to one embodiment can manage the reference point of the pre-recorded video. For example, the interactive video generating device (100) may request a response from a user at a reference point of a pre-recorded video, and when the user inputs a response to a user terminal (not shown), receive the user's response and generate a story of the video to be unfolded based on the received user's response information. The user's response according to one embodiment may be a prompt for the interactive video generating device (100). For example, the interactive video generating device (100) may ask the user to decide whether or not to open a door appearing in the video, and based on the received user's response, generate story data after the door is opened or generate story data when the door is not opened.

[0098] For example, the interactive video generating device (100) can generate story data using text data, and can also generate avatar data including facial expression fingerprints and action fingerprints of the avatar from the generated story data. The interactive video generating device (100) according to one embodiment can distinguish facial expression fingerprints and action fingerprints based on received user response data. For example, the interactive video generating device (100) can generate facial expression fingerprints and action fingerprint data of the avatar for each scene, each situation, and each emotion from the user's story data.

[0099] When video story data is generated, the interactive video generating device (100) according to one embodiment can generate a story that can communicate with the user based on extracted text data and emotion value data (S830).

[0100] An interactive video generating device (100) according to one embodiment can generate tags for each fingerprint of facial expression data, emotion data, and behavior data of an avatar, assign them to each of the facial expression, emotion, and behavior fingerprints, and generate story data by assigning a plurality of sequences. For example, the generated story data may be generated differently based on the user's emotion information. An avatar according to one embodiment may act or perform actions based on the generated story data.

[0101] FIG. 9 is a flowchart illustrating the control of emotions of an avatar appearing in an image in an interactive image generation method according to a disclosed embodiment.

[0102] Referring to FIGS. 3 and FIGS. 9, an interactive image generating device (100) according to one embodiment can generate worldview data related to the user's emotions (S910).

[0103] Worldview data according to one embodiment may be data generated by the user's usual behavior, average voice, and speech pattern. For example, since the voice, behavior, or speech pattern may differ in the same situation even if the emotion is the same for each user, the interactive video generating device (100) can generate worldview data based on the user's emotion.

[0104] When worldview data is generated, the interactive video generating device (100) according to one embodiment can generate emotions corresponding to input learning data based on the worldview data (S920).

[0105] An interactive video generating device (100) according to one embodiment can classify the user's emotions based on preset emotion values ​​and generate the emotions of an avatar.

[0106] An interactive video generating device (100) according to one embodiment can generate a story that can communicate with a user based on text data and emotion value data extracted from a user's response. The emotion value data according to one embodiment may be a numerical classification of the user's emotions. For example, the emotion value data may be learned by analyzing the user's voice in real time, and the user's emotion value may be extracted based on the learning result. The emotion value according to one embodiment may be classified to have a numerical value of 1 to 100. The interactive video generating device (100) according to one embodiment may analyze the pitch, speed, and tone of the user's voice and classify the user's emotions. The emotion value according to one embodiment may be expressed as a value close to 1 in the case of anxious and sad emotions, and as a value close to 100 in the case of confident and happy emotions.

[0107] An interactive video generating device (100) according to one embodiment can generate an action of an avatar corresponding to input learning data based on worldview data (S930).

[0108] An interactive video generating device (100) according to one embodiment can generate tags for each fingerprint of facial expression data, emotion data, and behavior data of an avatar, assign them to each of the facial expression, emotion, and behavior fingerprints, and generate story data by assigning a plurality of sequences. For example, the generated story data may be generated differently based on the user's emotion information. An avatar according to one embodiment may act or perform actions based on the generated story data.

[0109] FIG. 10 is a flowchart for explaining the generation of an image in an interactive image generation method according to a disclosed embodiment.

[0110] An interactive video generating device (100) according to one embodiment can generate first video data by synthesizing a pre-recorded performer and a virtual space (S1010).

[0111] According to one embodiment, the first video data may be video data in which a performer filmed in advance during video production is combined with a virtual space created by an interactive video generating device (100). For example, the interactive video generating device (100) may place performers filmed in advance in a video in which an avatar is active.

[0112] When the first image data is generated, the interactive image generating device (100) according to one embodiment can combine the first image data with a user's avatar received from at least one user terminal (S1020).

[0113] An interactive video generating device (100) according to one embodiment can generate video data including a virtual space that serves as the background of the video. For example, the background of the video may be the background of a pre-recorded movie. An interactive video generating device (100) according to one embodiment can generate video data by creating a virtual space set for an avatar corresponding to a user to be active, and by synthesizing the virtual space and the avatar.

[0114] An interactive image generating device (100) according to one embodiment can generate second image data (S1030).

[0115] According to one embodiment, the second video data may be video data in which avatar data is added to the first video data. That is, the interactive video generating device (100) can generate the second video data by correcting the video data so that pre-recorded performers appear together with the avatar created in the virtual space.

[0116] Meanwhile, the above-described method can be written as a program executable on a computer and can be implemented on a general-purpose digital computer that operates said program using a computer-readable medium. In addition, the structure of the data used in the above-described method can be recorded on a computer-readable medium through various means. A computer-readable medium storing executable computer code for performing various methods of the present invention includes storage media such as magnetic storage media (e.g., ROM, floppy disk, hard disk, etc.) and optical reading media (e.g., CD, DVD, etc.).

[0117] Those skilled in the art related to the embodiments of the present invention will understand that they may be implemented in modified forms without departing from the essential characteristics of the description. Therefore, the disclosed methods should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the detailed description of the invention, and all variations within the scope of the claims should be interpreted as being included within the scope of the invention. Explanation of the symbols

[0118] 10: Interactive Image Generation System 100: Interactive image generation device 110: Module 1 120: Module 2 130: Third Module 140: 4th Module 150: Image generation processor 200: User terminal

Claims

Claim 1 A method for generating an interactive video, implemented by at least one artificial intelligence module, comprising: receiving a user's response by a first module, generating avatar data appearing in the video, and adjusting the emotional expression of the avatar based on the user's response; generating video data including virtual space information that serves as the background of the video by a second module; generating story data of the video based on the user's response by a third module; and generating interactive video data based on the data generated by the first module, the second module, and the third module by a fourth module. A method for generating an interactive video, comprising the step of converting the interactive video data generated in the fourth module and displaying the interactive video on a display, wherein receiving the user's response includes receiving the response from a user terminal, and the step of displaying the interactive video includes displaying a video generated by a generative artificial intelligence module on a movie theater screen using the received user's response as a prompt, and the step of generating the interactive video includes generating a first video data by synthesizing a pre-recorded performer with a virtual space, and generating a second video data in which the user's avatar received from at least one user device appears together with the performer. Claim 2 In claim 1, the step of controlling the emotional expression of the avatar comprises: a step of extracting facial expression fingerprints, behavior fingerprints, text data, and emotional value data from the user's reaction; a step of recognizing a face region and a behavior region using the facial expression fingerprints and behavior fingerprints; and a step of generating vector information from feature points extracted from the face region and converting the vector information into data for the avatar; and the step of generating story data of the video comprises generating a story that can communicate with the user based on the extracted text data and emotional value data. Claim 3 An interactive video generation method according to claim 1, wherein the step of controlling the emotional expression of the avatar comprises generating worldview data related to the user's emotions, generating emotions corresponding to input learning data based on the worldview data, and generating behavior of the avatar corresponding to the input learning data based on the worldview data. Claim 4 In claim 1, the step of generating story data of the video is an interactive video generation method that generates a story based on the user's response received at a reference point of a previously filmed video. Claim 5 delete Claim 6 In claim 1, the step of receiving the user's response includes receiving it from either a device installed in a movie theater or a user terminal. Claim 7 An interactive video generation device comprising at least one artificial intelligence module and generating a video based on a user's response, wherein the at least one artificial intelligence module comprises: a first module that generates avatar data appearing in the video and controls the emotional expression of the avatar based on the user's response; a second module that generates video data including a virtual space that serves as the background of the video; a third module that generates story data of the video based on the user's response; a fourth module that generates interactive video data based on the data generated by the first module, the second module, and the third module; and a video generation processor configured to convert the interactive video data generated by the fourth module and display the interactive video on a display, wherein the video generation processor is configured to display the video generated by the generative artificial intelligence module using the user's response received on a movie theater screen as a prompt, and the fourth module generates first video data by synthesizing a pre-recorded performer with a virtual space, and generates second video data in which the user's avatar received from at least one user device appears together with the performer.

Images