system

Abstract

Provide a system. [Solution] means for collecting interactive input from a user; means for automatically generating computer graphics using computer generation techniques based on the input; A means for assembling the generated computer graphics to create a series of images; means for rendering and transmitting said video to a user terminal; means for enabling a user to interactively input new information while viewing the video; means for generating computer graphics again based on the new input and immediately reflecting the new input; A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] Video production typically requires advanced skills and a significant amount of time from creators, placing a heavy burden on them to create fresh and creative computer graphics (CG). It can also be difficult for viewers to view videos that meet their preferences in real time, limiting their viewing experience. These challenges create a demand for efficient video production and the provision of interactive viewing experiences. [Means for solving the problem]

[0005] The present invention solves the above-mentioned problems by providing a means for collecting interactive input from users in the production of movies and videos, a means for automatically generating computer graphics based on the input using a computer generation technique, a means for combining the generated computer graphics to create a series of images, a means for rendering the images and transmitting them to a user terminal, a means for allowing the user to interactively enter new input while viewing the images, and a means for regenerating computer graphics based on the new input and immediately reflecting the new input. Furthermore, by further adding a means for automatically applying scene effects and sound effects based on user input data and a database means for saving the generated computer graphics and images so that they can be reused in subsequent productions, an efficient and interactive video production and viewing experience is realized.

[0006] A "user" is an entity that uses the system to create and view movies and videos.

[0007] "Interactive input" refers to information about preferences and desires that a user provides when interacting with a system.

[0008] "Computer-generated" refers to the use of algorithms and artificial intelligence techniques to create computer graphics.

[0009] "Computer graphics" are visual representations that are generated using a computer and are part of a video.

[0010] "Auto-generation" means that a system creates an artifact through an algorithm or process without human intervention.

[0011] A "sequence of footage" is a visual story that consists of multiple scenes combined in succession.

[0012] "Rendering" is the process of converting computer graphics or scenes into a visually reproducible state.

[0013] A "user terminal" is a device that a user uses to access the system and perform operations and viewing.

[0014] "New input" refers to information that a user provides to the system as an additional or change request after initial setup.

[0015] The "database means" is a system component that stores generated computer graphics and images in an organized manner and makes them reusable. [Brief explanation of the drawings]

[0016] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10]1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0017] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0018] First, the terms used in the following description will be explained.

[0019] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0020] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0021] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0022] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0023] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0024] [First embodiment]

[0025] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0026] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0027] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0028] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0029] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0030] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0031] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0032] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0033] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0034] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0035] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0036] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0037] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) based on that input using computer generation techniques, and then combines the generated CG to create a series of images. It also has the function of rendering the images and transmitting them to the user's terminal. If the user provides new input while watching a video, the system can regenerate the CG and instantly reflect the input.

[0038] System Overview

[0039] 1. User Input

[0040] The device interactively receives preferences and requests from the user.

[0041] The collected information includes, for example, the atmosphere of the scene, the characteristics of the characters, the type of action, and the like.

[0042] 2. AI Scene Generation

[0043] The generation AI generates the CG of the scene based on the user request received by the server.

[0044] The generated CG is temporarily saved and used for the next generation process.

[0045] 3. Scene Arrangement and Composition

[0046] The server combines multiple CG scenes to create a series of images.

[0047] Effects and sound effects are also automatically generated as needed.

[0048] 4. Video output

[0049] The completed video is rendered on the server and generated as a final data file.

[0050] The server sends the video to the terminal so that the user can view it.

[0051] 5. Interactive Operation

[0052] When a user watches a video on a terminal, they can change or select scenes in real time.

[0053] The server immediately receives the user's new operations and regenerates and reflects the CG scene.

[0054] Specific Examples

[0055] Example of scene generation

[0056] Suppose a user is creating a horror movie and has selected a "dark and scary forest scene" and has specified that the characters should be "zombies wearing old clothes." In this case, the system operates as follows:

[0057] 1. User Input

[0058] The device collects the user's preferences and sends information such as "horror," "dark forest," "zombie characters," and "scary action" to the server.

[0059] 2. Scene Generation

[0060] The server uses generative AI to create a dark and scary CG forest scene, which also includes a user-specified zombie character.

[0061] 3. Scene construction

[0062] The server assembles scenes featuring zombies in a terrifying forest and adds appropriate background sounds and effects.

[0063] 4. Video output

[0064] The server renders the entire video and generates it as a file.

[0065] The server transmits the generated video to the terminal so that the user can view it.

[0066] 5. Interactive Operation

[0067] While watching the video on the device, the user makes a request to change the scene to one in which zombies suddenly appear.

[0068] This request is sent from the device to the server, which then generates a new scene and immediately reflects it.

[0069] In this way, users can interactively enjoy videos that suit their preferences and requests. In addition, the generated videos and CG are stored in a database and can be reused for future productions. This will significantly improve the efficiency of video production and enable the provision of new entertainment experiences.

[0070] The processing flow will be explained below.

[0071] Step 1:

[0072] The user inputs the settings and requests required for video production through the device interface, selecting parameters such as "horror movie," "dark forest," and "zombie characters."

[0073] Step 2:

[0074] The terminal collects the user's input, converts it into a data format, and sends it to the server, including all the parameters selected by the user.

[0075] Step 3:

[0076] The server analyzes the received data and issues instructions based on each parameter to the generation AI, which then generates computer graphics (CG) containing the scenes and characters desired by the user.

[0077] Step 4:

[0078] The server temporarily stores the generated CG in a database and assigns it a reference ID. Each saved CG data is used in the next process.

[0079] Step 5:

[0080] The server retrieves multiple CG data sets stored in a database, combines them, and edits them into a series of images. Scene transitions, effects, and sound effects are also added automatically at this stage.

[0081] Step 6:

[0082] The server renders the edited footage and stores it as a final digital file, which is then converted into a viewable format.

[0083] Step 7:

[0084] The server then sends the rendered video file to the device where the user can view it. A viewing URL or a link to a streaming service may be provided.

[0085] Step 8:

[0086] The user watches the video using the terminal. If a new request arises while watching, the user can input a request such as "I want to change to a scene where zombies suddenly appear."

[0087] Step 9:

[0088] The device sends a new user request to the server, which includes details of what part of the scene to change and how.

[0089] Step 10:

[0090] The server receives the new request, again using the generative AI to generate any additional CG needed, then adds the new scene in the appropriate location and re-renders the entire video.

[0091] Step 11:

[0092] The server transmits the updated video to the terminal again, making the video that reflects the user's new request available for viewing.

[0093] This series of processes allows users to generate, view, and change videos that suit their preferences in real time.

[0094] Example 1

[0095] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0096] Current video production systems have difficulty in instantly reflecting user interactive input and have limitations in automatically applying scene and sound effects. They also lack appropriate means for reusing generated computer graphics and video in future productions. This reduces the efficiency of video production and makes it difficult to quickly respond to user requests.

[0097] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0098] In this invention, the server includes: means for collecting interactive input from users; means for automatically generating computer graphics using a generative AI model; means for combining the generated computer graphics to create a series of videos; means for rendering the videos and transmitting them to a user terminal; means for allowing the user to interactively enter new input while viewing the videos; means for regenerating computer graphics based on the new input and immediately reflecting the new input; and database means for saving the generated computer graphics and videos so that they can be reused in subsequent productions. This allows the user's interactive input to be reflected in the videos in real time, and scene effects and sound effects can also be automatically applied. Furthermore, saving the generated computer graphics and videos and reusing them in subsequent productions can improve the efficiency of video production.

[0099] "User input" means information that a user interactively sets or requests to the system.

[0100] A "generative AI model" refers to an algorithm or system for automatically generating computer graphics based on user input data.

[0101] "Computer graphics" refers to images or pictures generated using a computer, generally including scenes, characters, etc.

[0102] "Interactive input" refers to new operations or requests made by the user in real time while watching a video.

[0103] "Rendering" refers to the process of converting generated computer graphics or a sequence of images into a final form.

[0104] "Database Means" means a system or method for storing generated computer graphics and video for reuse in future productions.

[0105] A "series of images" refers to a series of images created by combining multiple computer graphics.

[0106] "Effects" refers to special visual effects in a scene that are applied automatically based on user input.

[0107] "Sound Effects" means acoustic special effects for a scene that are applied automatically based on user input.

[0108] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) using a generative AI model based on that input, and then combines the generated CG to create a series of images. It also has the function of rendering the images and transmitting them to the user's terminal. If the user provides new input while watching a video, the CG can be regenerated and instantly reflected in the video.

[0109] Program processing overview

[0110] Collecting User Input

[0111] Users input their preferences and requests via their device. This input data includes the atmosphere of the scene, the characteristics of the characters, the type of action, etc. The device collects this information, converts it into an appropriate data format, and sends it to the server.

[0112] CG generation using generative AI models

[0113] The server receives user input data sent from the device and inputs it as a prompt sentence into the generative AI model. The generative AI model automatically generates a CG scene based on the prompt sentence. The generated CG is temporarily stored on the server and used for the next generation process.

[0114] CG scene arrangement and assembly

[0115] The server combines a large number of CG scenes into a series of images. If necessary, it automatically generates and adds effects and sound effects to each scene. It also adjusts the scenes to maintain a natural continuity.

[0116] Video rendering and output

[0117] The server renders the finished video and generates a final data file that is sent to the device and made available for viewing by the user.

[0118] Interactive operation while watching

[0119] While watching the video on their device, users can request scene changes or additions in real time. The device then sends this request to the server, which then inputs the new prompt into the generative AI model to generate a new CG scene. The new scene is then immediately reflected in the video.

[0120] Specific examples

[0121] If a user is creating a horror movie, he / she will select a "dark and scary forest scene" and specify "zombies wearing old clothes." In this case, the system will operate as follows:

[0122] User Input

[0123] The user inputs "horror," "dark forest," "zombie character," and "scary action" into the device, which then sends this information to the server.

[0124] Scene Generation

[0125] The server inputs prompt text into the generative AI model, generating CG scenes including a "dark and scary forest" and "zombie characters."

[0126] Scene arrangement and assembly

[0127] The server then combines the generated CG scenes to construct a scene featuring zombies in a terrifying forest, adding appropriate background sounds and effects.

[0128] Video Output

[0129] The server renders the entire video and sends it as a data file to the terminal, where the user can view the video.

[0130] Interactive Operation

[0131] While watching the video, the user makes a request to change the scene to one in which zombies suddenly appear. The device sends this request to the server, which then generates a new scene and immediately applies it to the video.

[0132] Prompt Sentence Examples

[0133] As a concrete example, the following prompt sentence is input to the generative AI model:

[0134] Scene Description: A dark and scary forest

[0135] Character: Zombie in old clothes

[0136] Action: Zombies suddenly appear

[0137] Special Effects: Music and fog effects to add to the horror in the background

[0138] In this way, the user can interactively enjoy video based on desired scenes and actions.

[0139] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0140] Step 1:

[0141] Collecting User Input

[0142] Users input their preferred scene and character settings via their device. Specifically, they enter the "scene atmosphere," "character characteristics," and "type of action" in the input boxes.

[0143] Input: Information based on user preferences (e.g. "horror", "dark forest", "zombie characters").

[0144] Output: A data file containing the user preferences.

[0145] Specific operation: The user enters information into each field on the terminal interface and clicks a button to send it to the server.

[0146] Step 2:

[0147] Sending user-entered data

[0148] The terminal sends the information entered by the user to the server, which converts the information into an appropriate data format (e.g., a prompt).

[0149] Input: A data file containing user-entered data.

[0150] Output: The prompt text sent to the server.

[0151] Specific operation: The terminal organizes the data, converts it into a prompt sentence format, and sends it to the server via the network.

[0152] Step 3:

[0153] AI-based CG generation

[0154] The server inputs the prompt text sent from the device into a generative AI model and generates CG based on the specified scene and characters.

[0155] Input: Prompt sentence (e.g. "Dark and scary forest" or "Zombie character").

[0156] Output: The generated CG scene.

[0157] Specific operation: The server receives the prompt sentence, inputs it into the algorithm associated with the generative AI model, and obtains a CG scene as the output.

[0158] Step 4:

[0159] CG scene arrangement and assembly

[0160] The server then combines the generated CG scenes to create a series of images, with effects and sound effects applied simultaneously.

[0161] Input: Multiple generated CG scenes.

[0162] Output: A stream of video data.

[0163] Specific operation: The server arranges CG scenes in time and automatically adds effects and music to generate continuous video.

[0164] Step 5:

[0165] Video rendering and output

[0166] The server renders the completed video data and generates the final data file, which is then sent to the user's device.

[0167] Input: A sequence of video data.

[0168] Output: The final rendered video file.

[0169] Specific operation: The server encodes the video data and sends it to the terminal in streaming format.

[0170] Step 6:

[0171] Interactive operation while watching

[0172] While watching the video on the device, the user can make additional scene changes or requests, which will change the scene in real time.

[0173] Input: A new request from the user (e.g., "Add a scene where zombies suddenly appear").

[0174] Output: Updated footage.

[0175] Specific operation: The user inputs a change request into the device, which then sends it to the server, which then generates a new scene and immediately reflects it in the video.

[0176] (Application example 1)

[0177] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0178] In recent years, the retail industry has required advanced interactive technologies to respond quickly and flexibly to diverse customer needs. However, customizing in-store displays and product introductions in real time remains a difficult challenge. For this reason, there is a demand for systems that can instantly change in-store displays and product introductions based on customer preferences. Furthermore, conventional systems have limited interactive operation, making it difficult to increase user satisfaction.

[0179] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0180] In this invention, the server includes means for collecting interactive input from users, means for automatically generating computer graphics based on the input using a computer generation technique, and means for combining the generated computer graphics to create a series of images, thereby enabling in-store displays and product introductions to be customized in real time based on user preferences, thereby improving customer satisfaction.

[0181] "Interactive input" refers to operations and instructions that a user gives to a system in real time.

[0182] "Computer-generated techniques" refers to methods of automatically generating computer graphics using algorithms and artificial intelligence techniques.

[0183] "Computer graphics" refers to visual images and video generated using a computer.

[0184] "Rendering" refers to the process of converting generated computer graphics into a form that can be displayed on a screen.

[0185] "User terminal" refers to a device that is directly operated by a user, such as a smartphone, smart glasses, or computer.

[0186] "Product display" refers to the visual arrangement and presentation of products in a store to highlight their appeal and appeal to customers.

[0187] "In-store decoration" refers to the decorations and layout used to create the atmosphere and theme of the store.

[0188] The system according to the present invention collects interactive input from users and automatically generates computer graphics (CG) based on the input using computer generation techniques, thereby customizing product displays and in-store decorations in real time. Specific embodiments for carrying out the present invention are described below.

[0189] System Overview

[0190] 1. User Input

[0191] Users input their product preferences and requirements to the system in real time using user devices such as smartphones or smart glasses. These inputs are collected based on the users' tastes and purchasing intentions.

[0192] 2. AI-based product scene generation

[0193] The server generates computer graphics (CG) of the product using a generative AI model based on the received user input, such as the OpenAI (registered trademark) GPT model or Unity's ML-agents.

[0194] 3. Scene construction and composition

[0195] The server combines the generated product CG to create visually appealing product displays and in-store decorations for users, using rendering engine services such as Unity and Unreal Engine.

[0196] 4. Rendering and outputting the video

[0197] The server renders the generated CG images of product displays and in-store decorations and sends them to the user's device. The rendering process uses GPU acceleration to ensure real-time performance.

[0198] 5. Interactive Operation

[0199] When a user inputs a new request while browsing products in a store, the server immediately receives the input, regenerates the CG, and updates the display in real time.

[0200] Processing Description

[0201] In this system, the user device uses a smartphone or smart glasses and sends user input to the server. The server processes the received information and generates CG using a generative AI model. This generated CG is then rendered using a rendering engine service such as Unity or Unreal Engine and sent to the user device. When the user performs interactive operations, the server regenerates the CG and immediately responds to the user's new input.

[0202] Specific examples

[0203] For example, if a user wears smart glasses and selects "red casual clothes," the system will generate a CG image of "red clothes" and display it in real time on the glasses' HUD. A possible prompt might look something like this:

[0204] text

[0205] I'm looking for casual clothes in red.

[0206] In this way, the system of the present invention can customize product displays and store décor in real time based on user preferences, providing a more personalized shopping experience.

[0207] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0208] Step 1:

[0209] A user connects to the system using a smartphone or smart glasses and inputs a product request. For example, they input the text "I'm looking for red casual clothes." The input information is processed on the user's terminal and sent to the server. The input of this step is the user's request, and the output is the user request data sent to the server.

[0210] Step 2:

[0211] The server analyzes the received user request data and identifies the user's preferences. This analysis uses natural language processing (NLP) technology. The requested color "red" and category "casual" are extracted. The input is the user request data, and the output is the analyzed user preference data.

[0212] Step 3:

[0213] The server generates computer graphics (CG) using a generative AI model based on the analyzed user preference data. This AI model (for example, an improved version of the OpenAI GPT model) generates product images according to specific requests. The input is the user preference data, and the output is the generated product CG.

[0214] Step 4:

[0215] The server combines the generated product CG to create in-store decorations and product displays. This process uses rendering engine services such as Unity and Unreal Engine. The input is the generated product CG, and the output is the configured in-store decorations and product display CG.

[0216] Step 5:

[0217] The server renders the configured in-store decoration and product display CG and sends it to the user's device. The rendering process uses GPU acceleration to ensure real-time performance. The input is the configured in-store decoration and product display CG, and the output is the rendered product display image.

[0218] Step 6:

[0219] Users browse the store using their smartphones or smart glasses and experience a customized product display in real time. When the user inputs a new request, the server generates a new CG image and immediately sends the updated display information to the user's device. The input for this step is the user's new request, and the output is the updated product display image.

[0220] Step 7:

[0221] The system allows users to view the displayed products in real time and select their next action, such as purchasing the product or browsing other products. This provides an improved interactive shopping experience. The input is the rendered product display image, and the output is the user's next action.

[0222] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0223] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) based on that input using computer generation techniques, and then combines the generated CG to create a series of videos. It also has the ability to render the video and transmit it to the user's device, allowing the user to provide new input while watching, which in turn generates new CG and reflects it in the video. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system provides a more advanced interactive viewing experience.

[0224] System Overview

[0225] 1. User Input

[0226] The device receives interactive settings and requests from the user. Information collected includes the atmosphere of the scene, the characteristics of the characters, and the type of action.

[0227] Additionally, an emotion engine recognizes the user's emotions and provides feedback to the system in real time.

[0228] 2. AI Scene Generation

[0229] The AI ​​generates CG scenes based on the user's requests received by the server, taking into account the results of the emotion engine and generating scenes and effects that match the user's emotions.

[0230] 3. Assembling the Scene

[0231] The server combines multiple generated CG scenes into a series of videos, and scene transitions, effects, and sound effects are also adjusted based on feedback from the emotion engine.

[0232] 4. Video output

[0233] The completed video is rendered on the server and generated as a final data file, which is then sent to the device.

[0234] 5. Interactive Operation

[0235] If a user has a new desire while watching a video on a device, they can interactively input a change request. The emotion engine continuously monitors the user's emotions while watching and dynamically adjusts scenes and effects as needed.

[0236] The server receives the new request, regenerates the necessary CG, and immediately reflects it in the video.

[0237] Specific Examples

[0238] Specific examples of scene generation and emotion recognition

[0239] Suppose a user is creating a horror movie and has selected a "dark and scary forest scene" and has specified that the characters should be "zombies wearing old clothes." In this case, the system operates as follows:

[0240] 1. User Input

[0241] The device collects the user's preferences and sends information such as "horror," "dark forest," "zombie characters," and "scary action" to the server. The emotion engine simultaneously recognizes the user's emotions and provides feedback to the server about their state of mind, such as "tension" or "excitement."

[0242] 2. Scene Generation

[0243] The server uses generative AI to create a dark and frightening CG forest scene, including user-specified zombie characters, with additional effects and sound effects based on the user's emotions.

[0244] 3. Assembling the Scene

[0245] The server assembles a scene featuring zombies in a terrifying forest and adds the necessary background sounds and effects. The darkness and type of sound in the scene are adjusted based on the results of the emotion engine.

[0246] 4. Video output

[0247] The server renders the entire video and generates a file, which is then sent to the device for viewing by the user.

[0248] 5. Interactive Operation

[0249] If a user watches a video on their device and requests a scene where zombies suddenly appear, a new request is sent to the server. The emotion engine continues to monitor the user's emotions, and a new scene is instantly generated and reflected in the video.

[0250] This allows users to create, modify, and view videos that suit their preferences and emotions in real time. Furthermore, the created videos and CG are stored in a database and can be reused for future productions, significantly improving the efficiency of video production and providing a new entertainment experience.

[0251] The processing flow will be explained below.

[0252] Step 1:

[0253] The user inputs the settings and requests for video production through the device interface, for example, selecting parameters such as "horror movie," "dark forest," or "zombie characters." At the same time, the device's built-in emotion recognition function monitors the user's emotional state and collects that data.

[0254] Step 2:

[0255] The terminal collects the user's input and emotional state data, converts it into a data format, and transmits it to the server. The transmitted data includes the user-selected parameters and emotional data.

[0256] Step 3:

[0257] The server analyzes the received data and issues instructions to the AI ​​generator based on each parameter. The instructions also include details of effects and sound effects that take emotional data into account.

[0258] Step 4:

[0259] The server uses AI to generate CG scenes based on the user's settings and emotional state, and effects and sounds corresponding to the user's emotions are automatically applied to the generated CG scenes.

[0260] Step 5:

[0261] The server temporarily stores the generated CG scenes in a database and assigns them a reference ID, allowing each piece of CG data to be reused in later processes.

[0262] Step 6:

[0263] The server retrieves multiple CG data sets stored in a database and combines them to create a series of images. Scene transitions, additional effects, and sound effects are also adjusted in real time based on the emotional data.

[0264] Step 7:

[0265] The server renders the edited footage and stores it as a final digital file, which is then converted into a viewable format.

[0266] Step 8:

[0267] The server sends the rendered video file to the device, and a viewing URL or streaming service link is provided so the user can watch the video.

[0268] Step 9:

[0269] The user watches the video using the device. If a new request arises while watching, for example, "I want to change to a scene where zombies suddenly appear," the user inputs the request into the device.

[0270] Step 10:

[0271] The device sends a new user request to the server, which includes details of which parts of the scene to change and how. The emotion engine continues to monitor the user's emotions and sends the data to the server.

[0272] Step 11:

[0273] The server uses the AI ​​to generate additional CG based on the new request and emotion data. Effects and sound effects based on the emotion data are also applied at the same time.

[0274] Step 12:

[0275] The server then integrates the newly generated scene into the existing footage and re-renders the entire footage.

[0276] Step 13:

[0277] The server transmits the updated video to the terminal again, making the video that reflects the user's new requests and emotions viewable.

[0278] This series of processes allows users to generate, view, and change images in real time according to their preferences and emotions.

[0279] Example 2

[0280] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0281] Conventional video production systems have had difficulty dynamically changing and generating video content based on real-time user feedback and emotions. Furthermore, the technology to recognize users' emotions while they are watching and instantly reflect them in the content of the video has not yet been fully developed. As a result, it has been impossible to provide the interactivity and personalized entertainment experience that users desire.

[0282] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0283] In this invention, the server includes means for collecting interactive input from a user, means for automatically generating computer graphics using a generative AI model, means for combining the generated computer graphics to create a series of videos, means for rendering the videos and transmitting them to a user terminal, means for allowing the user to interactively enter new input while viewing the videos, means for regenerating computer graphics based on the new input and immediately reflecting the input, and means for recognizing the user's emotions in real time and adjusting scenes and effects based on the emotions. This enables users to interactively generate and view videos based on feedback and emotions in real time, providing a personalized entertainment experience.

[0284] A "means for collecting interactive input from a user" is a device or software for receiving user settings, requests, or instructions.

[0285] "Means for automatically generating computer graphics using generative AI models" refers to devices or software that use artificial intelligence technology to automatically generate CG based on user input.

[0286] "Means for combining generated computer graphics to create a series of images" refers to a device or software that connects the individual generated CG images to create a visually continuous image.

[0287] The "means for rendering the video and transmitting it to the user terminal" refers to a device or software for rendering a series of videos in high quality and delivering the data to the terminal used by the user.

[0288] "Means for enabling the user to interactively input new information while viewing the video" refers to a device or software that provides an interface that allows the user to input changes or additional instructions while the video is being played.

[0289] "Means for regenerating computer graphics based on the new input and immediately reflecting it" refers to a device or software that generates additional or modified CG based on the user's new input and immediately incorporates it into the video.

[0290] "Means for recognizing a user's emotions in real time and adjusting scenes and effects based on said emotions" refers to a device or software that determines emotions from the user's facial expressions, voice, etc., and dynamically changes the content of the video and effects according to those emotions.

[0291] The system of this invention collects interactive input from users, automatically generates computer graphics (CG) based on that input, and uses that CG to create a series of images. It also has the function of rendering the generated images and transmitting them to the user's terminal. By providing new input while watching, the user can regenerate CG and instantly reflect it in the images. It can also recognize the user's emotions in real time and adjust scenes and effects based on those emotions.

[0292] Interactive input from the user

[0293] The device interactively receives settings and requests from the user, including the atmosphere of the scene, the characteristics of the characters, and the type of action. For example, if a user selects a "dark and scary forest scene" and specifies "zombies wearing old clothes" as the characters to create a horror movie, this information is collected by the device. The device then uses an emotion engine to analyze the user's emotions in real time and identify emotional states such as "tension" and "excitement."

[0294] Automatic generation of computer graphics

[0295] The server generates CG using a generative AI model based on the data received from the device and the results of the emotion engine. For example, if a user requests a "dark and scary forest scene," the server inputs "a dark and scary forest scene" as a prompt into the generative AI model and generates appropriate CG. This generative AI model uses Stable Diffusion and DALL-E 2, among others.

[0296] CG assembly and sequencing

[0297] The generated CG scenes are then combined by the server to create a series of images. The server then adjusts these, including scene transitions, effects, and sound effects, based on feedback from the emotion engine. For example, the darkness and type of sound in a scene are adjusted appropriately based on the results of the emotion engine.

[0298] Video rendering and output

[0299] The server renders the finished video and generates the final data file using rendering engines such as Blender and Unreal Engine. The rendered video is then sent to the device where it can be viewed by the user.

[0300] Interactive operation and immediate feedback

[0301] If a user has new requests while watching a video on their device, they can interactively input change requests. The emotion engine continuously monitors the user's emotions and dynamically adjusts the scenes and effects as needed. For example, if a user requests, "I want to change the scene to one where zombies suddenly appear," that request is sent to the server. The server then issues a new prompt to the generative AI model, instantly generating new CG and reflecting it in the video. In this way, users can interactively generate and change videos in real time.

[0302] Specific examples

[0303] Imagine a user creating a horror movie. The user inputs the following information through an interface:

[0304] "I want to make a horror movie. I'll choose a dark, scary forest scene and designate zombies in old-fashioned clothing as characters. I want to change the scene so that the zombies suddenly appear while I'm watching."

[0305] Based on this prompt, the system can generate, modify, and view videos in real time that match the user's preferences and emotions. The generated videos and CG are stored in a database and can be reused for future productions, significantly improving the efficiency of video production and providing a new entertainment experience.

[0306] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0307] Step 1: Collecting User Input

[0308] The device receives interactive input from the user, including the atmosphere of a particular scene, the characteristics of characters, the type of action, etc. The emotion engine then operates to obtain the user's emotion data.

[0309] Input: User-defined scene and character requests, emotional data (e.g., "dark and scary forest scene," "zombie characters," "scary action").

[0310] Output: Collected user requests and sentiment data.

[0311] Step 2: Send input data to the server

[0312] The terminal transmits the collected user input data and emotion data to the server.

[0313] Input: Collected user desires and sentiment data.

[0314] Output: User wishes and emotion data sent to the server.

[0315] Step 3: Generate the CG scene

[0316] The server generates CG using a generative AI model based on the received data, taking into account the results of the emotion engine and optimizing scenes and effects to match the user's emotions.

[0317] Input: User desires and emotion data.

[0318] Data processing: Generate CG using generative AI models (e.g., Stable Diffusion or DALL-E 2).

[0319] Output: CG data according to user's request.

[0320] Step 4: Combining and composing the CG scene

[0321] The server combines the generated CG scenes into a series of images, and adjusts scene transitions, effects, and sound effects based on information from the emotion engine.

[0322] Input: Generated CG data and emotion engine results.

[0323] Data processing: Combining CG scenes, scene transitions, adding effects and sound effects.

[0324] Output: A complete video sequence.

[0325] Step 5: Rendering the video and sending it to the device

[0326] The server renders the finished video and generates the final data file, which is then sent to the device for viewing by the user.

[0327] Input: A complete video sequence.

[0328] Data processing: Video rendering (using Blender, Unreal Engine, etc.).

[0329] Output: Rendered video file.

[0330] Step 6: Interact and Regenerate

[0331] If a user has a new request while watching a video on their device, they can input it. The emotion engine monitors the user's emotions and dynamically adjusts scenes and effects as needed. The server receives the new request, regenerates the CG, and immediately reflects it in the video.

[0332] Input: The user's new request and emotion data.

[0333] Data processing: Input new prompt text into the generative AI model and regenerate the necessary CG.

[0334] Output: Updated video data.

[0335] (Application example 2)

[0336] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0337] Current virtual store shopping experiences make it difficult for users to easily find products that suit their preferences or to try on items. Furthermore, there is a lack of systems that can recognize user emotions and dynamically generate content, making it impossible to provide a personalized shopping experience tailored to each individual user. This leads to a decline in user satisfaction and engagement.

[0338] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting interactive input from the user, means for automatically generating computer graphics using a computer generation method based on the input, and emotion recognition means for recognizing the user's emotions and dynamically adjusting images and effects. This enables a personalized shopping experience in a virtual store that is tailored to the user's preferences and emotions.

[0339] "Means for collecting interactive input from the user" refers to an interface through which the user inputs their desired settings and requests and the system receives them.

[0340] "Means for automatically generating computer graphics using computer generation methods" refers to the process of using AI technology to generate images or videos based on user input.

[0341] "A means of combining generated computer graphics to create a series of images" refers to the process of appropriately arranging individual CGs and assembling them into a single continuous video work.

[0342] "Means for rendering the video and transmitting it to the user terminal" refers to the process of converting the generated video into a format that can be displayed with high quality and delivering it to the user's device.

[0343] "Means that allow the user to interactively input new information while watching the video" refers to a function that allows the user to input new requests or changes to the system in real time while watching the video.

[0344] "Means of regenerating computer graphics based on new input and immediately reflecting it" refers to the process of receiving a new user request, generating new CG, and immediately reflecting that content in the video.

[0345] "Emotion recognition means that recognizes the user's emotions and dynamically adjusts images and effects" refers to technology that analyzes the user's emotional state in real time and dynamically changes images and effects accordingly.

[0346] "Avatar management means that enables users to set up virtual avatars and move freely within a virtual space" refers to a function that enables users to set up their own avatars and operate and move those avatars within a virtual space.

[0347] "Means for providing product information and try-on simulations in a virtual space" refers to a function that allows users to check out products in a virtual store environment and virtually try on outfits.

[0348] The system that realizes this invention automatically generates computer graphics based on interactive user input and combines them to create a series of images, providing a personalized shopping experience in a virtual store. The system program and its processing are described below.

[0349] The server collects user input and uses an AI model to generate CG based on it. In this system, users can customize their own avatars using their smartphones and move them freely in the virtual space. The main software and hardware used include smartphones, Emotion API, generation AI (e.g., DALL-E 2), and a rendering engine (e.g., Unity).

[0350] When a user enters their preferred settings through the smartphone interface, those settings are sent to the server. For example, if they select "casual summer clothing," the Emotion API analyzes the user's facial expressions and recognizes their emotions. The server then uses generative AI such as DALL-E 2 to automatically generate the necessary CG based on this emotional information and the user's settings.

[0351] The generated CG is assembled into a virtual store scene using a rendering engine such as Unity. The virtual store scene is equipped with a function that allows the user's avatar to move freely, and product information and a try-on simulation are also provided. The final rendered image is sent to a smartphone device, where the user can view and interact with it.

[0352] For example, when a user clicks on a specific product in a store, detailed information about that product and a try-on simulation are displayed. Furthermore, when the Emotion API recognizes the user's "interest" from their facial expression, they are dynamically guided to products and sections that match their interests.

[0353] Prompt Sentence Examples

[0354] Prompt for the generator: "Generate casual summer clothing for a virtual avatar."

[0355] Prompt for emotion recognition API: "Evaluate user interest from facial expressions and guide them to specific products."

[0356] As described above, this system dynamically generates and adjusts images and effects based on the user's emotions and inputs, providing a personalized virtual shopping experience tailored to each individual user.

[0357] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0358] Step 1:

[0359] A user launches the application on their smartphone and configures their virtual avatar. They upload a photo of their face and customize the avatar's appearance and clothing. This forms the avatar's basic information and the user's preferred settings as input data. The generative AI model is prompted with the following sentence: "Generate casual summer clothing for the virtual avatar."

[0360] Step 2:

[0361] The device sends the avatar data and user preferences to the server. The server receives this data and uses the Emotion API to analyze the user's facial expressions in real time to obtain emotional information. This emotional information is reflected in future CG generation and scene construction. The prompt to the emotion recognition API is: "Please evaluate the user's interest from their facial expressions and guide them to specific products."

[0362] Step 3:

[0363] The server uses a generation AI (e.g., DALL-E 2) to generate the necessary CG in the virtual space based on the user's preferred settings and emotional information. The generation process creates the specified costumes and objects required for the scene and generates them as 3D models. The input for this step is the user's settings and emotional information, and the output is the generated CG data.

[0364] Step 4:

[0365] The server constructs a virtual store scene based on the generated CG data. Using a rendering engine (e.g., Unity), the scene is constructed by combining CG. Within this scene, an area is placed where the user's avatar can move freely, allowing for product display and a try-on simulation. The input for this step is the generated CG data, and the output is rendered scene data.

[0366] Step 5:

[0367] The server sends the rendered scene data to the user's smartphone. The user can then freely walk around the virtual store on their smartphone, browsing product information and trying on items. Selecting a specific item displays detailed information about that item and a try-on simulation. The input for this step is the rendered scene data, and the output is an interactive display on the user's device.

[0368] Step 6:

[0369] Users can interactively request new products or scenes that interest them in the virtual store. The new request from the terminal is sent to the server, which regenerates the necessary CG based on the request and immediately reflects it in the scene. The input of this step is the user's new request data, and the output is the updated scene data.

[0370] This is the flow of the system program that realizes this application example. The specific operations and data processing / calculation performed at each step provide the user with a personalized shopping experience.

[0371] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0372] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0373] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0374] [Second embodiment]

[0375] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0376] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0377] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0378] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0379] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0380] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0381] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0382] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0383] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0384] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0385] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0386] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0387] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) based on that input using computer generation techniques, and then combines the generated CG to create a series of images. It also has the function of rendering the images and transmitting them to the user's terminal. If the user provides new input while watching a video, the system can regenerate the CG and instantly reflect the input.

[0388] System Overview

[0389] 1. User Input

[0390] The device interactively receives preferences and requests from the user.

[0391] The collected information includes, for example, the atmosphere of the scene, the characteristics of the characters, the type of action, and the like.

[0392] 2. AI Scene Generation

[0393] The generation AI generates the CG of the scene based on the user request received by the server.

[0394] The generated CG is temporarily saved and used for the next generation process.

[0395] 3. Scene Arrangement and Composition

[0396] The server combines multiple CG scenes to create a series of images.

[0397] Effects and sound effects are also automatically generated as needed.

[0398] 4. Video output

[0399] The completed video is rendered on the server and generated as a final data file.

[0400] The server sends the video to the terminal so that the user can view it.

[0401] 5. Interactive Operation

[0402] When a user watches a video on a terminal, they can change or select scenes in real time.

[0403] The server immediately receives the user's new operations and regenerates and reflects the CG scene.

[0404] Specific Examples

[0405] Example of scene generation

[0406] Suppose a user is creating a horror movie and has selected a "dark and scary forest scene" and has specified that the characters should be "zombies wearing old clothes." In this case, the system operates as follows:

[0407] 1. User Input

[0408] The device collects the user's preferences and sends information such as "horror," "dark forest," "zombie characters," and "scary action" to the server.

[0409] 2. Scene Generation

[0410] The server uses generative AI to create a dark and scary CG forest scene, which also includes a user-specified zombie character.

[0411] 3. Scene construction

[0412] The server assembles scenes featuring zombies in a terrifying forest and adds appropriate background sounds and effects.

[0413] 4. Video output

[0414] The server renders the entire video and generates it as a file.

[0415] The server transmits the generated video to the terminal so that the user can view it.

[0416] 5. Interactive Operation

[0417] While watching the video on the device, the user makes a request to change the scene to one in which zombies suddenly appear.

[0418] This request is sent from the device to the server, which then generates a new scene and immediately reflects it.

[0419] In this way, users can interactively enjoy videos that suit their preferences and requests. In addition, the generated videos and CG are stored in a database and can be reused for future productions. This will significantly improve the efficiency of video production and enable the provision of new entertainment experiences.

[0420] The processing flow will be explained below.

[0421] Step 1:

[0422] The user inputs the settings and requests required for video production through the device interface, selecting parameters such as "horror movie," "dark forest," and "zombie characters."

[0423] Step 2:

[0424] The terminal collects the user's input, converts it into a data format, and sends it to the server, including all the parameters selected by the user.

[0425] Step 3:

[0426] The server analyzes the received data and issues instructions based on each parameter to the generation AI, which then generates computer graphics (CG) containing the scenes and characters desired by the user.

[0427] Step 4:

[0428] The server temporarily stores the generated CG in a database and assigns it a reference ID. Each saved CG data is used in the next process.

[0429] Step 5:

[0430] The server retrieves multiple CG data sets stored in a database, combines them, and edits them into a series of images. Scene transitions, effects, and sound effects are also added automatically at this stage.

[0431] Step 6:

[0432] The server renders the edited footage and stores it as a final digital file, which is then converted into a viewable format.

[0433] Step 7:

[0434] The server then sends the rendered video file to the device where the user can view it. A viewing URL or a link to a streaming service may be provided.

[0435] Step 8:

[0436] The user watches the video using the terminal. If a new request arises while watching, the user can input a request such as "I want to change to a scene where zombies suddenly appear."

[0437] Step 9:

[0438] The device sends a new user request to the server, which includes details of what part of the scene to change and how.

[0439] Step 10:

[0440] The server receives the new request, again using the generative AI to generate any additional CG needed, then adds the new scene in the appropriate location and re-renders the entire video.

[0441] Step 11:

[0442] The server transmits the updated video to the terminal again, making the video that reflects the user's new request available for viewing.

[0443] This series of processes allows users to generate, view, and change videos that suit their preferences in real time.

[0444] Example 1

[0445] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0446] Current video production systems have difficulty in instantly reflecting user interactive input and have limitations in automatically applying scene and sound effects. They also lack appropriate means for reusing generated computer graphics and video in future productions. This reduces the efficiency of video production and makes it difficult to quickly respond to user requests.

[0447] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0448] In this invention, the server includes: means for collecting interactive input from users; means for automatically generating computer graphics using a generative AI model; means for combining the generated computer graphics to create a series of videos; means for rendering the videos and transmitting them to a user terminal; means for allowing the user to interactively enter new input while viewing the videos; means for regenerating computer graphics based on the new input and immediately reflecting the new input; and database means for saving the generated computer graphics and videos so that they can be reused in subsequent productions. This allows the user's interactive input to be reflected in the videos in real time, and scene effects and sound effects can also be automatically applied. Furthermore, saving the generated computer graphics and videos and reusing them in subsequent productions can improve the efficiency of video production.

[0449] "User input" means information that a user interactively sets or requests to the system.

[0450] A "generative AI model" refers to an algorithm or system for automatically generating computer graphics based on user input data.

[0451] "Computer graphics" refers to images or pictures generated using a computer, generally including scenes, characters, etc.

[0452] "Interactive input" refers to new operations or requests made by the user in real time while watching a video.

[0453] "Rendering" refers to the process of converting generated computer graphics or a sequence of images into a final form.

[0454] "Database Means" means a system or method for storing generated computer graphics and video for reuse in future productions.

[0455] A "series of images" refers to a series of images created by combining multiple computer graphics.

[0456] "Effects" refers to special visual effects in a scene that are applied automatically based on user input.

[0457] "Sound Effects" means acoustic special effects for a scene that are applied automatically based on user input.

[0458] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) using a generative AI model based on that input, and then combines the generated CG to create a series of images. It also has the function of rendering the images and transmitting them to the user's terminal. If the user provides new input while watching a video, the CG can be regenerated and instantly reflected in the video.

[0459] Program processing overview

[0460] Collecting User Input

[0461] Users input their preferences and requests via their device. This input data includes the atmosphere of the scene, the characteristics of the characters, the type of action, etc. The device collects this information, converts it into an appropriate data format, and sends it to the server.

[0462] CG generation using generative AI models

[0463] The server receives user input data sent from the device and inputs it as a prompt sentence into the generative AI model. The generative AI model automatically generates a CG scene based on the prompt sentence. The generated CG is temporarily stored on the server and used for the next generation process.

[0464] CG scene arrangement and assembly

[0465] The server combines a large number of CG scenes into a series of images. If necessary, it automatically generates and adds effects and sound effects to each scene. It also adjusts the scenes to maintain a natural continuity.

[0466] Video rendering and output

[0467] The server renders the finished video and generates a final data file that is sent to the device and made available for viewing by the user.

[0468] Interactive operation while watching

[0469] While watching the video on their device, users can request scene changes or additions in real time. The device then sends this request to the server, which then inputs the new prompt into the generative AI model to generate a new CG scene. The new scene is then immediately reflected in the video.

[0470] Specific examples

[0471] If a user is creating a horror movie, he / she will select a "dark and scary forest scene" and specify "zombies wearing old clothes." In this case, the system will operate as follows:

[0472] User Input

[0473] The user inputs "horror," "dark forest," "zombie character," and "scary action" into the device, which then sends this information to the server.

[0474] Scene Generation

[0475] The server inputs prompt text into the generative AI model, generating CG scenes including a "dark and scary forest" and "zombie characters."

[0476] Scene arrangement and assembly

[0477] The server then combines the generated CG scenes to construct a scene featuring zombies in a terrifying forest, adding appropriate background sounds and effects.

[0478] Video Output

[0479] The server renders the entire video and sends it as a data file to the terminal, where the user can view the video.

[0480] Interactive Operation

[0481] While watching the video, the user makes a request to change the scene to one in which zombies suddenly appear. The device sends this request to the server, which then generates a new scene and immediately applies it to the video.

[0482] Prompt Sentence Examples

[0483] As a concrete example, the following prompt sentence is input to the generative AI model:

[0484] Scene Description: A dark and scary forest

[0485] Character: Zombie in old clothes

[0486] Action: Zombies suddenly appear

[0487] Special Effects: Music and fog effects to add to the horror in the background

[0488] In this way, the user can interactively enjoy video based on desired scenes and actions.

[0489] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0490] Step 1:

[0491] Collecting User Input

[0492] Users input their preferred scene and character settings via their device. Specifically, they enter the "scene atmosphere," "character characteristics," and "type of action" in the input boxes.

[0493] Input: Information based on user preferences (e.g. "horror", "dark forest", "zombie characters").

[0494] Output: A data file containing the user preferences.

[0495] Specific operation: The user enters information into each field on the terminal interface and clicks a button to send it to the server.

[0496] Step 2:

[0497] Sending user-entered data

[0498] The terminal sends the information entered by the user to the server, which converts the information into an appropriate data format (e.g., a prompt).

[0499] Input: A data file containing user-entered data.

[0500] Output: The prompt text sent to the server.

[0501] Specific operation: The terminal organizes the data, converts it into a prompt sentence format, and sends it to the server via the network.

[0502] Step 3:

[0503] AI-based CG generation

[0504] The server inputs the prompt text sent from the device into a generative AI model and generates CG based on the specified scene and characters.

[0505] Input: Prompt sentence (e.g. "Dark and scary forest" or "Zombie character").

[0506] Output: The generated CG scene.

[0507] Specific operation: The server receives the prompt sentence, inputs it into the algorithm associated with the generative AI model, and obtains a CG scene as the output.

[0508] Step 4:

[0509] CG scene arrangement and assembly

[0510] The server then combines the generated CG scenes to create a series of images, with effects and sound effects applied simultaneously.

[0511] Input: Multiple generated CG scenes.

[0512] Output: A stream of video data.

[0513] Specific operation: The server arranges CG scenes in time and automatically adds effects and music to generate continuous video.

[0514] Step 5:

[0515] Video rendering and output

[0516] The server renders the completed video data and generates the final data file, which is then sent to the user's device.

[0517] Input: A sequence of video data.

[0518] Output: The final rendered video file.

[0519] Specific operation: The server encodes the video data and sends it to the terminal in streaming format.

[0520] Step 6:

[0521] Interactive operation while watching

[0522] While watching the video on the device, the user can make additional scene changes or requests, which will change the scene in real time.

[0523] Input: A new request from the user (e.g., "Add a scene where zombies suddenly appear").

[0524] Output: Updated footage.

[0525] Specific operation: The user inputs a change request into the device, which then sends it to the server, which then generates a new scene and immediately reflects it in the video.

[0526] (Application example 1)

[0527] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0528] In recent years, the retail industry has required advanced interactive technologies to respond quickly and flexibly to diverse customer needs. However, customizing in-store displays and product introductions in real time remains a difficult challenge. For this reason, there is a demand for systems that can instantly change in-store displays and product introductions based on customer preferences. Furthermore, conventional systems have limited interactive operation, making it difficult to increase user satisfaction.

[0529] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0530] In this invention, the server includes means for collecting interactive input from users, means for automatically generating computer graphics based on the input using a computer generation technique, and means for combining the generated computer graphics to create a series of images, thereby enabling in-store displays and product introductions to be customized in real time based on user preferences, thereby improving customer satisfaction.

[0531] "Interactive input" refers to operations and instructions that a user gives to a system in real time.

[0532] "Computer-generated techniques" refers to methods of automatically generating computer graphics using algorithms and artificial intelligence techniques.

[0533] "Computer graphics" refers to visual images and video generated using a computer.

[0534] "Rendering" refers to the process of converting generated computer graphics into a form that can be displayed on a screen.

[0535] "User terminal" refers to a device that is directly operated by a user, such as a smartphone, smart glasses, or computer.

[0536] "Product display" refers to the visual arrangement and presentation of products in a store to highlight their appeal and appeal to customers.

[0537] "In-store decoration" refers to the decorations and layout used to create the atmosphere and theme of the store.

[0538] The system according to the present invention collects interactive input from users and automatically generates computer graphics (CG) based on the input using computer generation techniques, thereby customizing product displays and in-store decorations in real time. Specific embodiments for carrying out the present invention are described below.

[0539] System Overview

[0540] 1. User Input

[0541] Users input their product preferences and requirements to the system in real time using user devices such as smartphones or smart glasses. These inputs are collected based on the users' tastes and purchasing intentions.

[0542] 2. AI-based product scene generation

[0543] The server generates computer graphics (CG) of the product using a generative AI model based on the received user input, such as the OpenAI GPT model or Unity's ML-agents.

[0544] 3. Scene construction and composition

[0545] The server combines the generated product CG to create visually appealing product displays and in-store decorations for users, using rendering engine services such as Unity and Unreal Engine.

[0546] 4. Rendering and outputting the video

[0547] The server renders the generated CG images of product displays and in-store decorations and sends them to the user's device. The rendering process uses GPU acceleration to ensure real-time performance.

[0548] 5. Interactive Operation

[0549] When a user inputs a new request while browsing products in a store, the server immediately receives the input, regenerates the CG, and updates the display in real time.

[0550] Processing Description

[0551] In this system, the user device uses a smartphone or smart glasses and sends user input to the server. The server processes the received information and generates CG using a generative AI model. This generated CG is then rendered using a rendering engine service such as Unity or Unreal Engine and sent to the user device. When the user performs interactive operations, the server regenerates the CG and immediately responds to the user's new input.

[0552] Specific examples

[0553] For example, if a user wears smart glasses and selects "red casual clothes," the system will generate a CG image of "red clothes" and display it in real time on the glasses' HUD. A possible prompt might look something like this:

[0554] text

[0555] I'm looking for casual clothes in red.

[0556] In this way, the system of the present invention can customize product displays and store décor in real time based on user preferences, providing a more personalized shopping experience.

[0557] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0558] Step 1:

[0559] A user connects to the system using a smartphone or smart glasses and inputs a product request. For example, they input the text "I'm looking for red casual clothes." The input information is processed on the user's terminal and sent to the server. The input of this step is the user's request, and the output is the user request data sent to the server.

[0560] Step 2:

[0561] The server analyzes the received user request data and identifies the user's preferences. This analysis uses natural language processing (NLP) technology. The requested color "red" and category "casual" are extracted. The input is the user request data, and the output is the analyzed user preference data.

[0562] Step 3:

[0563] The server generates computer graphics (CG) using a generative AI model based on the analyzed user preference data. This AI model (for example, an improved version of the OpenAI GPT model) generates product images according to specific requests. The input is the user preference data, and the output is the generated product CG.

[0564] Step 4:

[0565] The server combines the generated product CG to create in-store decorations and product displays. This process uses rendering engine services such as Unity and Unreal Engine. The input is the generated product CG, and the output is the configured in-store decorations and product display CG.

[0566] Step 5:

[0567] The server renders the configured in-store decoration and product display CG and sends it to the user's device. The rendering process uses GPU acceleration to ensure real-time performance. The input is the configured in-store decoration and product display CG, and the output is the rendered product display image.

[0568] Step 6:

[0569] Users browse the store using their smartphones or smart glasses and experience a customized product display in real time. When the user inputs a new request, the server generates a new CG image and immediately sends the updated display information to the user's device. The input for this step is the user's new request, and the output is the updated product display image.

[0570] Step 7:

[0571] The system allows users to view the displayed products in real time and select their next action, such as purchasing the product or browsing other products. This provides an improved interactive shopping experience. The input is the rendered product display image, and the output is the user's next action.

[0572] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0573] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) based on that input using computer generation techniques, and then combines the generated CG to create a series of videos. It also has the ability to render the video and transmit it to the user's device, allowing the user to provide new input while watching, which in turn generates new CG and reflects it in the video. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system provides a more advanced interactive viewing experience.

[0574] System Overview

[0575] 1. User Input

[0576] The device receives interactive settings and requests from the user. Information collected includes the atmosphere of the scene, the characteristics of the characters, and the type of action.

[0577] Additionally, an emotion engine recognizes the user's emotions and provides feedback to the system in real time.

[0578] 2. AI Scene Generation

[0579] The AI ​​generates CG scenes based on the user's requests received by the server, taking into account the results of the emotion engine and generating scenes and effects that match the user's emotions.

[0580] 3. Assembling the Scene

[0581] The server combines multiple generated CG scenes into a series of videos, and scene transitions, effects, and sound effects are also adjusted based on feedback from the emotion engine.

[0582] 4. Video output

[0583] The completed video is rendered on the server and generated as a final data file, which is then sent to the device.

[0584] 5. Interactive Operation

[0585] If a user has a new desire while watching a video on a device, they can interactively input a change request. The emotion engine continuously monitors the user's emotions while watching and dynamically adjusts scenes and effects as needed.

[0586] The server receives the new request, regenerates the necessary CG, and immediately reflects it in the video.

[0587] Specific Examples

[0588] Specific examples of scene generation and emotion recognition

[0589] Suppose a user is creating a horror movie and has selected a "dark and scary forest scene" and has specified that the characters should be "zombies wearing old clothes." In this case, the system operates as follows:

[0590] 1. User Input

[0591] The device collects the user's preferences and sends information such as "horror," "dark forest," "zombie characters," and "scary action" to the server. The emotion engine simultaneously recognizes the user's emotions and provides feedback to the server about their state of mind, such as "tension" or "excitement."

[0592] 2. Scene Generation

[0593] The server uses generative AI to create a dark and frightening CG forest scene, including user-specified zombie characters, with additional effects and sound effects based on the user's emotions.

[0594] 3. Assembling the Scene

[0595] The server assembles a scene featuring zombies in a terrifying forest and adds the necessary background sounds and effects. The darkness and type of sound in the scene are adjusted based on the results of the emotion engine.

[0596] 4. Video output

[0597] The server renders the entire video and generates a file, which is then sent to the device for viewing by the user.

[0598] 5. Interactive Operation

[0599] If a user watches a video on their device and requests a scene where zombies suddenly appear, a new request is sent to the server. The emotion engine continues to monitor the user's emotions, and a new scene is instantly generated and reflected in the video.

[0600] This allows users to create, modify, and view videos that suit their preferences and emotions in real time. Furthermore, the created videos and CG are stored in a database and can be reused for future productions, significantly improving the efficiency of video production and providing a new entertainment experience.

[0601] The processing flow will be explained below.

[0602] Step 1:

[0603] The user inputs the settings and requests for video production through the device interface, for example, selecting parameters such as "horror movie," "dark forest," or "zombie characters." At the same time, the device's built-in emotion recognition function monitors the user's emotional state and collects that data.

[0604] Step 2:

[0605] The terminal collects the user's input and emotional state data, converts it into a data format, and transmits it to the server. The transmitted data includes the user-selected parameters and emotional data.

[0606] Step 3:

[0607] The server analyzes the received data and issues instructions to the AI ​​generator based on each parameter. The instructions also include details of effects and sound effects that take emotional data into account.

[0608] Step 4:

[0609] The server uses AI to generate CG scenes based on the user's settings and emotional state, and effects and sounds corresponding to the user's emotions are automatically applied to the generated CG scenes.

[0610] Step 5:

[0611] The server temporarily stores the generated CG scenes in a database and assigns them a reference ID, allowing each piece of CG data to be reused in later processes.

[0612] Step 6:

[0613] The server retrieves multiple CG data sets stored in a database and combines them to create a series of images. Scene transitions, additional effects, and sound effects are also adjusted in real time based on the emotional data.

[0614] Step 7:

[0615] The server renders the edited footage and stores it as a final digital file, which is then converted into a viewable format.

[0616] Step 8:

[0617] The server sends the rendered video file to the device, and a viewing URL or streaming service link is provided so the user can watch the video.

[0618] Step 9:

[0619] The user watches the video using the device. If a new request arises while watching, for example, "I want to change to a scene where zombies suddenly appear," the user inputs the request into the device.

[0620] Step 10:

[0621] The device sends a new user request to the server, which includes details of which parts of the scene to change and how. The emotion engine continues to monitor the user's emotions and sends the data to the server.

[0622] Step 11:

[0623] The server uses the AI ​​to generate additional CG based on the new request and emotion data. Effects and sound effects based on the emotion data are also applied at the same time.

[0624] Step 12:

[0625] The server then integrates the newly generated scene into the existing footage and re-renders the entire footage.

[0626] Step 13:

[0627] The server transmits the updated video to the terminal again, making the video that reflects the user's new requests and emotions viewable.

[0628] This series of processes allows users to generate, view, and change images in real time according to their preferences and emotions.

[0629] Example 2

[0630] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0631] Conventional video production systems have had difficulty dynamically changing and generating video content based on real-time user feedback and emotions. Furthermore, the technology to recognize users' emotions while they are watching and instantly reflect them in the content of the video has not yet been fully developed. As a result, it has been impossible to provide the interactivity and personalized entertainment experience that users desire.

[0632] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0633] In this invention, the server includes means for collecting interactive input from a user, means for automatically generating computer graphics using a generative AI model, means for combining the generated computer graphics to create a series of videos, means for rendering the videos and transmitting them to a user terminal, means for allowing the user to interactively enter new input while viewing the videos, means for regenerating computer graphics based on the new input and immediately reflecting the input, and means for recognizing the user's emotions in real time and adjusting scenes and effects based on the emotions. This enables users to interactively generate and view videos based on feedback and emotions in real time, providing a personalized entertainment experience.

[0634] A "means for collecting interactive input from a user" is a device or software for receiving user settings, requests, or instructions.

[0635] "Means for automatically generating computer graphics using generative AI models" refers to devices or software that use artificial intelligence technology to automatically generate CG based on user input.

[0636] "Means for combining generated computer graphics to create a series of images" refers to a device or software that connects the individual generated CG images to create a visually continuous image.

[0637] The "means for rendering the video and transmitting it to the user terminal" refers to a device or software for rendering a series of videos in high quality and delivering the data to the terminal used by the user.

[0638] "Means for enabling the user to interactively input new information while viewing the video" refers to a device or software that provides an interface that allows the user to input changes or additional instructions while the video is being played.

[0639] "Means for regenerating computer graphics based on the new input and immediately reflecting it" refers to a device or software that generates additional or modified CG based on the user's new input and immediately incorporates it into the video.

[0640] "Means for recognizing a user's emotions in real time and adjusting scenes and effects based on said emotions" refers to a device or software that determines emotions from the user's facial expressions, voice, etc., and dynamically changes the content of the video and effects according to those emotions.

[0641] The system of this invention collects interactive input from users, automatically generates computer graphics (CG) based on that input, and uses that CG to create a series of images. It also has the function of rendering the generated images and transmitting them to the user's terminal. By providing new input while watching, the user can regenerate CG and instantly reflect it in the images. It can also recognize the user's emotions in real time and adjust scenes and effects based on those emotions.

[0642] Interactive input from the user

[0643] The device interactively receives settings and requests from the user, including the atmosphere of the scene, the characteristics of the characters, and the type of action. For example, if a user selects a "dark and scary forest scene" and specifies "zombies wearing old clothes" as the characters to create a horror movie, this information is collected by the device. The device then uses an emotion engine to analyze the user's emotions in real time and identify emotional states such as "tension" and "excitement."

[0644] Automatic generation of computer graphics

[0645] The server generates CG using a generative AI model based on the data received from the device and the results of the emotion engine. For example, if a user requests a "dark and scary forest scene," the server inputs "a dark and scary forest scene" as a prompt into the generative AI model and generates appropriate CG. This generative AI model uses Stable Diffusion and DALL-E 2, among others.

[0646] CG assembly and sequencing

[0647] The generated CG scenes are then combined by the server to create a series of images. The server then adjusts these, including scene transitions, effects, and sound effects, based on feedback from the emotion engine. For example, the darkness and type of sound in a scene are adjusted appropriately based on the results of the emotion engine.

[0648] Video rendering and output

[0649] The server renders the finished video and generates the final data file using rendering engines such as Blender and Unreal Engine. The rendered video is then sent to the device where it can be viewed by the user.

[0650] Interactive operation and immediate feedback

[0651] If a user has new requests while watching a video on their device, they can interactively input change requests. The emotion engine continuously monitors the user's emotions and dynamically adjusts the scenes and effects as needed. For example, if a user requests, "I want to change the scene to one where zombies suddenly appear," that request is sent to the server. The server then issues a new prompt to the generative AI model, instantly generating new CG and reflecting it in the video. In this way, users can interactively generate and change videos in real time.

[0652] Specific examples

[0653] Imagine a user creating a horror movie. The user inputs the following information through an interface:

[0654] "I want to make a horror movie. I'll choose a dark, scary forest scene and designate zombies in old-fashioned clothing as characters. I want to change the scene so that the zombies suddenly appear while I'm watching."

[0655] Based on this prompt, the system can generate, modify, and view videos in real time that match the user's preferences and emotions. The generated videos and CG are stored in a database and can be reused for future productions, significantly improving the efficiency of video production and providing a new entertainment experience.

[0656] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0657] Step 1: Collecting User Input

[0658] The device receives interactive input from the user, including the atmosphere of a particular scene, the characteristics of characters, the type of action, etc. The emotion engine then operates to obtain the user's emotion data.

[0659] Input: User-defined scene and character requests, emotional data (e.g., "dark and scary forest scene," "zombie characters," "scary action").

[0660] Output: Collected user requests and sentiment data.

[0661] Step 2: Send input data to the server

[0662] The terminal transmits the collected user input data and emotion data to the server.

[0663] Input: Collected user desires and sentiment data.

[0664] Output: User wishes and emotion data sent to the server.

[0665] Step 3: Generate the CG scene

[0666] The server generates CG using a generative AI model based on the received data, taking into account the results of the emotion engine and optimizing scenes and effects to match the user's emotions.

[0667] Input: User desires and emotion data.

[0668] Data processing: Generate CG using generative AI models (e.g., Stable Diffusion or DALL-E 2).

[0669] Output: CG data according to user's request.

[0670] Step 4: Combining and composing the CG scene

[0671] The server combines the generated CG scenes into a series of images, and adjusts scene transitions, effects, and sound effects based on information from the emotion engine.

[0672] Input: Generated CG data and emotion engine results.

[0673] Data processing: Combining CG scenes, scene transitions, adding effects and sound effects.

[0674] Output: A complete video sequence.

[0675] Step 5: Rendering the video and sending it to the device

[0676] The server renders the finished video and generates the final data file, which is then sent to the device for viewing by the user.

[0677] Input: A complete video sequence.

[0678] Data processing: Video rendering (using Blender, Unreal Engine, etc.).

[0679] Output: Rendered video file.

[0680] Step 6: Interact and Regenerate

[0681] If a user has a new request while watching a video on their device, they can input it. The emotion engine monitors the user's emotions and dynamically adjusts scenes and effects as needed. The server receives the new request, regenerates the CG, and immediately reflects it in the video.

[0682] Input: The user's new request and emotion data.

[0683] Data processing: Input new prompt text into the generative AI model and regenerate the necessary CG.

[0684] Output: Updated video data.

[0685] (Application example 2)

[0686] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0687] Current virtual store shopping experiences make it difficult for users to easily find products that suit their preferences or to try on items. Furthermore, there is a lack of systems that can recognize user emotions and dynamically generate content, making it impossible to provide a personalized shopping experience tailored to each individual user. This leads to a decline in user satisfaction and engagement.

[0688] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting interactive input from the user, means for automatically generating computer graphics using a computer generation method based on the input, and emotion recognition means for recognizing the user's emotions and dynamically adjusting images and effects. This enables a personalized shopping experience in a virtual store that is tailored to the user's preferences and emotions.

[0689] "Means for collecting interactive input from the user" refers to an interface through which the user inputs their desired settings and requests and the system receives them.

[0690] "Means for automatically generating computer graphics using computer generation methods" refers to the process of using AI technology to generate images or videos based on user input.

[0691] "A means of combining generated computer graphics to create a series of images" refers to the process of appropriately arranging individual CGs and assembling them into a single continuous video work.

[0692] "Means for rendering the video and transmitting it to the user terminal" refers to the process of converting the generated video into a format that can be displayed with high quality and delivering it to the user's device.

[0693] "Means that allow the user to interactively input new information while watching the video" refers to a function that allows the user to input new requests or changes to the system in real time while watching the video.

[0694] "Means of regenerating computer graphics based on new input and immediately reflecting it" refers to the process of receiving a new user request, generating new CG, and immediately reflecting that content in the video.

[0695] "Emotion recognition means that recognizes the user's emotions and dynamically adjusts images and effects" refers to technology that analyzes the user's emotional state in real time and dynamically changes images and effects accordingly.

[0696] "Avatar management means that enables users to set up virtual avatars and move freely within a virtual space" refers to a function that enables users to set up their own avatars and operate and move those avatars within a virtual space.

[0697] "Means for providing product information and try-on simulations in a virtual space" refers to a function that allows users to check out products in a virtual store environment and virtually try on outfits.

[0698] The system that realizes this invention automatically generates computer graphics based on interactive user input and combines them to create a series of images, providing a personalized shopping experience in a virtual store. The system program and its processing are described below.

[0699] The server collects user input and uses an AI model to generate CG based on it. In this system, users can customize their own avatars using their smartphones and move them freely in the virtual space. The main software and hardware used include smartphones, Emotion API, generation AI (e.g., DALL-E 2), and a rendering engine (e.g., Unity).

[0700] When a user enters their preferred settings through the smartphone interface, those settings are sent to the server. For example, if they select "casual summer clothing," the Emotion API analyzes the user's facial expressions and recognizes their emotions. The server then uses generative AI such as DALL-E 2 to automatically generate the necessary CG based on this emotional information and the user's settings.

[0701] The generated CG is assembled into a virtual store scene using a rendering engine such as Unity. The virtual store scene is equipped with a function that allows the user's avatar to move freely, and product information and a try-on simulation are also provided. The final rendered image is sent to a smartphone device, where the user can view and interact with it.

[0702] For example, when a user clicks on a specific product in a store, detailed information about that product and a try-on simulation are displayed. Furthermore, when the Emotion API recognizes the user's "interest" from their facial expression, they are dynamically guided to products and sections that match their interests.

[0703] Prompt Sentence Examples

[0704] Prompt for the generator: "Generate casual summer clothing for a virtual avatar."

[0705] Prompt for emotion recognition API: "Evaluate user interest from facial expressions and guide them to specific products."

[0706] As described above, this system dynamically generates and adjusts images and effects based on the user's emotions and inputs, providing a personalized virtual shopping experience tailored to each individual user.

[0707] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0708] Step 1:

[0709] A user launches the application on their smartphone and configures their virtual avatar. They upload a photo of their face and customize the avatar's appearance and clothing. This forms the avatar's basic information and the user's preferred settings as input data. The generative AI model is prompted with the following sentence: "Generate casual summer clothing for the virtual avatar."

[0710] Step 2:

[0711] The device sends the avatar data and user preferences to the server. The server receives this data and uses the Emotion API to analyze the user's facial expressions in real time to obtain emotional information. This emotional information is reflected in future CG generation and scene construction. The prompt to the emotion recognition API is: "Please evaluate the user's interest from their facial expressions and guide them to specific products."

[0712] Step 3:

[0713] The server uses a generation AI (e.g., DALL-E 2) to generate the necessary CG in the virtual space based on the user's preferred settings and emotional information. The generation process creates the specified costumes and objects required for the scene and generates them as 3D models. The input for this step is the user's settings and emotional information, and the output is the generated CG data.

[0714] Step 4:

[0715] The server constructs a virtual store scene based on the generated CG data. Using a rendering engine (e.g., Unity), the scene is constructed by combining CG. Within this scene, an area is placed where the user's avatar can move freely, allowing for product display and a try-on simulation. The input for this step is the generated CG data, and the output is rendered scene data.

[0716] Step 5:

[0717] The server sends the rendered scene data to the user's smartphone. The user can then freely walk around the virtual store on their smartphone, browsing product information and trying on items. Selecting a specific item displays detailed information about that item and a try-on simulation. The input for this step is the rendered scene data, and the output is an interactive display on the user's device.

[0718] Step 6:

[0719] Users can interactively request new products or scenes that interest them in the virtual store. The new request from the terminal is sent to the server, which regenerates the necessary CG based on the request and immediately reflects it in the scene. The input of this step is the user's new request data, and the output is the updated scene data.

[0720] This is the flow of the system program that realizes this application example. The specific operations and data processing / calculation performed at each step provide the user with a personalized shopping experience.

[0721] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0722] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0723] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0724] [Third embodiment]

[0725] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0726] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0727] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0728] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0729] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0730] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0731] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0732] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0733] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0734] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0735] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0736] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0737] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) based on that input using computer generation techniques, and then combines the generated CG to create a series of images. It also has the function of rendering the images and transmitting them to the user's terminal. If the user provides new input while watching a video, the system can regenerate the CG and instantly reflect the input.

[0738] System Overview

[0739] 1. User Input

[0740] The device interactively receives preferences and requests from the user.

[0741] The collected information includes, for example, the atmosphere of the scene, the characteristics of the characters, the type of action, and the like.

[0742] 2. AI Scene Generation

[0743] The generation AI generates the CG of the scene based on the user request received by the server.

[0744] The generated CG is temporarily saved and used for the next generation process.

[0745] 3. Scene Arrangement and Composition

[0746] The server combines multiple CG scenes to create a series of images.

[0747] Effects and sound effects are also automatically generated as needed.

[0748] 4. Video output

[0749] The completed video is rendered on the server and generated as a final data file.

[0750] The server sends the video to the terminal so that the user can view it.

[0751] 5. Interactive Operation

[0752] When a user watches a video on a terminal, they can change or select scenes in real time.

[0753] The server immediately receives the user's new operations and regenerates and reflects the CG scene.

[0754] Specific Examples

[0755] Example of scene generation

[0756] Suppose a user is creating a horror movie and has selected a "dark and scary forest scene" and has specified that the characters should be "zombies wearing old clothes." In this case, the system operates as follows:

[0757] 1. User Input

[0758] The device collects the user's preferences and sends information such as "horror," "dark forest," "zombie characters," and "scary action" to the server.

[0759] 2. Scene Generation

[0760] The server uses generative AI to create a dark and scary CG forest scene, which also includes a user-specified zombie character.

[0761] 3. Scene construction

[0762] The server assembles scenes featuring zombies in a terrifying forest and adds appropriate background sounds and effects.

[0763] 4. Video output

[0764] The server renders the entire video and generates it as a file.

[0765] The server transmits the generated video to the terminal so that the user can view it.

[0766] 5. Interactive Operation

[0767] While watching the video on the device, the user makes a request to change the scene to one in which zombies suddenly appear.

[0768] This request is sent from the device to the server, which then generates a new scene and immediately reflects it.

[0769] In this way, users can interactively enjoy videos that suit their preferences and requests. In addition, the generated videos and CG are stored in a database and can be reused for future productions. This will significantly improve the efficiency of video production and enable the provision of new entertainment experiences.

[0770] The processing flow will be explained below.

[0771] Step 1:

[0772] The user inputs the settings and requests required for video production through the device interface, selecting parameters such as "horror movie," "dark forest," and "zombie characters."

[0773] Step 2:

[0774] The terminal collects the user's input, converts it into a data format, and sends it to the server, including all the parameters selected by the user.

[0775] Step 3:

[0776] The server analyzes the received data and issues instructions based on each parameter to the generation AI, which then generates computer graphics (CG) containing the scenes and characters desired by the user.

[0777] Step 4:

[0778] The server temporarily stores the generated CG in a database and assigns it a reference ID. Each saved CG data is used in the next process.

[0779] Step 5:

[0780] The server retrieves multiple CG data sets stored in a database, combines them, and edits them into a series of images. Scene transitions, effects, and sound effects are also added automatically at this stage.

[0781] Step 6:

[0782] The server renders the edited footage and stores it as a final digital file, which is then converted into a viewable format.

[0783] Step 7:

[0784] The server then sends the rendered video file to the device where the user can view it. A viewing URL or a link to a streaming service may be provided.

[0785] Step 8:

[0786] The user watches the video using the terminal. If a new request arises while watching, the user can input a request such as "I want to change to a scene where zombies suddenly appear."

[0787] Step 9:

[0788] The device sends a new user request to the server, which includes details of what part of the scene to change and how.

[0789] Step 10:

[0790] The server receives the new request, again using the generative AI to generate any additional CG needed, then adds the new scene in the appropriate location and re-renders the entire video.

[0791] Step 11:

[0792] The server transmits the updated video to the terminal again, making the video that reflects the user's new request available for viewing.

[0793] This series of processes allows users to generate, view, and change videos that suit their preferences in real time.

[0794] Example 1

[0795] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0796] Current video production systems have difficulty in instantly reflecting user interactive input and have limitations in automatically applying scene and sound effects. They also lack appropriate means for reusing generated computer graphics and video in future productions. This reduces the efficiency of video production and makes it difficult to quickly respond to user requests.

[0797] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0798] In this invention, the server includes: means for collecting interactive input from users; means for automatically generating computer graphics using a generative AI model; means for combining the generated computer graphics to create a series of videos; means for rendering the videos and transmitting them to a user terminal; means for allowing the user to interactively enter new input while viewing the videos; means for regenerating computer graphics based on the new input and immediately reflecting the new input; and database means for saving the generated computer graphics and videos so that they can be reused in subsequent productions. This allows the user's interactive input to be reflected in the videos in real time, and scene effects and sound effects can also be automatically applied. Furthermore, saving the generated computer graphics and videos and reusing them in subsequent productions can improve the efficiency of video production.

[0799] "User input" means information that a user interactively sets or requests to the system.

[0800] A "generative AI model" refers to an algorithm or system for automatically generating computer graphics based on user input data.

[0801] "Computer graphics" refers to images or pictures generated using a computer, generally including scenes, characters, etc.

[0802] "Interactive input" refers to new operations or requests made by the user in real time while watching a video.

[0803] "Rendering" refers to the process of converting generated computer graphics or a sequence of images into a final form.

[0804] "Database Means" means a system or method for storing generated computer graphics and video for reuse in future productions.

[0805] A "series of images" refers to a series of images created by combining multiple computer graphics.

[0806] "Effects" refers to special visual effects in a scene that are applied automatically based on user input.

[0807] "Sound Effects" means acoustic special effects for a scene that are applied automatically based on user input.

[0808] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) using a generative AI model based on that input, and then combines the generated CG to create a series of images. It also has the function of rendering the images and transmitting them to the user's terminal. If the user provides new input while watching a video, the CG can be regenerated and instantly reflected in the video.

[0809] Program processing overview

[0810] Collecting User Input

[0811] Users input their preferences and requests via their device. This input data includes the atmosphere of the scene, the characteristics of the characters, the type of action, etc. The device collects this information, converts it into an appropriate data format, and sends it to the server.

[0812] CG generation using generative AI models

[0813] The server receives user input data sent from the device and inputs it as a prompt sentence into the generative AI model. The generative AI model automatically generates a CG scene based on the prompt sentence. The generated CG is temporarily stored on the server and used for the next generation process.

[0814] CG scene arrangement and assembly

[0815] The server combines a large number of CG scenes into a series of images. If necessary, it automatically generates and adds effects and sound effects to each scene. It also adjusts the scenes to maintain a natural continuity.

[0816] Video rendering and output

[0817] The server renders the finished video and generates a final data file that is sent to the device and made available for viewing by the user.

[0818] Interactive operation while watching

[0819] While watching the video on their device, users can request scene changes or additions in real time. The device then sends this request to the server, which then inputs the new prompt into the generative AI model to generate a new CG scene. The new scene is then immediately reflected in the video.

[0820] Specific examples

[0821] If a user is creating a horror movie, he / she will select a "dark and scary forest scene" and specify "zombies wearing old clothes." In this case, the system will operate as follows:

[0822] User Input

[0823] The user inputs "horror," "dark forest," "zombie character," and "scary action" into the device, which then sends this information to the server.

[0824] Scene Generation

[0825] The server inputs prompt text into the generative AI model, generating CG scenes including a "dark and scary forest" and "zombie characters."

[0826] Scene arrangement and assembly

[0827] The server then combines the generated CG scenes to construct a scene featuring zombies in a terrifying forest, adding appropriate background sounds and effects.

[0828] Video Output

[0829] The server renders the entire video and sends it as a data file to the terminal, where the user can view the video.

[0830] Interactive Operation

[0831] While watching the video, the user makes a request to change the scene to one in which zombies suddenly appear. The device sends this request to the server, which then generates a new scene and immediately applies it to the video.

[0832] Prompt Sentence Examples

[0833] As a concrete example, the following prompt sentence is input to the generative AI model:

[0834] Scene Description: A dark and scary forest

[0835] Character: Zombie in old clothes

[0836] Action: Zombies suddenly appear

[0837] Special Effects: Music and fog effects to add to the horror in the background

[0838] In this way, the user can interactively enjoy video based on desired scenes and actions.

[0839] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0840] Step 1:

[0841] Collecting User Input

[0842] Users input their preferred scene and character settings via their device. Specifically, they enter the "scene atmosphere," "character characteristics," and "type of action" in the input boxes.

[0843] Input: Information based on user preferences (e.g. "horror", "dark forest", "zombie characters").

[0844] Output: A data file containing the user preferences.

[0845] Specific operation: The user enters information into each field on the terminal interface and clicks a button to send it to the server.

[0846] Step 2:

[0847] Sending user-entered data

[0848] The terminal sends the information entered by the user to the server, which converts the information into an appropriate data format (e.g., a prompt).

[0849] Input: A data file containing user-entered data.

[0850] Output: The prompt text sent to the server.

[0851] Specific operation: The terminal organizes the data, converts it into a prompt sentence format, and sends it to the server via the network.

[0852] Step 3:

[0853] AI-based CG generation

[0854] The server inputs the prompt text sent from the device into a generative AI model and generates CG based on the specified scene and characters.

[0855] Input: Prompt sentence (e.g. "Dark and scary forest" or "Zombie character").

[0856] Output: The generated CG scene.

[0857] Specific operation: The server receives the prompt sentence, inputs it into the algorithm associated with the generative AI model, and obtains a CG scene as the output.

[0858] Step 4:

[0859] CG scene arrangement and assembly

[0860] The server then combines the generated CG scenes to create a series of images, with effects and sound effects applied simultaneously.

[0861] Input: Multiple generated CG scenes.

[0862] Output: A stream of video data.

[0863] Specific operation: The server arranges CG scenes in time and automatically adds effects and music to generate continuous video.

[0864] Step 5:

[0865] Video rendering and output

[0866] The server renders the completed video data and generates the final data file, which is then sent to the user's device.

[0867] Input: A sequence of video data.

[0868] Output: The final rendered video file.

[0869] Specific operation: The server encodes the video data and sends it to the terminal in streaming format.

[0870] Step 6:

[0871] Interactive operation while watching

[0872] While watching the video on the device, the user can make additional scene changes or requests, which will change the scene in real time.

[0873] Input: A new request from the user (e.g., "Add a scene where zombies suddenly appear").

[0874] Output: Updated footage.

[0875] Specific operation: The user inputs a change request into the device, which then sends it to the server, which then generates a new scene and immediately reflects it in the video.

[0876] (Application example 1)

[0877] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0878] In recent years, the retail industry has required advanced interactive technologies to respond quickly and flexibly to diverse customer needs. However, customizing in-store displays and product introductions in real time remains a difficult challenge. For this reason, there is a demand for systems that can instantly change in-store displays and product introductions based on customer preferences. Furthermore, conventional systems have limited interactive operation, making it difficult to increase user satisfaction.

[0879] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0880] In this invention, the server includes means for collecting interactive input from users, means for automatically generating computer graphics based on the input using a computer generation technique, and means for combining the generated computer graphics to create a series of images, thereby enabling in-store displays and product introductions to be customized in real time based on user preferences, thereby improving customer satisfaction.

[0881] "Interactive input" refers to operations and instructions that a user gives to a system in real time.

[0882] "Computer-generated techniques" refers to methods of automatically generating computer graphics using algorithms and artificial intelligence techniques.

[0883] "Computer graphics" refers to visual images and video generated using a computer.

[0884] "Rendering" refers to the process of converting generated computer graphics into a form that can be displayed on a screen.

[0885] "User terminal" refers to a device that is directly operated by a user, such as a smartphone, smart glasses, or computer.

[0886] "Product display" refers to the visual arrangement and presentation of products in a store to highlight their appeal and appeal to customers.

[0887] "In-store decoration" refers to the decorations and layout used to create the atmosphere and theme of the store.

[0888] The system according to the present invention collects interactive input from users and automatically generates computer graphics (CG) based on the input using computer generation techniques, thereby customizing product displays and in-store decorations in real time. Specific embodiments for carrying out the present invention are described below.

[0889] System Overview

[0890] 1. User Input

[0891] Users input their product preferences and requirements to the system in real time using user devices such as smartphones or smart glasses. These inputs are collected based on the users' tastes and purchasing intentions.

[0892] 2. AI-based product scene generation

[0893] The server generates computer graphics (CG) of the product using a generative AI model based on the received user input, such as the OpenAI GPT model or Unity's ML-agents.

[0894] 3. Scene construction and composition

[0895] The server combines the generated product CG to create visually appealing product displays and in-store decorations for users, using rendering engine services such as Unity and Unreal Engine.

[0896] 4. Rendering and outputting the video

[0897] The server renders the generated CG images of product displays and in-store decorations and sends them to the user's device. The rendering process uses GPU acceleration to ensure real-time performance.

[0898] 5. Interactive Operation

[0899] When a user inputs a new request while browsing products in a store, the server immediately receives the input, regenerates the CG, and updates the display in real time.

[0900] Processing Description

[0901] In this system, the user device uses a smartphone or smart glasses and sends user input to the server. The server processes the received information and generates CG using a generative AI model. This generated CG is then rendered using a rendering engine service such as Unity or Unreal Engine and sent to the user device. When the user performs interactive operations, the server regenerates the CG and immediately responds to the user's new input.

[0902] Specific examples

[0903] For example, if a user wears smart glasses and selects "red casual clothes," the system will generate a CG image of "red clothes" and display it in real time on the glasses' HUD. A possible prompt might look something like this:

[0904] text

[0905] I'm looking for casual clothes in red.

[0906] In this way, the system of the present invention can customize product displays and store décor in real time based on user preferences, providing a more personalized shopping experience.

[0907] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0908] Step 1:

[0909] A user connects to the system using a smartphone or smart glasses and inputs a product request. For example, they input the text "I'm looking for red casual clothes." The input information is processed on the user's terminal and sent to the server. The input of this step is the user's request, and the output is the user request data sent to the server.

[0910] Step 2:

[0911] The server analyzes the received user request data and identifies the user's preferences. This analysis uses natural language processing (NLP) technology. The requested color "red" and category "casual" are extracted. The input is the user request data, and the output is the analyzed user preference data.

[0912] Step 3:

[0913] The server generates computer graphics (CG) using a generative AI model based on the analyzed user preference data. This AI model (for example, an improved version of the OpenAI GPT model) generates product images according to specific requests. The input is the user preference data, and the output is the generated product CG.

[0914] Step 4:

[0915] The server combines the generated product CG to create in-store decorations and product displays. This process uses rendering engine services such as Unity and Unreal Engine. The input is the generated product CG, and the output is the configured in-store decorations and product display CG.

[0916] Step 5:

[0917] The server renders the configured in-store decoration and product display CG and sends it to the user's device. The rendering process uses GPU acceleration to ensure real-time performance. The input is the configured in-store decoration and product display CG, and the output is the rendered product display image.

[0918] Step 6:

[0919] Users browse the store using their smartphones or smart glasses and experience a customized product display in real time. When the user inputs a new request, the server generates a new CG image and immediately sends the updated display information to the user's device. The input for this step is the user's new request, and the output is the updated product display image.

[0920] Step 7:

[0921] The system allows users to view the displayed products in real time and select their next action, such as purchasing the product or browsing other products. This provides an improved interactive shopping experience. The input is the rendered product display image, and the output is the user's next action.

[0922] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0923] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) based on that input using computer generation techniques, and then combines the generated CG to create a series of videos. It also has the ability to render the video and transmit it to the user's device, allowing the user to provide new input while watching, which in turn generates new CG and reflects it in the video. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system provides a more advanced interactive viewing experience.

[0924] System Overview

[0925] 1. User Input

[0926] The device receives interactive settings and requests from the user. Information collected includes the atmosphere of the scene, the characteristics of the characters, and the type of action.

[0927] Additionally, an emotion engine recognizes the user's emotions and provides feedback to the system in real time.

[0928] 2. AI Scene Generation

[0929] The AI ​​generates CG scenes based on the user's requests received by the server, taking into account the results of the emotion engine and generating scenes and effects that match the user's emotions.

[0930] 3. Assembling the Scene

[0931] The server combines multiple generated CG scenes into a series of videos, and scene transitions, effects, and sound effects are also adjusted based on feedback from the emotion engine.

[0932] 4. Video output

[0933] The completed video is rendered on the server and generated as a final data file, which is then sent to the device.

[0934] 5. Interactive Operation

[0935] If a user has a new desire while watching a video on a device, they can interactively input a change request. The emotion engine continuously monitors the user's emotions while watching and dynamically adjusts scenes and effects as needed.

[0936] The server receives the new request, regenerates the necessary CG, and immediately reflects it in the video.

[0937] Specific Examples

[0938] Specific examples of scene generation and emotion recognition

[0939] Suppose a user is creating a horror movie and has selected a "dark and scary forest scene" and has specified that the characters should be "zombies wearing old clothes." In this case, the system operates as follows:

[0940] 1. User Input

[0941] The device collects the user's preferences and sends information such as "horror," "dark forest," "zombie characters," and "scary action" to the server. The emotion engine simultaneously recognizes the user's emotions and provides feedback to the server about their state of mind, such as "tension" or "excitement."

[0942] 2. Scene Generation

[0943] The server uses generative AI to create a dark and frightening CG forest scene, including user-specified zombie characters, with additional effects and sound effects based on the user's emotions.

[0944] 3. Assembling the Scene

[0945] The server assembles a scene featuring zombies in a terrifying forest and adds the necessary background sounds and effects. The darkness and type of sound in the scene are adjusted based on the results of the emotion engine.

[0946] 4. Video output

[0947] The server renders the entire video and generates a file, which is then sent to the device for viewing by the user.

[0948] 5. Interactive Operation

[0949] If a user watches a video on their device and requests a scene where zombies suddenly appear, a new request is sent to the server. The emotion engine continues to monitor the user's emotions, and a new scene is instantly generated and reflected in the video.

[0950] This allows users to create, modify, and view videos that suit their preferences and emotions in real time. Furthermore, the created videos and CG are stored in a database and can be reused for future productions, significantly improving the efficiency of video production and providing a new entertainment experience.

[0951] The processing flow will be explained below.

[0952] Step 1:

[0953] The user inputs the settings and requests for video production through the device interface, for example, selecting parameters such as "horror movie," "dark forest," or "zombie characters." At the same time, the device's built-in emotion recognition function monitors the user's emotional state and collects that data.

[0954] Step 2:

[0955] The terminal collects the user's input and emotional state data, converts it into a data format, and transmits it to the server. The transmitted data includes the user-selected parameters and emotional data.

[0956] Step 3:

[0957] The server analyzes the received data and issues instructions to the AI ​​generator based on each parameter. The instructions also include details of effects and sound effects that take emotional data into account.

[0958] Step 4:

[0959] The server uses AI to generate CG scenes based on the user's settings and emotional state, and effects and sounds corresponding to the user's emotions are automatically applied to the generated CG scenes.

[0960] Step 5:

[0961] The server temporarily stores the generated CG scenes in a database and assigns them a reference ID, allowing each piece of CG data to be reused in later processes.

[0962] Step 6:

[0963] The server retrieves multiple CG data sets stored in a database and combines them to create a series of images. Scene transitions, additional effects, and sound effects are also adjusted in real time based on the emotional data.

[0964] Step 7:

[0965] The server renders the edited footage and stores it as a final digital file, which is then converted into a viewable format.

[0966] Step 8:

[0967] The server sends the rendered video file to the device, and a viewing URL or streaming service link is provided so the user can watch the video.

[0968] Step 9:

[0969] The user watches the video using the device. If a new request arises while watching, for example, "I want to change to a scene where zombies suddenly appear," the user inputs the request into the device.

[0970] Step 10:

[0971] The device sends a new user request to the server, which includes details of which parts of the scene to change and how. The emotion engine continues to monitor the user's emotions and sends the data to the server.

[0972] Step 11:

[0973] The server uses the AI ​​to generate additional CG based on the new request and emotion data. Effects and sound effects based on the emotion data are also applied at the same time.

[0974] Step 12:

[0975] The server then integrates the newly generated scene into the existing footage and re-renders the entire footage.

[0976] Step 13:

[0977] The server transmits the updated video to the terminal again, making the video that reflects the user's new requests and emotions viewable.

[0978] This series of processes allows users to generate, view, and change images in real time according to their preferences and emotions.

[0979] Example 2

[0980] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0981] Conventional video production systems have had difficulty dynamically changing and generating video content based on real-time user feedback and emotions. Furthermore, the technology to recognize users' emotions while they are watching and instantly reflect them in the content of the video has not yet been fully developed. As a result, it has been impossible to provide the interactivity and personalized entertainment experience that users desire.

[0982] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[0983] In this invention, the server includes means for collecting interactive input from a user, means for automatically generating computer graphics using a generative AI model, means for combining the generated computer graphics to create a series of videos, means for rendering the videos and transmitting them to a user terminal, means for allowing the user to interactively enter new input while viewing the videos, means for regenerating computer graphics based on the new input and immediately reflecting the input, and means for recognizing the user's emotions in real time and adjusting scenes and effects based on the emotions. This enables users to interactively generate and view videos based on feedback and emotions in real time, providing a personalized entertainment experience.

[0984] A "means for collecting interactive input from a user" is a device or software for receiving user settings, requests, or instructions.

[0985] "Means for automatically generating computer graphics using generative AI models" refers to devices or software that use artificial intelligence technology to automatically generate CG based on user input.

[0986] "Means for combining generated computer graphics to create a series of images" refers to a device or software that connects the individual generated CG images to create a visually continuous image.

[0987] The "means for rendering the video and transmitting it to the user terminal" refers to a device or software for rendering a series of videos in high quality and delivering the data to the terminal used by the user.

[0988] "Means for enabling the user to interactively input new information while viewing the video" refers to a device or software that provides an interface that allows the user to input changes or additional instructions while the video is being played.

[0989] "Means for regenerating computer graphics based on the new input and immediately reflecting it" refers to a device or software that generates additional or modified CG based on the user's new input and immediately incorporates it into the video.

[0990] "Means for recognizing a user's emotions in real time and adjusting scenes and effects based on said emotions" refers to a device or software that determines emotions from the user's facial expressions, voice, etc., and dynamically changes the content of the video and effects according to those emotions.

[0991] The system of this invention collects interactive input from users, automatically generates computer graphics (CG) based on that input, and uses that CG to create a series of images. It also has the function of rendering the generated images and transmitting them to the user's terminal. By providing new input while watching, the user can regenerate CG and instantly reflect it in the images. It can also recognize the user's emotions in real time and adjust scenes and effects based on those emotions.

[0992] Interactive input from the user

[0993] The device interactively receives settings and requests from the user, including the atmosphere of the scene, the characteristics of the characters, and the type of action. For example, if a user selects a "dark and scary forest scene" and specifies "zombies wearing old clothes" as the characters to create a horror movie, this information is collected by the device. The device then uses an emotion engine to analyze the user's emotions in real time and identify emotional states such as "tension" and "excitement."

[0994] Automatic generation of computer graphics

[0995] The server generates CG using a generative AI model based on the data received from the device and the results of the emotion engine. For example, if a user requests a "dark and scary forest scene," the server inputs "a dark and scary forest scene" as a prompt into the generative AI model and generates appropriate CG. This generative AI model uses Stable Diffusion and DALL-E 2, among others.

[0996] CG assembly and sequencing

[0997] The generated CG scenes are then combined by the server to create a series of images. The server then adjusts these, including scene transitions, effects, and sound effects, based on feedback from the emotion engine. For example, the darkness and type of sound in a scene are adjusted appropriately based on the results of the emotion engine.

[0998] Video rendering and output

[0999] The server renders the finished video and generates the final data file using rendering engines such as Blender and Unreal Engine. The rendered video is then sent to the device where it can be viewed by the user.

[1000] Interactive operation and immediate feedback

[1001] If a user has new requests while watching a video on their device, they can interactively input change requests. The emotion engine continuously monitors the user's emotions and dynamically adjusts the scenes and effects as needed. For example, if a user requests, "I want to change the scene to one where zombies suddenly appear," that request is sent to the server. The server then issues a new prompt to the generative AI model, instantly generating new CG and reflecting it in the video. In this way, users can interactively generate and change videos in real time.

[1002] Specific examples

[1003] Imagine a user creating a horror movie. The user inputs the following information through an interface:

[1004] "I want to make a horror movie. I'll choose a dark, scary forest scene and designate zombies in old-fashioned clothing as characters. I want to change the scene so that the zombies suddenly appear while I'm watching."

[1005] Based on this prompt, the system can generate, modify, and view videos in real time that match the user's preferences and emotions. The generated videos and CG are stored in a database and can be reused for future productions, significantly improving the efficiency of video production and providing a new entertainment experience.

[1006] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1007] Step 1: Collecting User Input

[1008] The device receives interactive input from the user, including the atmosphere of a particular scene, the characteristics of characters, the type of action, etc. The emotion engine then operates to obtain the user's emotion data.

[1009] Input: User-defined scene and character requests, emotional data (e.g., "dark and scary forest scene," "zombie characters," "scary action").

[1010] Output: Collected user requests and sentiment data.

[1011] Step 2: Send input data to the server

[1012] The terminal transmits the collected user input data and emotion data to the server.

[1013] Input: Collected user desires and sentiment data.

[1014] Output: User wishes and emotion data sent to the server.

[1015] Step 3: Generate the CG scene

[1016] The server generates CG using a generative AI model based on the received data, taking into account the results of the emotion engine and optimizing scenes and effects to match the user's emotions.

[1017] Input: User desires and emotion data.

[1018] Data processing: Generate CG using generative AI models (e.g., Stable Diffusion or DALL-E 2).

[1019] Output: CG data according to user's request.

[1020] Step 4: Combining and composing the CG scene

[1021] The server combines the generated CG scenes into a series of images, and adjusts scene transitions, effects, and sound effects based on information from the emotion engine.

[1022] Input: Generated CG data and emotion engine results.

[1023] Data processing: Combining CG scenes, scene transitions, adding effects and sound effects.

[1024] Output: A complete video sequence.

[1025] Step 5: Rendering the video and sending it to the device

[1026] The server renders the finished video and generates the final data file, which is then sent to the device for viewing by the user.

[1027] Input: A complete video sequence.

[1028] Data processing: Video rendering (using Blender, Unreal Engine, etc.).

[1029] Output: Rendered video file.

[1030] Step 6: Interact and Regenerate

[1031] If a user has a new request while watching a video on their device, they can input it. The emotion engine monitors the user's emotions and dynamically adjusts scenes and effects as needed. The server receives the new request, regenerates the CG, and immediately reflects it in the video.

[1032] Input: The user's new request and emotion data.

[1033] Data processing: Input new prompt text into the generative AI model and regenerate the necessary CG.

[1034] Output: Updated video data.

[1035] (Application example 2)

[1036] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1037] Current virtual store shopping experiences make it difficult for users to easily find products that suit their preferences or to try on items. Furthermore, there is a lack of systems that can recognize user emotions and dynamically generate content, making it impossible to provide a personalized shopping experience tailored to each individual user. This leads to a decline in user satisfaction and engagement.

[1038] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting interactive input from the user, means for automatically generating computer graphics using a computer generation method based on the input, and emotion recognition means for recognizing the user's emotions and dynamically adjusting images and effects. This enables a personalized shopping experience in a virtual store that is tailored to the user's preferences and emotions.

[1039] "Means for collecting interactive input from the user" refers to an interface through which the user inputs their desired settings and requests and the system receives them.

[1040] "Means for automatically generating computer graphics using computer generation methods" refers to the process of using AI technology to generate images or videos based on user input.

[1041] "A means of combining generated computer graphics to create a series of images" refers to the process of appropriately arranging individual CGs and assembling them into a single continuous video work.

[1042] "Means for rendering the video and transmitting it to the user terminal" refers to the process of converting the generated video into a format that can be displayed with high quality and delivering it to the user's device.

[1043] "Means that allow the user to interactively input new information while watching the video" refers to a function that allows the user to input new requests or changes to the system in real time while watching the video.

[1044] "Means of regenerating computer graphics based on new input and immediately reflecting it" refers to the process of receiving a new user request, generating new CG, and immediately reflecting that content in the video.

[1045] "Emotion recognition means that recognizes the user's emotions and dynamically adjusts images and effects" refers to technology that analyzes the user's emotional state in real time and dynamically changes images and effects accordingly.

[1046] "Avatar management means that enables users to set up virtual avatars and move freely within a virtual space" refers to a function that enables users to set up their own avatars and operate and move those avatars within a virtual space.

[1047] "Means for providing product information and try-on simulations in a virtual space" refers to a function that allows users to check out products in a virtual store environment and virtually try on outfits.

[1048] The system that realizes this invention automatically generates computer graphics based on interactive user input and combines them to create a series of images, providing a personalized shopping experience in a virtual store. The system program and its processing are described below.

[1049] The server collects user input and uses an AI model to generate CG based on it. In this system, users can customize their own avatars using their smartphones and move them freely in the virtual space. The main software and hardware used include smartphones, Emotion API, generation AI (e.g., DALL-E 2), and a rendering engine (e.g., Unity).

[1050] When a user enters their preferred settings through the smartphone interface, those settings are sent to the server. For example, if they select "casual summer clothing," the Emotion API analyzes the user's facial expressions and recognizes their emotions. The server then uses generative AI such as DALL-E 2 to automatically generate the necessary CG based on this emotional information and the user's settings.

[1051] The generated CG is assembled into a virtual store scene using a rendering engine such as Unity. The virtual store scene is equipped with a function that allows the user's avatar to move freely, and product information and a try-on simulation are also provided. The final rendered image is sent to a smartphone device, where the user can view and interact with it.

[1052] For example, when a user clicks on a specific product in a store, detailed information about that product and a try-on simulation are displayed. Furthermore, when the Emotion API recognizes the user's "interest" from their facial expression, they are dynamically guided to products and sections that match their interests.

[1053] Prompt Sentence Examples

[1054] Prompt for the generator: "Generate casual summer clothing for a virtual avatar."

[1055] Prompt for emotion recognition API: "Evaluate user interest from facial expressions and guide them to specific products."

[1056] As described above, this system dynamically generates and adjusts images and effects based on the user's emotions and inputs, providing a personalized virtual shopping experience tailored to each individual user.

[1057] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1058] Step 1:

[1059] A user launches the application on their smartphone and configures their virtual avatar. They upload a photo of their face and customize the avatar's appearance and clothing. This forms the avatar's basic information and the user's preferred settings as input data. The generative AI model is prompted with the following sentence: "Generate casual summer clothing for the virtual avatar."

[1060] Step 2:

[1061] The device sends the avatar data and user preferences to the server. The server receives this data and uses the Emotion API to analyze the user's facial expressions in real time to obtain emotional information. This emotional information is reflected in future CG generation and scene construction. The prompt to the emotion recognition API is: "Please evaluate the user's interest from their facial expressions and guide them to specific products."

[1062] Step 3:

[1063] The server uses a generation AI (e.g., DALL-E 2) to generate the necessary CG in the virtual space based on the user's preferred settings and emotional information. The generation process creates the specified costumes and objects required for the scene and generates them as 3D models. The input for this step is the user's settings and emotional information, and the output is the generated CG data.

[1064] Step 4:

[1065] The server constructs a virtual store scene based on the generated CG data. Using a rendering engine (e.g., Unity), the scene is constructed by combining CG. Within this scene, an area is placed where the user's avatar can move freely, allowing for product display and a try-on simulation. The input for this step is the generated CG data, and the output is rendered scene data.

[1066] Step 5:

[1067] The server sends the rendered scene data to the user's smartphone. The user can then freely walk around the virtual store on their smartphone, browsing product information and trying on items. Selecting a specific item displays detailed information about that item and a try-on simulation. The input for this step is the rendered scene data, and the output is an interactive display on the user's device.

[1068] Step 6:

[1069] Users can interactively request new products or scenes that interest them in the virtual store. The new request from the terminal is sent to the server, which regenerates the necessary CG based on the request and immediately reflects it in the scene. The input of this step is the user's new request data, and the output is the updated scene data.

[1070] This is the flow of the system program that realizes this application example. The specific operations and data processing / calculation performed at each step provide the user with a personalized shopping experience.

[1071] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1072] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1073] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1074] [Fourth embodiment]

[1075] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1076] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1077] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1078] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1079] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1080] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1081] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1082] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1083] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1084] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1085] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1086] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1087] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1088] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) based on that input using computer generation techniques, and then combines the generated CG to create a series of images. It also has the function of rendering the images and transmitting them to the user's terminal. If the user provides new input while watching a video, the system can regenerate the CG and instantly reflect the input.

[1089] System Overview

[1090] 1. User Input

[1091] The device interactively receives preferences and requests from the user.

[1092] The collected information includes, for example, the atmosphere of the scene, the characteristics of the characters, the type of action, and the like.

[1093] 2. AI Scene Generation

[1094] The generation AI generates the CG of the scene based on the user request received by the server.

[1095] The generated CG is temporarily saved and used for the next generation process.

[1096] 3. Scene Arrangement and Composition

[1097] The server combines multiple CG scenes to create a series of images.

[1098] Effects and sound effects are also automatically generated as needed.

[1099] 4. Video output

[1100] The completed video is rendered on the server and generated as a final data file.

[1101] The server sends the video to the terminal so that the user can view it.

[1102] 5. Interactive Operation

[1103] When a user watches a video on a terminal, they can change or select scenes in real time.

[1104] The server immediately receives the user's new operations and regenerates and reflects the CG scene.

[1105] Specific Examples

[1106] Example of scene generation

[1107] Suppose a user is creating a horror movie and has selected a "dark and scary forest scene" and has specified that the characters should be "zombies wearing old clothes." In this case, the system operates as follows:

[1108] 1. User Input

[1109] The device collects the user's preferences and sends information such as "horror," "dark forest," "zombie characters," and "scary action" to the server.

[1110] 2. Scene Generation

[1111] The server uses generative AI to create a dark and scary CG forest scene, which also includes a user-specified zombie character.

[1112] 3. Scene construction

[1113] The server assembles scenes featuring zombies in a terrifying forest and adds appropriate background sounds and effects.

[1114] 4. Video output

[1115] The server renders the entire video and generates it as a file.

[1116] The server transmits the generated video to the terminal so that the user can view it.

[1117] 5. Interactive Operation

[1118] While watching the video on the device, the user makes a request to change the scene to one in which zombies suddenly appear.

[1119] This request is sent from the device to the server, which then generates a new scene and immediately reflects it.

[1120] In this way, users can interactively enjoy videos that suit their preferences and requests. In addition, the generated videos and CG are stored in a database and can be reused for future productions. This will significantly improve the efficiency of video production and enable the provision of new entertainment experiences.

[1121] The processing flow will be explained below.

[1122] Step 1:

[1123] The user inputs the settings and requests required for video production through the device interface, selecting parameters such as "horror movie," "dark forest," and "zombie characters."

[1124] Step 2:

[1125] The terminal collects the user's input, converts it into a data format, and sends it to the server, including all the parameters selected by the user.

[1126] Step 3:

[1127] The server analyzes the received data and issues instructions based on each parameter to the generation AI, which then generates computer graphics (CG) containing the scenes and characters desired by the user.

[1128] Step 4:

[1129] The server temporarily stores the generated CG in a database and assigns it a reference ID. Each saved CG data is used in the next process.

[1130] Step 5:

[1131] The server retrieves multiple CG data sets stored in a database, combines them, and edits them into a series of images. Scene transitions, effects, and sound effects are also added automatically at this stage.

[1132] Step 6:

[1133] The server renders the edited footage and stores it as a final digital file, which is then converted into a viewable format.

[1134] Step 7:

[1135] The server then sends the rendered video file to the device where the user can view it. A viewing URL or a link to a streaming service may be provided.

[1136] Step 8:

[1137] The user watches the video using the terminal. If a new request arises while watching, the user can input a request such as "I want to change to a scene where zombies suddenly appear."

[1138] Step 9:

[1139] The device sends a new user request to the server, which includes details of what part of the scene to change and how.

[1140] Step 10:

[1141] The server receives the new request, again using the generative AI to generate any additional CG needed, then adds the new scene in the appropriate location and re-renders the entire video.

[1142] Step 11:

[1143] The server transmits the updated video to the terminal again, making the video that reflects the user's new request available for viewing.

[1144] This series of processes allows users to generate, view, and change videos that suit their preferences in real time.

[1145] Example 1

[1146] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1147] Current video production systems have difficulty in instantly reflecting user interactive input and have limitations in automatically applying scene and sound effects. They also lack appropriate means for reusing generated computer graphics and video in future productions. This reduces the efficiency of video production and makes it difficult to quickly respond to user requests.

[1148] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1149] In this invention, the server includes: means for collecting interactive input from users; means for automatically generating computer graphics using a generative AI model; means for combining the generated computer graphics to create a series of videos; means for rendering the videos and transmitting them to a user terminal; means for allowing the user to interactively enter new input while viewing the videos; means for regenerating computer graphics based on the new input and immediately reflecting the new input; and database means for saving the generated computer graphics and videos so that they can be reused in subsequent productions. This allows the user's interactive input to be reflected in the videos in real time, and scene effects and sound effects can also be automatically applied. Furthermore, saving the generated computer graphics and videos and reusing them in subsequent productions can improve the efficiency of video production.

[1150] "User input" means information that a user interactively sets or requests to the system.

[1151] A "generative AI model" refers to an algorithm or system for automatically generating computer graphics based on user input data.

[1152] "Computer graphics" refers to images or pictures generated using a computer, generally including scenes, characters, etc.

[1153] "Interactive input" refers to new operations or requests made by the user in real time while watching a video.

[1154] "Rendering" refers to the process of converting generated computer graphics or a sequence of images into a final form.

[1155] "Database Means" means a system or method for storing generated computer graphics and video for reuse in future productions.

[1156] A "series of images" refers to a series of images created by combining multiple computer graphics.

[1157] "Effects" refers to special visual effects in a scene that are applied automatically based on user input.

[1158] "Sound Effects" means acoustic special effects for a scene that are applied automatically based on user input.

[1159] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) using a generative AI model based on that input, and then combines the generated CG to create a series of images. It also has the function of rendering the images and transmitting them to the user's terminal. If the user provides new input while watching a video, the CG can be regenerated and instantly reflected in the video.

[1160] Program processing overview

[1161] Collecting User Input

[1162] Users input their preferences and requests via their device. This input data includes the atmosphere of the scene, the characteristics of the characters, the type of action, etc. The device collects this information, converts it into an appropriate data format, and sends it to the server.

[1163] CG generation using generative AI models

[1164] The server receives user input data sent from the device and inputs it as a prompt sentence into the generative AI model. The generative AI model automatically generates a CG scene based on the prompt sentence. The generated CG is temporarily stored on the server and used for the next generation process.

[1165] CG scene arrangement and assembly

[1166] The server combines a large number of CG scenes into a series of images. If necessary, it automatically generates and adds effects and sound effects to each scene. It also adjusts the scenes to maintain a natural continuity.

[1167] Video rendering and output

[1168] The server renders the finished video and generates a final data file that is sent to the device and made available for viewing by the user.

[1169] Interactive operation while watching

[1170] While watching the video on their device, users can request scene changes or additions in real time. The device then sends this request to the server, which then inputs the new prompt into the generative AI model to generate a new CG scene. The new scene is then immediately reflected in the video.

[1171] Specific examples

[1172] If a user is creating a horror movie, he / she will select a "dark and scary forest scene" and specify "zombies wearing old clothes." In this case, the system will operate as follows:

[1173] User Input

[1174] The user inputs "horror," "dark forest," "zombie character," and "scary action" into the device, which then sends this information to the server.

[1175] Scene Generation

[1176] The server inputs prompt text into the generative AI model, generating CG scenes including a "dark and scary forest" and "zombie characters."

[1177] Scene arrangement and assembly

[1178] The server then combines the generated CG scenes to construct a scene featuring zombies in a terrifying forest, adding appropriate background sounds and effects.

[1179] Video Output

[1180] The server renders the entire video and sends it as a data file to the terminal, where the user can view the video.

[1181] Interactive Operation

[1182] While watching the video, the user makes a request to change the scene to one in which zombies suddenly appear. The device sends this request to the server, which then generates a new scene and immediately applies it to the video.

[1183] Prompt Sentence Examples

[1184] As a concrete example, the following prompt sentence is input to the generative AI model:

[1185] Scene Description: A dark and scary forest

[1186] Character: Zombie in old clothes

[1187] Action: Zombies suddenly appear

[1188] Special Effects: Music and fog effects to add to the horror in the background

[1189] In this way, the user can interactively enjoy video based on desired scenes and actions.

[1190] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1191] Step 1:

[1192] Collecting User Input

[1193] Users input their preferred scene and character settings via their device. Specifically, they enter the "scene atmosphere," "character characteristics," and "type of action" in the input boxes.

[1194] Input: Information based on user preferences (e.g. "horror", "dark forest", "zombie characters").

[1195] Output: A data file containing the user preferences.

[1196] Specific operation: The user enters information into each field on the terminal interface and clicks a button to send it to the server.

[1197] Step 2:

[1198] Sending user-entered data

[1199] The terminal sends the information entered by the user to the server, which converts the information into an appropriate data format (e.g., a prompt).

[1200] Input: A data file containing user-entered data.

[1201] Output: The prompt text sent to the server.

[1202] Specific operation: The terminal organizes the data, converts it into a prompt sentence format, and sends it to the server via the network.

[1203] Step 3:

[1204] AI-based CG generation

[1205] The server inputs the prompt text sent from the device into a generative AI model and generates CG based on the specified scene and characters.

[1206] Input: Prompt sentence (e.g. "Dark and scary forest" or "Zombie character").

[1207] Output: The generated CG scene.

[1208] Specific operation: The server receives the prompt sentence, inputs it into the algorithm associated with the generative AI model, and obtains a CG scene as the output.

[1209] Step 4:

[1210] CG scene arrangement and assembly

[1211] The server then combines the generated CG scenes to create a series of images, with effects and sound effects applied simultaneously.

[1212] Input: Multiple generated CG scenes.

[1213] Output: A stream of video data.

[1214] Specific operation: The server arranges CG scenes in time and automatically adds effects and music to generate continuous video.

[1215] Step 5:

[1216] Video rendering and output

[1217] The server renders the completed video data and generates the final data file, which is then sent to the user's device.

[1218] Input: A sequence of video data.

[1219] Output: The final rendered video file.

[1220] Specific operation: The server encodes the video data and sends it to the terminal in streaming format.

[1221] Step 6:

[1222] Interactive operation while watching

[1223] While watching the video on the device, the user can make additional scene changes or requests, which will change the scene in real time.

[1224] Input: A new request from the user (e.g., "Add a scene where zombies suddenly appear").

[1225] Output: Updated footage.

[1226] Specific operation: The user inputs a change request into the device, which then sends it to the server, which then generates a new scene and immediately reflects it in the video.

[1227] (Application example 1)

[1228] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1229] In recent years, the retail industry has required advanced interactive technologies to respond quickly and flexibly to diverse customer needs. However, customizing in-store displays and product introductions in real time remains a difficult challenge. For this reason, there is a demand for systems that can instantly change in-store displays and product introductions based on customer preferences. Furthermore, conventional systems have limited interactive operation, making it difficult to increase user satisfaction.

[1230] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1231] In this invention, the server includes means for collecting interactive input from users, means for automatically generating computer graphics based on the input using a computer generation technique, and means for combining the generated computer graphics to create a series of images, thereby enabling in-store displays and product introductions to be customized in real time based on user preferences, thereby improving customer satisfaction.

[1232] "Interactive input" refers to operations and instructions that a user gives to a system in real time.

[1233] "Computer-generated techniques" refers to methods of automatically generating computer graphics using algorithms and artificial intelligence techniques.

[1234] "Computer graphics" refers to visual images and video generated using a computer.

[1235] "Rendering" refers to the process of converting generated computer graphics into a form that can be displayed on a screen.

[1236] "User terminal" refers to a device that is directly operated by a user, such as a smartphone, smart glasses, or computer.

[1237] "Product display" refers to the visual arrangement and presentation of products in a store to highlight their appeal and appeal to customers.

[1238] "In-store decoration" refers to the decorations and layout used to create the atmosphere and theme of the store.

[1239] The system according to the present invention collects interactive input from users and automatically generates computer graphics (CG) based on the input using computer generation techniques, thereby customizing product displays and in-store decorations in real time. Specific embodiments for carrying out the present invention are described below.

[1240] System Overview

[1241] 1. User Input

[1242] Users input their product preferences and requirements to the system in real time using user devices such as smartphones or smart glasses. These inputs are collected based on the users' tastes and purchasing intentions.

[1243] 2. AI-based product scene generation

[1244] The server generates computer graphics (CG) of the product using a generative AI model based on the received user input, such as the OpenAI GPT model or Unity's ML-agents.

[1245] 3. Scene construction and composition

[1246] The server combines the generated product CG to create visually appealing product displays and in-store decorations for users, using rendering engine services such as Unity and Unreal Engine.

[1247] 4. Rendering and outputting the video

[1248] The server renders the generated CG images of product displays and in-store decorations and sends them to the user's device. The rendering process uses GPU acceleration to ensure real-time performance.

[1249] 5. Interactive Operation

[1250] When a user inputs a new request while browsing products in a store, the server immediately receives the input, regenerates the CG, and updates the display in real time.

[1251] Processing Description

[1252] In this system, the user device uses a smartphone or smart glasses and sends user input to the server. The server processes the received information and generates CG using a generative AI model. This generated CG is then rendered using a rendering engine service such as Unity or Unreal Engine and sent to the user device. When the user performs interactive operations, the server regenerates the CG and immediately responds to the user's new input.

[1253] Specific examples

[1254] For example, if a user wears smart glasses and selects "red casual clothes," the system will generate a CG image of "red clothes" and display it in real time on the glasses' HUD. A possible prompt might look something like this:

[1255] text

[1256] I'm looking for casual clothes in red.

[1257] In this way, the system of the present invention can customize product displays and store décor in real time based on user preferences, providing a more personalized shopping experience.

[1258] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1259] Step 1:

[1260] A user connects to the system using a smartphone or smart glasses and inputs a product request. For example, they input the text "I'm looking for red casual clothes." The input information is processed on the user's terminal and sent to the server. The input of this step is the user's request, and the output is the user request data sent to the server.

[1261] Step 2:

[1262] The server analyzes the received user request data and identifies the user's preferences. This analysis uses natural language processing (NLP) technology. The requested color "red" and category "casual" are extracted. The input is the user request data, and the output is the analyzed user preference data.

[1263] Step 3:

[1264] The server generates computer graphics (CG) using a generative AI model based on the analyzed user preference data. This AI model (for example, an improved version of the OpenAI GPT model) generates product images according to specific requests. The input is the user preference data, and the output is the generated product CG.

[1265] Step 4:

[1266] The server combines the generated product CG to create in-store decorations and product displays. This process uses rendering engine services such as Unity and Unreal Engine. The input is the generated product CG, and the output is the configured in-store decorations and product display CG.

[1267] Step 5:

[1268] The server renders the configured in-store decoration and product display CG and sends it to the user's device. The rendering process uses GPU acceleration to ensure real-time performance. The input is the configured in-store decoration and product display CG, and the output is the rendered product display image.

[1269] Step 6:

[1270] Users browse the store using their smartphones or smart glasses and experience a customized product display in real time. When the user inputs a new request, the server generates a new CG image and immediately sends the updated display information to the user's device. The input for this step is the user's new request, and the output is the updated product display image.

[1271] Step 7:

[1272] The system allows users to view the displayed products in real time and select their next action, such as purchasing the product or browsing other products. This provides an improved interactive shopping experience. The input is the rendered product display image, and the output is the user's next action.

[1273] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1274] The system of the present invention collects interactive input from users, automatically generates computer graphics (CG) based on that input using computer generation techniques, and then combines the generated CG to create a series of videos. It also has the ability to render the video and transmit it to the user's device, allowing the user to provide new input while watching, which in turn generates new CG and reflects it in the video. Furthermore, by incorporating an emotion engine that recognizes the user's emotions, the system provides a more advanced interactive viewing experience.

[1275] System Overview

[1276] 1. User Input

[1277] The device receives interactive settings and requests from the user. Information collected includes the atmosphere of the scene, the characteristics of the characters, and the type of action.

[1278] Additionally, an emotion engine recognizes the user's emotions and provides feedback to the system in real time.

[1279] 2. AI Scene Generation

[1280] The AI ​​generates CG scenes based on the user's requests received by the server, taking into account the results of the emotion engine and generating scenes and effects that match the user's emotions.

[1281] 3. Assembling the Scene

[1282] The server combines multiple generated CG scenes into a series of videos, and scene transitions, effects, and sound effects are also adjusted based on feedback from the emotion engine.

[1283] 4. Video output

[1284] The completed video is rendered on the server and generated as a final data file, which is then sent to the device.

[1285] 5. Interactive Operation

[1286] If a user has a new desire while watching a video on a device, they can interactively input a change request. The emotion engine continuously monitors the user's emotions while watching and dynamically adjusts scenes and effects as needed.

[1287] The server receives the new request, regenerates the necessary CG, and immediately reflects it in the video.

[1288] Specific Examples

[1289] Specific examples of scene generation and emotion recognition

[1290] Suppose a user is creating a horror movie and has selected a "dark and scary forest scene" and has specified that the characters should be "zombies wearing old clothes." In this case, the system operates as follows:

[1291] 1. User Input

[1292] The device collects the user's preferences and sends information such as "horror," "dark forest," "zombie characters," and "scary action" to the server. The emotion engine simultaneously recognizes the user's emotions and provides feedback to the server about their state of mind, such as "tension" or "excitement."

[1293] 2. Scene Generation

[1294] The server uses generative AI to create a dark and frightening CG forest scene, including user-specified zombie characters, with additional effects and sound effects based on the user's emotions.

[1295] 3. Assembling the Scene

[1296] The server assembles a scene featuring zombies in a terrifying forest and adds the necessary background sounds and effects. The darkness and type of sound in the scene are adjusted based on the results of the emotion engine.

[1297] 4. Video output

[1298] The server renders the entire video and generates a file, which is then sent to the device for viewing by the user.

[1299] 5. Interactive Operation

[1300] If a user watches a video on their device and requests a scene where zombies suddenly appear, a new request is sent to the server. The emotion engine continues to monitor the user's emotions, and a new scene is instantly generated and reflected in the video.

[1301] This allows users to create, modify, and view videos that suit their preferences and emotions in real time. Furthermore, the created videos and CG are stored in a database and can be reused for future productions, significantly improving the efficiency of video production and providing a new entertainment experience.

[1302] The processing flow will be explained below.

[1303] Step 1:

[1304] The user inputs the settings and requests for video production through the device interface, for example, selecting parameters such as "horror movie," "dark forest," or "zombie characters." At the same time, the device's built-in emotion recognition function monitors the user's emotional state and collects that data.

[1305] Step 2:

[1306] The terminal collects the user's input and emotional state data, converts it into a data format, and transmits it to the server. The transmitted data includes the user-selected parameters and emotional data.

[1307] Step 3:

[1308] The server analyzes the received data and issues instructions to the AI ​​generator based on each parameter. The instructions also include details of effects and sound effects that take emotional data into account.

[1309] Step 4:

[1310] The server uses AI to generate CG scenes based on the user's settings and emotional state, and effects and sounds corresponding to the user's emotions are automatically applied to the generated CG scenes.

[1311] Step 5:

[1312] The server temporarily stores the generated CG scenes in a database and assigns them a reference ID, allowing each piece of CG data to be reused in later processes.

[1313] Step 6:

[1314] The server retrieves multiple CG data sets stored in a database and combines them to create a series of images. Scene transitions, additional effects, and sound effects are also adjusted in real time based on the emotional data.

[1315] Step 7:

[1316] The server renders the edited footage and stores it as a final digital file, which is then converted into a viewable format.

[1317] Step 8:

[1318] The server sends the rendered video file to the device, and a viewing URL or streaming service link is provided so the user can watch the video.

[1319] Step 9:

[1320] The user watches the video using the device. If a new request arises while watching, for example, "I want to change to a scene where zombies suddenly appear," the user inputs the request into the device.

[1321] Step 10:

[1322] The device sends a new user request to the server, which includes details of which parts of the scene to change and how. The emotion engine continues to monitor the user's emotions and sends the data to the server.

[1323] Step 11:

[1324] The server uses the AI ​​to generate additional CG based on the new request and emotion data. Effects and sound effects based on the emotion data are also applied at the same time.

[1325] Step 12:

[1326] The server then integrates the newly generated scene into the existing footage and re-renders the entire footage.

[1327] Step 13:

[1328] The server transmits the updated video to the terminal again, making the video that reflects the user's new requests and emotions viewable.

[1329] This series of processes allows users to generate, view, and change images in real time according to their preferences and emotions.

[1330] Example 2

[1331] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1332] Conventional video production systems have had difficulty dynamically changing and generating video content based on real-time user feedback and emotions. Furthermore, the technology to recognize users' emotions while they are watching and instantly reflect them in the content of the video has not yet been fully developed. As a result, it has been impossible to provide the interactivity and personalized entertainment experience that users desire.

[1333] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means.

[1334] In this invention, the server includes means for collecting interactive input from a user, means for automatically generating computer graphics using a generative AI model, means for combining the generated computer graphics to create a series of videos, means for rendering the videos and transmitting them to a user terminal, means for allowing the user to interactively enter new input while viewing the videos, means for regenerating computer graphics based on the new input and immediately reflecting the input, and means for recognizing the user's emotions in real time and adjusting scenes and effects based on the emotions. This enables users to interactively generate and view videos based on feedback and emotions in real time, providing a personalized entertainment experience.

[1335] A "means for collecting interactive input from a user" is a device or software for receiving user settings, requests, or instructions.

[1336] "Means for automatically generating computer graphics using generative AI models" refers to devices or software that use artificial intelligence technology to automatically generate CG based on user input.

[1337] "Means for combining generated computer graphics to create a series of images" refers to a device or software that connects the individual generated CG images to create a visually continuous image.

[1338] The "means for rendering the video and transmitting it to the user terminal" refers to a device or software for rendering a series of videos in high quality and delivering the data to the terminal used by the user.

[1339] "Means for enabling the user to interactively input new information while viewing the video" refers to a device or software that provides an interface that allows the user to input changes or additional instructions while the video is being played.

[1340] "Means for regenerating computer graphics based on the new input and immediately reflecting it" refers to a device or software that generates additional or modified CG based on the user's new input and immediately incorporates it into the video.

[1341] "Means for recognizing a user's emotions in real time and adjusting scenes and effects based on said emotions" refers to a device or software that determines emotions from the user's facial expressions, voice, etc., and dynamically changes the content of the video and effects according to those emotions.

[1342] The system of this invention collects interactive input from users, automatically generates computer graphics (CG) based on that input, and uses that CG to create a series of images. It also has the function of rendering the generated images and transmitting them to the user's terminal. By providing new input while watching, the user can regenerate CG and instantly reflect it in the images. It can also recognize the user's emotions in real time and adjust scenes and effects based on those emotions.

[1343] Interactive input from the user

[1344] The device interactively receives settings and requests from the user, including the atmosphere of the scene, the characteristics of the characters, and the type of action. For example, if a user selects a "dark and scary forest scene" and specifies "zombies wearing old clothes" as the characters to create a horror movie, this information is collected by the device. The device then uses an emotion engine to analyze the user's emotions in real time and identify emotional states such as "tension" and "excitement."

[1345] Automatic generation of computer graphics

[1346] The server generates CG using a generative AI model based on the data received from the device and the results of the emotion engine. For example, if a user requests a "dark and scary forest scene," the server inputs "a dark and scary forest scene" as a prompt into the generative AI model and generates appropriate CG. This generative AI model uses Stable Diffusion and DALL-E 2, among others.

[1347] CG assembly and sequencing

[1348] The generated CG scenes are then combined by the server to create a series of images. The server then adjusts these, including scene transitions, effects, and sound effects, based on feedback from the emotion engine. For example, the darkness and type of sound in a scene are adjusted appropriately based on the results of the emotion engine.

[1349] Video rendering and output

[1350] The server renders the finished video and generates the final data file using rendering engines such as Blender and Unreal Engine. The rendered video is then sent to the device where it can be viewed by the user.

[1351] Interactive operation and immediate feedback

[1352] If a user has new requests while watching a video on their device, they can interactively input change requests. The emotion engine continuously monitors the user's emotions and dynamically adjusts the scenes and effects as needed. For example, if a user requests, "I want to change the scene to one where zombies suddenly appear," that request is sent to the server. The server then issues a new prompt to the generative AI model, instantly generating new CG and reflecting it in the video. In this way, users can interactively generate and change videos in real time.

[1353] Specific examples

[1354] Imagine a user creating a horror movie. The user inputs the following information through an interface:

[1355] "I want to make a horror movie. I'll choose a dark, scary forest scene and designate zombies in old-fashioned clothing as characters. I want to change the scene so that the zombies suddenly appear while I'm watching."

[1356] Based on this prompt, the system can generate, modify, and view videos in real time that match the user's preferences and emotions. The generated videos and CG are stored in a database and can be reused for future productions, significantly improving the efficiency of video production and providing a new entertainment experience.

[1357] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1358] Step 1: Collecting User Input

[1359] The device receives interactive input from the user, including the atmosphere of a particular scene, the characteristics of characters, the type of action, etc. The emotion engine then operates to obtain the user's emotion data.

[1360] Input: User-defined scene and character requests, emotional data (e.g., "dark and scary forest scene," "zombie characters," "scary action").

[1361] Output: Collected user requests and sentiment data.

[1362] Step 2: Send input data to the server

[1363] The terminal transmits the collected user input data and emotion data to the server.

[1364] Input: Collected user desires and sentiment data.

[1365] Output: User wishes and emotion data sent to the server.

[1366] Step 3: Generate the CG scene

[1367] The server generates CG using a generative AI model based on the received data, taking into account the results of the emotion engine and optimizing scenes and effects to match the user's emotions.

[1368] Input: User desires and emotion data.

[1369] Data processing: Generate CG using generative AI models (e.g., Stable Diffusion or DALL-E 2).

[1370] Output: CG data according to user's request.

[1371] Step 4: Combining and composing the CG scene

[1372] The server combines the generated CG scenes into a series of images, and adjusts scene transitions, effects, and sound effects based on information from the emotion engine.

[1373] Input: Generated CG data and emotion engine results.

[1374] Data processing: Combining CG scenes, scene transitions, adding effects and sound effects.

[1375] Output: A complete video sequence.

[1376] Step 5: Rendering the video and sending it to the device

[1377] The server renders the finished video and generates the final data file, which is then sent to the device for viewing by the user.

[1378] Input: A complete video sequence.

[1379] Data processing: Video rendering (using Blender, Unreal Engine, etc.).

[1380] Output: Rendered video file.

[1381] Step 6: Interact and Regenerate

[1382] If a user has a new request while watching a video on their device, they can input it. The emotion engine monitors the user's emotions and dynamically adjusts scenes and effects as needed. The server receives the new request, regenerates the CG, and immediately reflects it in the video.

[1383] Input: The user's new request and emotion data.

[1384] Data processing: Input new prompt text into the generative AI model and regenerate the necessary CG.

[1385] Output: Updated video data.

[1386] (Application example 2)

[1387] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1388] Current virtual store shopping experiences make it difficult for users to easily find products that suit their preferences or to try on items. Furthermore, there is a lack of systems that can recognize user emotions and dynamically generate content, making it impossible to provide a personalized shopping experience tailored to each individual user. This leads to a decline in user satisfaction and engagement.

[1389] The specific processing by the specific processing unit 290 of the data processing device 12 in Application Example 2 is realized by the following means. In this invention, the server includes means for collecting interactive input from the user, means for automatically generating computer graphics using a computer generation method based on the input, and emotion recognition means for recognizing the user's emotions and dynamically adjusting images and effects. This enables a personalized shopping experience in a virtual store that is tailored to the user's preferences and emotions.

[1390] "Means for collecting interactive input from the user" refers to an interface through which the user inputs their desired settings and requests and the system receives them.

[1391] "Means for automatically generating computer graphics using computer generation methods" refers to the process of using AI technology to generate images or videos based on user input.

[1392] "A means of combining generated computer graphics to create a series of images" refers to the process of appropriately arranging individual CGs and assembling them into a single continuous video work.

[1393] "Means for rendering the video and transmitting it to the user terminal" refers to the process of converting the generated video into a format that can be displayed with high quality and delivering it to the user's device.

[1394] "Means that allow the user to interactively input new information while watching the video" refers to a function that allows the user to input new requests or changes to the system in real time while watching the video.

[1395] "Means of regenerating computer graphics based on new input and immediately reflecting it" refers to the process of receiving a new user request, generating new CG, and immediately reflecting that content in the video.

[1396] "Emotion recognition means that recognizes the user's emotions and dynamically adjusts images and effects" refers to technology that analyzes the user's emotional state in real time and dynamically changes images and effects accordingly.

[1397] "Avatar management means that enables users to set up virtual avatars and move freely within a virtual space" refers to a function that enables users to set up their own avatars and operate and move those avatars within a virtual space.

[1398] "Means for providing product information and try-on simulations in a virtual space" refers to a function that allows users to check out products in a virtual store environment and virtually try on outfits.

[1399] The system that realizes this invention automatically generates computer graphics based on interactive user input and combines them to create a series of images, providing a personalized shopping experience in a virtual store. The system program and its processing are described below.

[1400] The server collects user input and uses an AI model to generate CG based on it. In this system, users can customize their own avatars using their smartphones and move them freely in the virtual space. The main software and hardware used include smartphones, Emotion API, generation AI (e.g., DALL-E 2), and a rendering engine (e.g., Unity).

[1401] When a user enters their preferred settings through the smartphone interface, those settings are sent to the server. For example, if they select "casual summer clothing," the Emotion API analyzes the user's facial expressions and recognizes their emotions. The server then uses generative AI such as DALL-E 2 to automatically generate the necessary CG based on this emotional information and the user's settings.

[1402] The generated CG is assembled into a virtual store scene using a rendering engine such as Unity. The virtual store scene is equipped with a function that allows the user's avatar to move freely, and product information and a try-on simulation are also provided. The final rendered image is sent to a smartphone device, where the user can view and interact with it.

[1403] For example, when a user clicks on a specific product in a store, detailed information about that product and a try-on simulation are displayed. Furthermore, when the Emotion API recognizes the user's "interest" from their facial expression, they are dynamically guided to products and sections that match their interests.

[1404] Prompt Sentence Examples

[1405] Prompt for the generator: "Generate casual summer clothing for a virtual avatar."

[1406] Prompt for emotion recognition API: "Evaluate user interest from facial expressions and guide them to specific products."

[1407] As described above, this system dynamically generates and adjusts images and effects based on the user's emotions and inputs, providing a personalized virtual shopping experience tailored to each individual user.

[1408] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1409] Step 1:

[1410] A user launches the application on their smartphone and configures their virtual avatar. They upload a photo of their face and customize the avatar's appearance and clothing. This forms the avatar's basic information and the user's preferred settings as input data. The generative AI model is prompted with the following sentence: "Generate casual summer clothing for the virtual avatar."

[1411] Step 2:

[1412] The device sends the avatar data and user preferences to the server. The server receives this data and uses the Emotion API to analyze the user's facial expressions in real time to obtain emotional information. This emotional information is reflected in future CG generation and scene construction. The prompt to the emotion recognition API is: "Please evaluate the user's interest from their facial expressions and guide them to specific products."

[1413] Step 3:

[1414] The server uses a generation AI (e.g., DALL-E 2) to generate the necessary CG in the virtual space based on the user's preferred settings and emotional information. The generation process creates the specified costumes and objects required for the scene and generates them as 3D models. The input for this step is the user's settings and emotional information, and the output is the generated CG data.

[1415] Step 4:

[1416] The server constructs a virtual store scene based on the generated CG data. Using a rendering engine (e.g., Unity), the scene is constructed by combining CG. Within this scene, an area is placed where the user's avatar can move freely, allowing for product display and a try-on simulation. The input for this step is the generated CG data, and the output is rendered scene data.

[1417] Step 5:

[1418] The server sends the rendered scene data to the user's smartphone. The user can then freely walk around the virtual store on their smartphone, browsing product information and trying on items. Selecting a specific item displays detailed information about that item and a try-on simulation. The input for this step is the rendered scene data, and the output is an interactive display on the user's device.

[1419] Step 6:

[1420] Users can interactively request new products or scenes that interest them in the virtual store. The new request from the terminal is sent to the server, which regenerates the necessary CG based on the request and immediately reflects it in the scene. The input of this step is the user's new request data, and the output is the updated scene data.

[1421] This is the flow of the system program that realizes this application example. The specific operations and data processing / calculation performed at each step provide the user with a personalized shopping experience.

[1422] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1423] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1424] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1425] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1426] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1427] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1428] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1429] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1430] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1431] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1432] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1433] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1434] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1435] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1436] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1437] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1438] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1439] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1440] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1441] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1442] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1443] The following is further disclosed regarding the above embodiment.

[1444] (Claim 1)

[1445] means for collecting interactive input from a user;

[1446] means for automatically generating computer graphics using computer generation techniques based on the input;

[1447] A means for assembling the generated computer graphics to create a series of images;

[1448] means for rendering and transmitting said video to a user terminal;

[1449] means for enabling a user to interactively input new information while viewing the video;

[1450] means for generating computer graphics again based on the new input and immediately reflecting the new input;

[1451] A system including:

[1452] (Claim 2)

[1453] 10. The system of claim 1, further comprising means for automatically applying scene effects and sound effects based on user input data.

[1454] (Claim 3)

[1455] 10. The system of claim 1, further comprising database means for storing the generated computer graphics and images so that they can be reused in subsequent productions.

[1456] "Example 1"

[1457] (Claim 1)

[1458] means for collecting interactive input from a user;

[1459] means for automatically generating computer graphics using a generative AI model based on the input;

[1460] A means for assembling the generated computer graphics to create a series of images;

[1461] means for rendering and transmitting said video to a user terminal;

[1462] means for enabling a user to interactively input new information while viewing the video;

[1463] means for generating computer graphics again based on the new input and immediately reflecting the new input;

[1464] a database means for storing the generated computer graphics and images so that they can be reused in the next production;

[1465] A system including:

[1466] (Claim 2)

[1467] 10. The system of claim 1, further comprising means for automatically applying scene effects and sound effects based on user input data.

[1468] (Claim 3)

[1469] 10. The system according to claim 1, further comprising means for temporarily storing the generated computer graphics and using the generated graphics for subsequent generation processing.

[1470] "Application Example 1"

[1471] (Claim 1)

[1472] means for collecting interactive input from a user;

[1473] means for automatically generating computer graphics using computer generation techniques based on the input;

[1474] A means for assembling the generated computer graphics to create a series of images;

[1475] means for rendering and transmitting said video to a user terminal;

[1476] means for enabling a user to interactively input new information while viewing the video;

[1477] means for generating computer graphics again based on the new input and immediately reflecting the new input;

[1478] A means for generating and changing product displays and in-store decorations in real time based on collected user preferences;

[1479] A system including:

[1480] (Claim 2)

[1481] 10. The system of claim 1, further comprising means for automatically applying scene effects and sound effects based on user input data.

[1482] (Claim 3)

[1483] 10. The system of claim 1, further comprising database means for storing the generated computer graphics and images so that they can be reused in subsequent productions.

[1484] "Example 2: Combining Emotion Engines"

[1485] (Claim 1)

[1486] means for collecting interactive input from a user;

[1487] means for automatically generating computer graphics using a generative AI model based on the input;

[1488] A means for assembling the generated computer graphics to create a series of images;

[1489] means for rendering and transmitting said video to a user terminal;

[1490] means for enabling a user to interactively input new information while viewing the video;

[1491] means for generating computer graphics again based on the new input and immediately reflecting the new input;

[1492] means for recognizing a user's emotions in real time and adjusting scenes and effects based on said emotions;

[1493] A system including:

[1494] (Claim 2)

[1495] 10. The system of claim 1, further comprising means for automatically applying scene effects and sound effects based on user input data.

[1496] (Claim 3)

[1497] 10. The system of claim 1, further comprising database means for storing the generated computer graphics and images so that they can be reused in subsequent productions.

[1498] "Application example 2 when combining emotion engines"

[1499] (Claim 1)

[1500] means for collecting interactive input from a user;

[1501] means for automatically generating computer graphics using computer generation techniques based on the input;

[1502] A means for assembling the generated computer graphics to create a series of images;

[1503] means for rendering and transmitting said video to a user terminal;

[1504] means for enabling a user to interactively input new information while viewing the video;

[1505] means for generating computer graphics again based on the new input and immediately reflecting the new input;

[1506] an emotion recognition means for recognizing a user's emotion and dynamically adjusting images and effects;

[1507] A system including:

[1508] (Claim 2)

[1509] 10. The system of claim 1, further comprising means for automatically applying scene effects and sound effects based on user input data.

[1510] (Claim 3)

[1511] 10. The system of claim 1, further comprising database means for storing the generated computer graphics and images so that they can be reused in subsequent productions.

[1512] (Claim 4)

[1513] 2. The system according to claim 1, further comprising an avatar management means for enabling a user to set a virtual avatar and move freely within the virtual space.

[1514] (Claim 5)

[1515] 2. The system according to claim 1, further comprising means for providing product information and a try-on simulation in the virtual space. [Explanation of symbols]

[1516] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for collecting interactive input from a user; means for automatically generating computer graphics using computer generation techniques based on the input; A means for assembling the generated computer graphics to create a series of images; means for rendering and transmitting said video to a user terminal; means for enabling a user to interactively input new information while viewing the video; means for generating computer graphics again based on the new input and immediately reflecting the new input; A system including:

2. 10. The system of claim 1, further comprising means for automatically applying scene effects and sound effects based on user input data.

3. 2. The system of claim 1, further comprising database means for storing the generated computer graphics and images so that they can be reused in subsequent productions.

Images