system

Abstract

Provide a system. The method includes: generating a 3D avatar based on prompts received from a user; A means for acquiring a camera image of a user in real time and analyzing the image to detect the user's facial expressions and movements; A means for reflecting the detected facial expressions and movements in a 3D avatar so that the avatar moves in response to the user's movements; A means for transmitting the moving 3D avatar image to a remote conference or live streaming A system including:

Description

[Technical Field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including an instruction sentence related to a description of the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] Japanese Patent Publication No. 2022-180282 Summary of the Invention [Problem to be solved by the invention]

[0004] In conventional remote meetings and live streaming, individual avatar design and movement linking are complex and require advanced expertise, making them difficult for general users to use. Furthermore, the accuracy of camera image analysis is low, making it difficult to link movements in real time, making it difficult to accurately reproduce user movements. This limits the expressiveness of individual users in remote meetings and live streaming. [Means for solving the problem]

[0005] The present invention provides a means for generating a 3D avatar based on prompts received from a user. The system further includes means for capturing a user's camera image in real time and analyzing the image to detect the user's facial expressions and movements. The system also includes means for reflecting the detected facial expressions and movements in the 3D avatar, causing the avatar to move in response to the user's movements. The system also includes means for transmitting the moving 3D avatar image to a remote conference or live streaming. This means allows for the use of a 3D avatar that reflects the user's movements easily and with high accuracy, thereby improving individual expressiveness in remote conferences and live streaming.

[0006] A "prompt" refers to the natural language or keywords that a user inputs into the system, which then generates a 3D avatar.

[0007] A "3D avatar" refers to a three-dimensional virtual character generated using computer graphics and customized based on user specifications.

[0008] "Camera footage" refers to video data used to capture user movements and is acquired in real time.

[0009] "Analyzing video" refers to the process of extracting specific information from captured camera footage, particularly detecting the user's facial expressions and movements.

[0010] "Detecting facial expressions and movements" refers to identifying the user's facial features and body posture and other movements during the video analysis process and acquiring this as data.

[0011] "Real-time" refers to a state in which processing and reactions are carried out immediately without delay, meaning that the movement link between the user and the 3D avatar is instantaneous.

[0012] "Reflecting movements" refers to synthesizing the detected user's facial expressions and movements into the movements of a 3D avatar, so that the avatar behaves in the same way as the user.

[0013] A "remote meeting" refers to an online meeting conducted between participants in remote locations via the Internet.

[0014] "Live streaming" refers to transmitting video and audio in real time over the Internet and broadcasting to a large audience. [Brief explanation of the drawings]

[0015] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Figure 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. [Figure 3] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Figure 5] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 10 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 10 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] FIG. 3 is a sequence diagram showing a processing flow of the data processing system according to the first embodiment. [Figure 12] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 1. [Figure 13] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 10 is a sequence diagram showing the flow of processing in the data processing system in Application Example 2 when an emotion engine is combined. DETAILED DESCRIPTION OF THE INVENTION

[0016] An example of an embodiment of a system according to the technology of the present disclosure will be described below with reference to the accompanying drawings.

[0017] First, the terms used in the following description will be explained.

[0018] In the following embodiments, a coded processor (hereinafter simply referred to as a "processor") may be a single arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be a single type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), and an APU (Accelerated Processing Unit).

[0019] In the following embodiments, a coded RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by a processor.

[0020] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0021] In the following embodiments, a communication I / F (Interface) with a symbol is an interface including a communication processor, an antenna, etc. The communication I / F controls communication between multiple computers. Examples of communication standards applied to the communication I / F include wireless communication standards including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0022] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. Furthermore, in this specification, the same concept as "A and / or B" is also applied when three or more things are expressed connected by "and / or."

[0023] [First embodiment]

[0024] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0025] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0026] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0027] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0028] The reception device 38 includes a touch panel 38A, a microphone 38B, and the like, and receives user input. The touch panel 38A detects contact with an indicator (for example, a pen or a finger) to receive user input by the touch of the indicator. The microphone 38B detects the user's voice to receive user input by voice. The control unit 46A transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the data indicating the user input.

[0029] The output device 40 includes a display 40A and a speaker 40B, and presents data to the user 20 by outputting the data in a form of expression that the user 20 can perceive (for example, audio and / or text). The display 40A displays visible information such as text and images in accordance with instructions from the processor 46. The speaker 40B outputs audio in accordance with instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0030] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 control the exchange of various information between the processor 46 and the processor 28 via the network 54.

[0031] FIG. 2 shows an example of the main functions of the data processing device 12 and the smart device 14.

[0032] 2, in the data processing device 12, a specific process is performed by the processor 28. A specific processing program 56 is stored in the storage 32. The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0033] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0034] In the smart device 14, the processor 46 performs the reception output process. The storage 50 stores a reception output program 60. The reception output program 60 is used in conjunction with the specific processing program 56 by the data processing system 10. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0035] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0036] System Overview

[0037] This invention is a system that allows users to input prompts, generate original 3D avatars based on those prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar, allowing the avatar to move in accordance with the user's movements.

[0038] Program processing explanation (Example)

[0039] Avatar generation

[0040] The user enters a prompt, which the device sends to the system. For example, if the user enters a prompt such as "a blue character with deer antlers," the device sends this information to the server. The server receives the prompt and uses a 3D modeling engine to generate a corresponding 3D avatar. The generated avatar is temporarily stored on the server and then sent as data to the user's device. The device receives this data and displays the generated avatar to the user.

[0041] Camera footage analysis

[0042] When a user starts a remote meeting or live broadcast using a camera connected to their device, the video is captured in real time. The device analyzes the camera video and detects facial features and body movements. For example, when a user smiles or moves their hands, computer vision technology is applied to recognize these movements in real time.

[0043] Avatar Action Link

[0044] The device then moves each part of the 3D avatar it generates based on the analysis results. For example, if the user smiles, the 3D avatar's face will also move to smile. Similarly, if the user moves their hand, the corresponding movement is transmitted to the avatar. In this way, the user's movements are directly linked to the 3D avatar.

[0045] Real-time video streaming

[0046] Users connect their 3D avatars to remote conference and live streaming software. Specifically, they can set the 3D avatar as a virtual camera in video conferencing software such as Zoom or streaming software such as YouTube® Live. The device transmits video of the 3D avatar moving in real time through this virtual camera. When the user speaks or moves, the avatar also moves in the same way, allowing viewers and participants to communicate with the user through the avatar.

[0047] Specific examples

[0048] 1. Prompt Input: The user types "a wizard-looking character."

[0049] 2. Avatar generation: The server receives the prompt, generates a wizard-like 3D avatar, and sends it to the user's device.

[0050] 3. Camera analysis: The user starts the meeting through the camera, and the device analyzes the user's facial expressions and movements.

[0051] 4. Avatar Reflection: Based on the analysis results, the 3D avatar moves in accordance with the user's movements. For example, if the user raises their hand, the avatar will raise their hand as well.

[0052] 5. Broadcasting: The user sets up the generated 3D avatar in Zoom, and the avatar acts on the user's behalf during the meeting.

[0053] Thus, the present invention provides a system that allows users to easily generate 3D avatars and use them in remote meetings and live streaming.

[0054] The processing flow will be explained below.

[0055] Program processing flow

[0056] Avatar generation process

[0057] Step 1:

[0058] The user types a prompt into the system, for example, "a blue character with deer antlers."

[0059] Step 2:

[0060] The terminal sends the user's prompt data to the server.

[0061] Step 3:

[0062] The server receives the prompt and parses and understands its contents.

[0063] Step 4:

[0064] The server's 3D modeling engine generates a 3D avatar based on the prompts.

[0065] Step 5:

[0066] The server stores the generated 3D avatar data and sends it to the user's device.

[0067] Step 6:

[0068] The terminal displays the received 3D avatar data and allows the user to confirm it.

[0069] Camera image analysis processing

[0070] Step 1:

[0071] The user connects the camera to their computer and launches the remote conferencing software or live streaming software.

[0072] Step 2:

[0073] The device acquires video data from the camera in real time.

[0074] Step 3:

[0075] The device analyzes the captured camera footage.

[0076] Step 4:

[0077] The device detects the user's facial features and body posture, for example, identifying the user's eye and mouth movements and hand positions.

[0078] Avatar movement link processing

[0079] Step 1:

[0080] The device moves each part of the 3D avatar based on the analysis results.

[0081] Step 2:

[0082] For example, if the user smiles, the 3D avatar's face will also smile. Similarly, if the user moves their hand, the avatar's hand will also move in a similar way.

[0083] Step 3:

[0084] The device repeats this real-time processing, constantly updating the 3D avatar in accordance with the user's movements.

[0085] Real-time video distribution processing

[0086] Step 1:

[0087] The user connects the generated 3D avatar to remote conferencing software or live streaming software, for example, by setting it as a virtual camera on Zoom or YouTube Live.

[0088] Step 2:

[0089] The device continues to maintain a link to the 3D avatar's movements.

[0090] Step 3:

[0091] The device transmits images of a 3D avatar moving in real time via a virtual camera to remote conferencing software or live streaming software.

[0092] Step 4:

[0093] Users communicate through 3D avatars during remote meetings and live broadcasts. For example, when a user speaks, the 3D avatar's mouth moves in the same way.

[0094] In this way, the system reflects the user's movements through each step in real time onto a 3D avatar, allowing the footage to be used for remote meetings and live streaming.

[0095] Example 1

[0096] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0097] In modern remote meetings and live streaming, it is difficult for users to easily create and use original 3D avatars that reflect their own facial expressions and movements in real time. Another problem is that existing technologies lack the ability to accurately detect user movements through analysis of camera footage and then coordinate avatar movements in real time. By resolving these issues, users can enjoy more natural and intuitive remote communication.

[0098] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0099] In this invention, the server includes means for generating a 3D avatar based on a prompt received from the user, means for the server to analyze the prompt using a generation AI model and generate a corresponding 3D avatar, means for acquiring a camera image of the user in real time and analyzing the image to detect the user's facial expressions and movements, means for reflecting the detected facial expressions and movements in the 3D avatar so that the avatar moves in response to the user's movements, and means for transmitting a video of the moving 3D avatar to a remote conference or live streaming. This allows users to easily generate a 3D avatar, and the avatar moves in real time in coordination with the user's movements, enabling more effective remote communication.

[0100] A "prompt" is a sentence or phrase that a user enters to indicate the features or elements required to generate a 3D avatar.

[0101] A "generative AI model" is a type of artificial intelligence technology that analyzes prompts received from users and generates corresponding 3D avatars.

[0102] A "3D avatar" is a three-dimensional virtual character generated based on a user's characteristics and preferences.

[0103] The "server" is a computing system that receives prompts, generates a 3D avatar using a generative AI model, and transmits the data to the user's device.

[0104] "Camera video" refers to video data acquired in real time from a camera device used by a user.

[0105] "Real-time analysis" is a technology that instantly processes camera footage and detects the user's facial expressions and movements.

[0106] "Means for detecting facial expressions and movements" refers to computer vision technology for recognizing the user's facial features and body movements.

[0107] A "virtual camera" is a technology that allows software-generated images to be treated as if they were real camera images.

[0108] A "remote conference" is a conference in which multiple users participate from remote locations via the Internet.

[0109] "Live streaming" refers to the distribution of video and audio to viewers in real time via the Internet.

[0110] The present invention provides a system for generating a 3D avatar based on prompts input by a user, and for the avatar to move in real time in response to the user's actions. The system includes the following means.

[0111] 1. Enter and submit the prompt

[0112] It starts with the user entering a prompt into the terminal interface, for example, "a red dragon with wings." Once the prompt is entered, the terminal sends the information to the system.

[0113] 2. Prompt Processing and Avatar Creation

[0114] The device sends the prompt to a server, which uses a generative AI model (such as DALLE-2 or DeepArt) to analyze the prompt and generate a corresponding 3D avatar. For example, if a user enters the prompt "a red dragon with wings," the server can generate the avatar using a 3D modeling tool like Blender.

[0115] 3. Save and send your avatar

[0116] The server temporarily stores the generated 3D avatar and sends the data to the device. The device receives the data and displays it as a preview to the user. An initial preview image of the generated avatar is displayed on the device screen.

[0117] 4. Real-time analysis of camera footage

[0118] A user starts a remote conference or live broadcast using a camera connected to their device. The device uses an application (such as OpenCV or MediaPipe) to analyze the camera image in real time and detect facial features and body movements. When the user smiles, the change in facial expression is detected in real time.

[0119] 5. Avatar Action Link

[0120] The device updates the behavior of the 3D avatar in real time based on the analysis results. For example, if the user raises their hand, the device detects that movement and mirrors the avatar's hand by raising it. Similarly, facial expressions are updated simultaneously; if the user smiles, the avatar also smiles.

[0121] 6. Real-time video distribution

[0122] The user sets the generated 3D avatar as a virtual camera in remote conferencing software such as Zoom or YouTube Live. The device then uses virtual camera software such as OBS Virtual Camera to transmit the avatar's image, which is updated in real time. When the user speaks or moves, their movements are reflected in the avatar and displayed to viewers and participants in real time. For example, if a user waves their hand during a meeting, the avatar will also wave.

[0123] In this way, a 3D avatar generated from user-entered prompts can be linked to the user's movements in real time and used in remote meetings and live broadcasts.

[0124] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0125] Step 1:

[0126] The user enters a prompt (e.g., "a red dragon with wings") into the terminal interface. The entered prompt is sent to the system by the terminal. This prompt becomes input data to the system. The terminal sends the prompt to the server in JSON format, for example.

[0127] Step 2:

[0128] The server analyzes the received prompt text. First, it inputs the input prompt text into a generative AI model (e.g., DALLE-2 or DeepArt). The generative AI model analyzes the prompt text and generates design data for the corresponding 3D avatar. This data includes information such as the avatar's shape, color, and movement patterns. The server generates the 3D avatar using 3D modeling software (e.g., Blender) based on the design data. The generated 3D avatar is temporarily stored on the server.

[0129] Step 3:

[0130] The server sends the temporarily saved 3D avatar data to the device. The device analyzes the received 3D avatar design data and displays a preview to the user. This display includes the overall image and main features of the avatar. The user can check the avatar and request changes or adjustments as necessary.

[0131] Step 4:

[0132] To start a remote meeting or live broadcast, a user begins capturing video using a camera. The device acquires video from the camera in real time and analyzes it using computer vision technologies such as OpenCV and MediaPipe. This analysis detects facial features and body movements. Analyzed data includes the user's facial expressions, eye movements, and hand movements.

[0133] Step 5:

[0134] The device then uses the analysis results to adjust each part of the 3D avatar in real time. For example, if the user raises their hand, the device detects that movement and adjusts the 3D avatar's hand so that it also raises. Similarly, if the user smiles, the avatar's face also changes to a smile. The device uses this data to update the 3D avatar's movements in real time.

[0135] Step 6:

[0136] Users set up a 3D avatar as a virtual camera in remote conferencing or live streaming software (e.g., Zoom or YouTube Live). The device then uses virtual camera software such as OBS Virtual Camera to transmit a live image of the avatar, which is updated in real time. When the user speaks or moves, their actions are reflected in the avatar in real time, allowing for natural communication with viewers and participants.

[0137] (Application example 1)

[0138] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0139] There is a demand for virtual stores to provide users with a realistic shopping experience, guided in real time. However, existing systems lack avatar guidance functions that synchronize with the user's movements, and users have difficulty in easily customizing and using 3D avatars. In addition, to provide more attractive virtual store guidance, it is desirable for the avatar to synchronize with the user's movements and facial expressions in real time.

[0140] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0141] In this invention, the server includes: means for generating a 3D avatar based on prompts received from the user; means for capturing camera footage of the user in real time and analyzing the footage to detect the user's facial expressions and movements; means for reflecting the detected facial expressions and movements in the 3D avatar so that the avatar moves in response to the user's movements; means for displaying the moving 3D avatar on the user's device; and means for the user to move with the avatar in a virtual space and for the avatar to provide guidance as the user moves. This allows the user to easily create a customized 3D avatar based on their own movements and prompts, enabling a shopping experience in which they are guided through a virtual store in real time.

[0142] A "prompt" is a word or phrase entered by a user to indicate an instruction or request.

[0143] A "3D avatar" is a three-dimensional virtual character that is generated based on the user's prompts and camera footage, and moves in response to the user's movements and facial expressions.

[0144] "User's camera footage" refers to footage captured in real time by the camera of the device used by the user.

[0145] "Analysis of camera footage" is the process of detecting the user's facial expressions and movements from camera footage acquired in real time and evaluating their features and posture.

[0146] "Real-time" refers to a state in which operations and actions are reflected instantly with almost no delay.

[0147] "Moving 3D avatar video" refers to a 3D avatar that moves in real time in response to the user's facial expressions and movements, and is displayed as an image or video.

[0148] "Virtual space" refers to an imaginary environment or space generated using computer graphics or virtual reality technology.

[0149] A "virtual store" is a virtual shopping environment created using the Internet and virtual reality technology that has the same functions as a physical store.

[0150] "Means for providing guidance" refers to the function of the avatar to provide information to the user and guide the user to their destination when the user moves within the virtual space.

[0151] A "device" is an electronic device used by a user, such as a smartphone, tablet, or computer.

[0152] System implementation

[0153] This system generates a 3D avatar based on prompts received from the user, analyzes the user's camera footage in real time to detect the user's facial expressions and movements, and reflects those movements in the 3D avatar. Specifically, the following hardware and software are required:

[0154] Hardware: devices such as smartphones or tablets, cameras

[0155] Software: 3D modeling engine (Unity, Unreal Engine, etc.), facial recognition software (OpenCV, Dlib, etc.)

[0156] Program processing explanation

[0157] 1. Prompt input and avatar generation

[0158] A user uses a smartphone app to input a prompt such as "Nordic-style guide character." For example, the user types "Nordic-style guide."

[0159] The device sends this prompt to the server, which uses the generative AI model to generate a corresponding 3D avatar, which is then sent from the server to the device and displayed to the user.

[0160] 2. Analysis of camera footage

[0161] As users begin to move around the virtual store through the camera, the device captures camera footage and uses facial recognition software such as OpenCV or Dlib to detect facial features and body poses in real time.

[0162] 3. Movement and Avatar Movement Links

[0163] The detected facial expressions and movements are then reflected in a 3D avatar in real time based on the user's movements: for example, if the user raises their hand, the avatar will do the same.

[0164] 4. Real-time store information

[0165] The device provides a mechanism for a moving 3D avatar to be displayed on the user's device. As the user moves through the virtual space, the avatar walks ahead of the user, providing real-time guidance and explaining the features of various products and stores.

[0166] Specific examples

[0167] 1. Example of prompt input:

[0168] For example, a user enters "Scandinavian style guide."

[0169] 2. Avatar generation example:

[0170] The server receives the prompt, generates a Nordic-style guide character using a 3D modeling engine, and sends it to the user's device.

[0171] 3. Camera analysis example:

[0172] As users begin to move around the virtual store using their device's camera, the device analyzes the camera footage and detects facial features and body posture in real time.

[0173] 4. Avatar reflection example:

[0174] Based on the detection results, the 3D avatar moves in sync with the user's movements. For example, when the user smiles, the avatar also smiles.

[0175] 5. Store location example:

[0176] As the user moves around the virtual space with the device, the avatar explains product information and makes comments based on the product the user is looking at.

[0177] This system allows users to easily generate 3D avatars, further enhancing the shopping experience in virtual stores.

[0178] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0179] Step 1:

[0180] Prompt input and submit

[0181] The user starts the smartphone app and inputs a prompt for the 3D avatar they want to create, such as a "Nordic-style guide character." The input prompt is sent to the server by the device.

[0182] Input: User prompt (e.g. "Nordic style guide")

[0183] Output: Send prompt to server

[0184] Specific operation: The user enters a prompt using text input and presses the send button. The device sends this prompt to the server as an HTTP request.

[0185] Step 2:

[0186] 3D avatar generation

[0187] The server analyzes the received prompt using a generative AI model to generate a 3D avatar with the specified characteristics. The generated 3D avatar is temporarily stored on the server and then sent to the device as data.

[0188] Input: prompt (e.g. "Nordic style guide")

[0189] Output: Generated 3D avatar

[0190] How it works: The server uses a generative AI model to analyze the prompt and generates a 3D avatar using Unity or Unreal Engine, which is then stored on the server and sent to the device as an HTTP response.

[0191] Step 3:

[0192] Displaying 3D avatars

[0193] The device displays the 3D avatar received from the server.

[0194] Input: 3D avatar data sent from the server

[0195] Output: 3D avatar displayed on device

[0196] Specific operation: The device deserializes the received 3D avatar data and displays it to the user using a Unity viewer or similar.

[0197] Step 4:

[0198] Camera footage capture and analysis

[0199] As the user moves around the virtual store, the device captures and analyzes camera footage in real time, using OpenCV and Dlib to detect facial features and body poses.

[0200] Input: User's camera video

[0201] Output: Analysis results of user's facial expressions and movements

[0202] How it works: The device's camera captures video in real time, and then uses libraries such as OpenCV and Dlib to analyze facial features and body movements. The analysis results are output as data showing the user's current facial expressions and movements.

[0203] Step 5:

[0204] Avatar movement reflection

[0205] The device then uses the analysis results to update the 3D avatar's movements in real time, adjusting the avatar to perform similar movements based on the user's movements.

[0206] Input: Analyzed user facial expression and movement data

[0207] Output: A moving 3D avatar

[0208] Specific operation: Based on the analysis results, each joint and facial expression of the 3D avatar is updated in real time using a game engine such as Unity.

[0209] Step 6:

[0210] Guidance within the virtual store

[0211] As users move through the virtual space, a 3D avatar on the device acts as a real-time guide.

[0212] Input: User movement data in virtual space

[0213] Output: Guidance information using 3D avatars

[0214] Specific operation: The device uses the user's location and movement data to provide a 3D avatar with product information and directions. For example, when the user stands in front of a nearby product, the avatar will explain the product's features.

[0215] Step 7:

[0216] Display on user devices

[0217] The terminal displays a moving 3D avatar and real-time information on the user's device, and the user can view and interact with the avatar's guidance through their own device.

[0218] Input: A moving 3D avatar and related information

[0219] Output: What is displayed on the user's device

[0220] Specific operation: The terminal renders the display data and displays it in real time on the user's device, allowing the user to experience the virtual store through interaction with a 3D avatar on the device.

[0221] Furthermore, an emotion engine that estimates the user's emotion may be combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59 and perform identification processing using the user's emotion.

[0222] System Overview

[0223] This invention is a system that allows users to input prompts, generate original 3D avatars based on those prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar. It also has the ability to recognize the user's emotions using an emotion engine and reflect those emotions in the 3D avatar.

[0224] Program processing explanation (Example)

[0225] Avatar generation

[0226] A user enters a prompt into the system, and the device sends the prompt to the server. For example, if a user enters a prompt such as "a blue character with deer antlers," the device sends this information to the server. The server receives the prompt and uses a 3D modeling engine to generate a corresponding 3D avatar. This generated avatar is temporarily stored on the server and then sent as data to the user's device. The device receives this data and displays the generated avatar to the user.

[0227] Camera video and audio analysis

[0228] When a user starts a remote meeting or live broadcast using a camera and microphone connected to their device, video and audio are captured in real time. The device acquires and analyzes the camera video and audio data. Specifically, the device detects facial features and body posture, and simultaneously analyzes the audio data to recognize the user's emotional state. For example, when a user smiles or moves their hands, computer vision and voice recognition technologies are applied to recognize these movements in real time.

[0229] Emotion recognition by emotion engine

[0230] The device passes the captured video and audio data to the emotion engine, which recognizes the user's emotions in real time. The emotion engine detects positive (happiness, satisfaction) and negative (anger, sadness) emotions from facial and audio data. For example, when a user speaks with a smile, the emotion engine recognizes that the user is "happy" from their voice and facial expression.

[0231] Linking avatar movements and emotions

[0232] Based on the analysis results and the recognition results of the emotion engine, the device links each part of the generated 3D avatar with its emotional expression. For example, if the user is smiling and happy, the 3D avatar's face will also smile and show a happy expression. Similarly, if the user is angry, the avatar's face will show an angry expression. In this way, the user's movements and emotions are reflected directly in the 3D avatar in real time.

[0233] Real-time video streaming

[0234] The user connects the generated 3D avatar to remote conferencing software or live streaming software. Specifically, the 3D avatar can be set as a virtual camera in video conferencing software such as Zoom or streaming software such as YouTube Live. The device sends video of the 3D avatar, which moves and expresses emotions in real time, to the remote conferencing software or streaming software via this virtual camera. When the user speaks or moves, the avatar also expresses the same movements and emotions, allowing viewers and participants to communicate with the user through the avatar.

[0235] Specific examples

[0236] 1. Prompt Input: The user types "a wizard-looking character."

[0237] 2. Avatar generation: The server receives the prompt, generates a wizard-like 3D avatar, and sends it to the user's device.

[0238] 3. Video and audio analysis: The user initiates a remote meeting via a camera and microphone, and the device analyzes the user's facial expressions and voice.

[0239] 4. Emotion recognition: The emotion engine recognizes emotions from the user's video and audio, detecting, for example, a "happy" state.

[0240] 5. Avatar Reflection: Based on the analysis results and emotion recognition results, a 3D avatar expresses the user's movements and emotions in real time.

[0241] 6. Distribution: The user sets the generated 3D avatar in remote conferencing software, and the avatar acts on behalf of the user during the meeting and expresses emotions.

[0242] Thus, the present invention provides a system that generates a 3D avatar that reflects a user's movements and emotions in real time and can be used in remote meetings and live streaming.

[0243] The processing flow will be explained below.

[0244] Program processing flow

[0245] Avatar generation process

[0246] Step 1:

[0247] The user types a prompt into the system, for example, "a blue character with deer antlers."

[0248] Step 2:

[0249] The terminal sends the user's prompt data to the server.

[0250] Step 3:

[0251] The server receives the prompt and parses and understands its contents.

[0252] Step 4:

[0253] The server's 3D modeling engine generates a 3D avatar based on the prompts.

[0254] Step 5:

[0255] The server stores the generated 3D avatar data and sends it to the user's device.

[0256] Step 6:

[0257] The terminal displays the received 3D avatar data and allows the user to confirm it.

[0258] Camera video and audio analysis processing

[0259] Step 1:

[0260] The user connects a camera and microphone to their computer and launches remote conferencing software or live streaming software.

[0261] Step 2:

[0262] The device acquires video data in real time from the camera and audio data from the microphone.

[0263] Step 3:

[0264] The device analyzes the captured camera footage.

[0265] Step 4:

[0266] The device detects the user's facial features and body posture, for example, identifying the user's eye and mouth movements and hand positions.

[0267] Step 5:

[0268] The device analyzes the acquired voice data and recognizes the user's emotional state from the voice.

[0269] Emotion recognition processing using an emotion engine

[0270] Step 1:

[0271] The device passes the acquired video and audio data to the emotion engine.

[0272] Step 2:

[0273] The emotion engine analyzes the user's facial expressions from video data and recognizes emotions by analyzing the tone and speaking style from audio data.

[0274] Step 3:

[0275] For example, if a user speaks with a smile, the emotion engine will recognize from this data that the user is "happy."

[0276] Avatar movement and emotion link processing

[0277] Step 1:

[0278] The device moves each part of the 3D avatar based on the analysis results and the recognition results of the emotion engine.

[0279] Step 2:

[0280] For example, if the user is smiling and "happy," the 3D avatar's face will also smile and move to show a happy expression. Similarly, if the user moves their hand, the avatar's hand movement will reflect that movement.

[0281] Step 3:

[0282] The device repeats this real-time processing, constantly updating the 3D avatar to match the user's movements and emotions.

[0283] Real-time video distribution processing

[0284] Step 1:

[0285] The user connects the generated 3D avatar to remote conferencing software or live streaming software, for example, by setting it as a virtual camera on Zoom or YouTube Live.

[0286] Step 2:

[0287] The device maintains a link between the movements and emotions of the 3D avatar.

[0288] Step 3:

[0289] The device transmits images of a 3D avatar that moves and expresses emotions in real time via a virtual camera to remote conferencing software or live streaming software.

[0290] Step 4:

[0291] Users communicate through 3D avatars during remote meetings and live broadcasts. For example, when a user speaks, the 3D avatar moves its mouth in the same way and changes its facial expression depending on the recognized emotion.

[0292] In this way, the system reflects the user's movements and emotions through each step in real time onto a 3D avatar, allowing the footage to be used for remote meetings and live streaming.

[0293] Example 2

[0294] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0295] When users use 3D avatars in remote meetings or live streaming, conventional systems often fail to accurately reflect the user's movements and facial expressions. Furthermore, the 3D avatars generated based on prompts may not match the user's preferences, making customization difficult. This reduces the quality of the user experience and makes efficient communication difficult.

[0296] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for generating a 3D avatar based on a prompt received from the user, a means for acquiring the user's camera video and audio in real time and analyzing the video and audio to detect the user's facial expressions and movements, a means for recognizing the user's emotions using an emotion recognition engine, a means for reflecting the detected facial expressions, movements, and recognized emotions in the 3D avatar so that the avatar moves in accordance with the user's movements and movements, and a means for transmitting the video and audio of the moving 3D avatar to a remote conference or live streaming. This makes it possible to provide a customizable 3D avatar based on the prompt, reflecting the user's movements and movements in real time.

[0297] A "prompt" is text information that indicates a request or instruction that a user inputs to a system.

[0298] A "3D avatar" is a character or icon that can be displayed in three-dimensional space and operates in real time by reflecting the user's movements and emotions.

[0299] "Camera footage" refers to real-time video data captured by a camera connected to a user's terminal.

[0300] "Audio Data" refers to real-time audio data captured by a microphone connected to a user's terminal.

[0301] "Analyzing" refers to the process of analyzing the acquired data and extracting information such as feature points, posture, and emotions.

[0302] An "emotion recognition engine" refers to software or algorithms for detecting a user's emotional state from facial expression data and voice data.

[0303] A "remote conference" is a conference conducted in real time by multiple participants using video and audio over the Internet.

[0304] "Live streaming" refers to a form of broadcasting in which video and audio are transmitted in real time to a large number of viewers via the Internet.

[0305] "Customizable" refers to the ability to change the appearance and behavior of the generated 3D avatar based on user specifications.

[0306] "Detecting facial expressions and movements" refers to the process of recognizing facial features and body posture from camera footage and extracting that information.

[0307] This invention is a system that allows users to input prompts, generate original 3D avatars based on the prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar. It also has the function of recognizing the user's emotions using an emotion engine and reflecting them in the 3D avatar.

[0308] Hardware and Software Configuration

[0309] The server is a computer with high-performance computing power, and has Blender or Unity installed as a 3D modeling engine. The server also receives prompts from users and provides an environment for running programs to generate 3D avatars. Specifically, the following software is used:

[0310] 3D modeling engine: Blender, Unity

[0311] The device is connected to an interface for users to input prompt sentences, and hardware (camera and microphone) for capturing camera images and audio. The device also has software (OpenCV, Dlib, Google® Cloud Speech-to-Text API) for analyzing camera images and audio data in real time. The device also has an emotion engine (Affectiva SDK) installed that performs emotion recognition.

[0312] Example of operation

[0313] 1. Prompt input: The user inputs "a wizard-like character." This prompt is sent from the terminal to the server.

[0314] 2. Avatar generation: The server receives the prompt and generates a wizard-like 3D avatar using Blender or Unity. The generated 3D avatar is sent to the user's device.

[0315] 3. Video and audio analysis: When a user starts a remote meeting through a camera and microphone, the device captures and analyzes camera video and audio data in real time.

[0316] 4. Emotion Recognition: The emotion engine (Affectiva SDK) recognizes the user's emotions based on video and audio data. For example, if the user is smiling, the system will recognize that the user is "happy."

[0317] 5. Avatar reflection: Based on the analysis results and emotion recognition results, the device moves each part of a 3D avatar created using Blender or Unity, reflecting emotional expressions in real time.

[0318] 6. Broadcasting: Users can set the generated 3D avatar in remote conferencing or broadcasting software such as Zoom or YouTube Live, and control the avatar in real time through a virtual camera. Viewers and participants can communicate with the user through the avatar.

[0319] This invention provides a 3D avatar that reflects the user's movements and emotions in real time, significantly improving the user experience in remote meetings and live streaming.

[0320] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0321] Step 1:

[0322] The user inputs a prompt. The input is in the form of text, such as a request for a "magic-looking character." This input prompt is then sent to the server. Here, the user uses the system's interface to enter the prompt text into a text field.

[0323] Input: User prompt (e.g., "A character that looks like a wizard")

[0324] Output: Data containing a prompt (e.g., {"prompt": "A wizard-like character"})

[0325] Step 2:

[0326] The terminal sends the user's prompt to the server via a protocol such as an HTTP request, which passes the prompt to the server, where the terminal receives it and prepares the data in the appropriate format for transmission to the server.

[0327] Input: Data containing a prompt (e.g., {"prompt": "A wizard-like character"})

[0328] Output: Status of completion of sending to server

[0329] Step 3:

[0330] The server receives the prompt and generates a 3D avatar using a 3D modeling engine such as Blender or Unity. The server analyzes the prompt and generates an avatar based on the corresponding 3D model template.

[0331] Input: Prompt text (e.g., {"prompt": "A wizard-like character"})

[0332] Output: Generated 3D avatar data (e.g. 3D modeling data file)

[0333] Step 4:

[0334] The server sends the generated 3D avatar to the user's device. The generated 3D avatar data is appropriately compressed or formatted and transferred to the user's device.

[0335] Input: Generated 3D avatar data (e.g., 3D modeling data file)

[0336] Output: Transmission completed status to user terminal

[0337] Step 5:

[0338] The user starts broadcasting using the camera and microphone. The user starts the remote conferencing software or live broadcasting software and enables the camera and microphone. This prepares the video and audio to be captured in real time.

[0339] Input: Connecting and enabling cameras and microphones

[0340] Output: Real-time video and audio stream data

[0341] Step 6:

[0342] The device captures and analyzes camera images and audio data in real time, detects facial features and posture using OpenCV and Dlib, and converts the audio data into text using the Google Cloud Speech-to-Text API.

[0343] Input: Real-time video and audio stream data

[0344] Output: Analysis results (e.g., facial feature points, body posture, voice text data)

[0345] Step 7:

[0346] The data acquired by the device is passed to the emotion recognition engine to recognize the user's emotions. The Affectiva SDK is used to detect the user's emotional state from video and audio data.

[0347] Input: Analysis results (e.g., facial feature points, body posture, voice text data)

[0348] Output: Emotion recognition results (e.g., joy, sadness, anger)

[0349] Step 8:

[0350] The device updates the 3D avatar based on the analysis and emotion recognition results. Using Blender or Unity, the 3D avatar moves in real time and changes its facial expression according to the user's movements and emotions.

[0351] Input: Analysis results and emotion recognition results

[0352] Output: 3D avatar data updated in real time

[0353] Step 9:

[0354] The device transmits the 3D avatar's movements to remote conferencing or live streaming software, and the updated 3D avatar video is sent as a live stream using software such as OBS Virtual Camera.

[0355] Input: 3D avatar data updated in real time

[0356] Output: Stream data to remote conference and live streaming software

[0357] (Application example 2)

[0358] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0359] Conventional 3D avatar generation systems are limited to reflecting a user's facial expressions and movements, making it difficult to recognize a user's emotions in real time and reflect them in the 3D avatar. Furthermore, there is a lack of a way to express a user's movements and emotions more naturally during remote meetings and live streaming. For these reasons, there is a demand for improved user experience and more efficient communication.

[0360] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0361] In this invention, the server includes means for generating a 3D model based on prompts received from the user, means for capturing camera footage of the user in real time and analyzing the footage to detect the user's facial expressions and movements, means for recognizing the user's emotional state based on the analyzed voice data, and means for reflecting the recognized emotions in the 3D model. This allows the user's movements and emotions to be reflected in the 3D avatar in real time, enabling more natural and interactive communication in remote meetings and live streaming.

[0362] A "prompt" is a text instruction that a user enters to generate a particular 3D model.

[0363] A "3D model" is an object in three-dimensional space generated using computer graphics.

[0364] "User's facial expression" refers to the emotional expression shown by the user's facial muscles and posture.

[0365] "User movement" refers to the movement or change in pose of a part or the whole of the user's body.

[0366] "Voice data" is information that is generated by detecting a user's speech and digitizing it.

[0367] "Emotional state" refers to the psychological state or emotion that a user expresses through their voice or facial expression.

[0368] A "remote conference" is a conference held over the Internet with multiple participants in remote locations.

[0369] "Live streaming" refers to the transmission of video and audio to viewers in real time via the Internet.

[0370] A "server" is a computer system that provides various services and data over a network.

[0371] "Camera footage" is video footage data captured by a camera.

[0372] "Analysis" is the process of breaking down acquired data into elements and examining them in detail.

[0373] "Recognition" is the process by which a system understands specific information from inputs such as a user's facial expressions or voice.

[0374] "Reflection" refers to reflecting the acquired information in the movements and expressions of the 3D model.

[0375] The system embodying this invention generates a 3D model based on prompts entered by the user and includes a set of means for reflecting the user's facial expressions and movements in the model. The system also recognizes the user's emotional state and reflects it in the 3D model, enabling more natural and interactive communication in remote meetings and live broadcasts.

[0376] The system is implemented using the following hardware and software: The hardware includes a user's device (e.g., smartphone, tablet) and a server. The software includes a 3D model generation engine (e.g., Blender + Python script), a speech recognition API (e.g., Google Cloud Speech-to-Text API), and an emotion recognition API (e.g., Microsoft® Azure® Emotion API).

[0377] Program processing explanation

[0378] 1. Receiving prompts and generating 3D models

[0379] A user types a prompt into an application on their device, such as "a cheerful store clerk in colorful clothing."

[0380] The server analyzes the received prompt and generates the specified 3D model using a 3D model generation engine, which then stores the model on the server and sends it to the user's device.

[0381] 2. Video and audio analysis

[0382] Users can use their device's camera and microphone to start remote meetings and live broadcasts.

[0383] The device captures camera footage and audio data in real time and uses a speech recognition API to analyze the audio data, for example, when a user says, "What are the features of this smartphone?"

[0384] 3. Recognizing emotional states

[0385] The device sends the captured video to an emotion recognition API, which recognizes the user's emotions in real time. For example, if a user is smiling while talking, the emotion recognition API will recognize the user's state as "happy."

[0386] 4. Reflecting emotions and actions

[0387] The server updates the 3D model based on the analyzed voice and emotion data. Specifically, if the voice analysis results identify a description of the "smartphone's features" and the emotion recognition results indicate "happiness," the 3D model will also explain with a smile.

[0388] 5. Real-time video transmission

[0389] The device transmits real-time video of the generated 3D model to remote conferencing or live streaming applications, where it can be configured as a virtual camera and used in remote conferencing or live streaming software (e.g., Zoom or YouTube Live).

[0390] Specific examples

[0391] A user types in "cheerful sales clerks in colorful clothing."

[0392] The server receives the prompt, generates the corresponding 3D model, and sends it to the user's device.

[0393] When a user asks a question using the camera and microphone, the device analyzes the video and audio, and the emotion recognition API recognizes the user's emotion of joy.

[0394] The server updates the 3D model based on the analysis and recognition results and answers the user's questions with a smile.

[0395] A user connects to a remote conferencing app, and the generated 3D model acts and expresses emotions on behalf of the user during the meeting.

[0396] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0397] Step 1:

[0398] The user enters a prompt into an application on the device, such as "a cheerful store clerk in colorful clothing." This prompt is sent from the user's device to the server as an HTTP POST request.

[0399] Step 2:

[0400] The server analyzes the received prompt sentence and generates the specified 3D model using a 3D model generation engine (e.g. Blender + Python script). The generated 3D model is saved on the server and the generated data is sent to the user's device. As an output, 3D model data is generated and sent to the user's device.

[0401] Step 3:

[0402] A user can use an application installed on the device to start a remote meeting or live broadcast using the camera and microphone, and the device acquires the real-time camera video and audio data captured at that time. As output, the camera video and audio data are obtained.

[0403] Step 4:

[0404] The device analyzes the captured camera video and audio data. When analyzing the camera video, a computer vision algorithm (e.g., OpenCV) is used to detect facial features and body posture. When analyzing the audio data, a speech recognition API (e.g., Google Cloud Speech-to-Text API) is used. This allows the user's facial expressions, movements, and speech content to be analyzed.

[0405] Step 5:

[0406] The device uses an emotion recognition API (e.g., Microsoft Azure Emotion API) to recognize the user's emotional state based on the analyzed voice data and camera image data. For example, if the user is smiling while speaking, it will be recognized as "happy." Emotional information is obtained as output.

[0407] Step 6:

[0408] The server updates the 3D model based on the analyzed facial expression, motion, emotion, and voice data. Specifically, it changes the 3D model's facial expressions and movements in real time to reflect the user's actions and emotions. The updated 3D model data is obtained as the output.

[0409] Step 7:

[0410] The user's device transmits real-time video of the generated 3D model to a remote meeting or live streaming platform (e.g., Zoom, YouTube Live). This video is set up as a virtual camera and connected to the remote meeting or streaming software. The output is a video of the 3D model that moves and expresses emotions in real time during the remote meeting or live streaming.

[0411] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0412] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search engine).<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0413] In the above embodiment, an example in which the specific process is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0414] [Second embodiment]

[0415] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0416] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0417] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0418] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0419] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0420] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0421] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0422] Fig. 4 shows an example of the main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0423] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0424] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0425] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0426] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the smart glasses 214 will be referred to as the "terminal."

[0427] System Overview

[0428] This invention is a system that allows users to input prompts, generate original 3D avatars based on those prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar, allowing the avatar to move in accordance with the user's movements.

[0429] Program processing explanation (Example)

[0430] Avatar generation

[0431] The user enters a prompt, which the device sends to the system. For example, if the user enters a prompt such as "a blue character with deer antlers," the device sends this information to the server. The server receives the prompt and uses a 3D modeling engine to generate a corresponding 3D avatar. The generated avatar is temporarily stored on the server and then sent as data to the user's device. The device receives this data and displays the generated avatar to the user.

[0432] Camera footage analysis

[0433] When a user starts a remote meeting or live broadcast using a camera connected to their device, the video is captured in real time. The device analyzes the camera video and detects facial features and body movements. For example, when a user smiles or moves their hands, computer vision technology is applied to recognize these movements in real time.

[0434] Avatar Action Link

[0435] The device then moves each part of the 3D avatar it generates based on the analysis results. For example, if the user smiles, the 3D avatar's face will also move to smile. Similarly, if the user moves their hand, the corresponding movement is transmitted to the avatar. In this way, the user's movements are directly linked to the 3D avatar.

[0436] Real-time video streaming

[0437] Users connect their 3D avatar to remote conferencing or live streaming software. Specifically, they can set the 3D avatar as a virtual camera in video conferencing software such as Zoom or streaming software such as YouTube Live. The device transmits video of the 3D avatar moving in real time through this virtual camera. When the user speaks or moves, the avatar also moves in the same way, allowing viewers and participants to communicate with the user through the avatar.

[0438] Specific examples

[0439] 1. Prompt Input: The user types "a wizard-looking character."

[0440] 2. Avatar generation: The server receives the prompt, generates a wizard-like 3D avatar, and sends it to the user's device.

[0441] 3. Camera analysis: The user starts the meeting through the camera, and the device analyzes the user's facial expressions and movements.

[0442] 4. Avatar Reflection: Based on the analysis results, the 3D avatar moves in accordance with the user's movements. For example, if the user raises their hand, the avatar will raise their hand as well.

[0443] 5. Broadcasting: The user sets up the generated 3D avatar in Zoom, and the avatar acts on the user's behalf during the meeting.

[0444] Thus, the present invention provides a system that allows users to easily generate 3D avatars and use them in remote meetings and live streaming.

[0445] The processing flow will be explained below.

[0446] Program processing flow

[0447] Avatar generation process

[0448] Step 1:

[0449] The user types a prompt into the system, for example, "a blue character with deer antlers."

[0450] Step 2:

[0451] The terminal sends the user's prompt data to the server.

[0452] Step 3:

[0453] The server receives the prompt and parses and understands its contents.

[0454] Step 4:

[0455] The server's 3D modeling engine generates a 3D avatar based on the prompts.

[0456] Step 5:

[0457] The server stores the generated 3D avatar data and sends it to the user's device.

[0458] Step 6:

[0459] The terminal displays the received 3D avatar data and allows the user to confirm it.

[0460] Camera image analysis processing

[0461] Step 1:

[0462] The user connects the camera to their computer and launches the remote conferencing software or live streaming software.

[0463] Step 2:

[0464] The device acquires video data from the camera in real time.

[0465] Step 3:

[0466] The device analyzes the captured camera footage.

[0467] Step 4:

[0468] The device detects the user's facial features and body posture, for example, identifying the user's eye and mouth movements and hand positions.

[0469] Avatar movement link processing

[0470] Step 1:

[0471] The device moves each part of the 3D avatar based on the analysis results.

[0472] Step 2:

[0473] For example, if the user smiles, the 3D avatar's face will also smile. Similarly, if the user moves their hand, the avatar's hand will also move in a similar way.

[0474] Step 3:

[0475] The device repeats this real-time processing, constantly updating the 3D avatar in accordance with the user's movements.

[0476] Real-time video distribution processing

[0477] Step 1:

[0478] The user connects the generated 3D avatar to remote conferencing software or live streaming software, for example, by setting it as a virtual camera on Zoom or YouTube Live.

[0479] Step 2:

[0480] The device continues to maintain a link to the 3D avatar's movements.

[0481] Step 3:

[0482] The device transmits images of a 3D avatar moving in real time via a virtual camera to remote conferencing software or live streaming software.

[0483] Step 4:

[0484] Users communicate through 3D avatars during remote meetings and live broadcasts. For example, when a user speaks, the 3D avatar's mouth moves in the same way.

[0485] In this way, the system reflects the user's movements through each step in real time onto a 3D avatar, allowing the footage to be used for remote meetings and live streaming.

[0486] Example 1

[0487] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0488] In modern remote meetings and live streaming, it is difficult for users to easily create and use original 3D avatars that reflect their own facial expressions and movements in real time. Another problem is that existing technologies lack the ability to accurately detect user movements through analysis of camera footage and then coordinate avatar movements in real time. By resolving these issues, users can enjoy more natural and intuitive remote communication.

[0489] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0490] In this invention, the server includes means for generating a 3D avatar based on a prompt received from the user, means for the server to analyze the prompt using a generation AI model and generate a corresponding 3D avatar, means for acquiring a camera image of the user in real time and analyzing the image to detect the user's facial expressions and movements, means for reflecting the detected facial expressions and movements in the 3D avatar so that the avatar moves in response to the user's movements, and means for transmitting a video of the moving 3D avatar to a remote conference or live streaming. This allows users to easily generate a 3D avatar, and the avatar moves in real time in coordination with the user's movements, enabling more effective remote communication.

[0491] A "prompt" is a sentence or phrase that a user enters to indicate the features or elements required to generate a 3D avatar.

[0492] A "generative AI model" is a type of artificial intelligence technology that analyzes prompts received from users and generates corresponding 3D avatars.

[0493] A "3D avatar" is a three-dimensional virtual character generated based on a user's characteristics and preferences.

[0494] The "server" is a computing system that receives prompts, generates a 3D avatar using a generative AI model, and transmits the data to the user's device.

[0495] "Camera video" refers to video data acquired in real time from a camera device used by a user.

[0496] "Real-time analysis" is a technology that instantly processes camera footage and detects the user's facial expressions and movements.

[0497] "Means for detecting facial expressions and movements" refers to computer vision technology for recognizing the user's facial features and body movements.

[0498] A "virtual camera" is a technology that allows software-generated images to be treated as if they were real camera images.

[0499] A "remote conference" is a conference in which multiple users participate from remote locations via the Internet.

[0500] "Live streaming" refers to the distribution of video and audio to viewers in real time via the Internet.

[0501] The present invention provides a system for generating a 3D avatar based on prompts input by a user, and for the avatar to move in real time in response to the user's actions. The system includes the following means.

[0502] 1. Enter and submit the prompt

[0503] It starts with the user entering a prompt into the terminal interface, for example, "a red dragon with wings." Once the prompt is entered, the terminal sends the information to the system.

[0504] 2. Prompt Processing and Avatar Creation

[0505] The device sends the prompt to a server, which uses a generative AI model (such as DALLE-2 or DeepArt) to analyze the prompt and generate a corresponding 3D avatar. For example, if a user enters the prompt "a red dragon with wings," the server can generate the avatar using a 3D modeling tool like Blender.

[0506] 3. Save and send your avatar

[0507] The server temporarily stores the generated 3D avatar and sends the data to the device. The device receives the data and displays it as a preview to the user. An initial preview image of the generated avatar is displayed on the device screen.

[0508] 4. Real-time analysis of camera footage

[0509] A user starts a remote conference or live broadcast using a camera connected to their device. The device uses an application (such as OpenCV or MediaPipe) to analyze the camera image in real time and detect facial features and body movements. When the user smiles, the change in facial expression is detected in real time.

[0510] 5. Avatar Action Link

[0511] The device updates the behavior of the 3D avatar in real time based on the analysis results. For example, if the user raises their hand, the device detects that movement and mirrors the avatar's hand by raising it. Similarly, facial expressions are updated simultaneously; if the user smiles, the avatar also smiles.

[0512] 6. Real-time video distribution

[0513] The user sets the generated 3D avatar as a virtual camera in remote conferencing software such as Zoom or YouTube Live. The device then uses virtual camera software such as OBS Virtual Camera to transmit the avatar's image, which is updated in real time. When the user speaks or moves, their movements are reflected in the avatar and displayed to viewers and participants in real time. For example, if a user waves their hand during a meeting, the avatar will also wave.

[0514] In this way, a 3D avatar generated from user-entered prompts can be linked to the user's movements in real time and used in remote meetings and live broadcasts.

[0515] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0516] Step 1:

[0517] The user enters a prompt (e.g., "a red dragon with wings") into the terminal interface. The entered prompt is sent to the system by the terminal. This prompt becomes input data to the system. The terminal sends the prompt to the server in JSON format, for example.

[0518] Step 2:

[0519] The server analyzes the received prompt text. First, it inputs the input prompt text into a generative AI model (e.g., DALLE-2 or DeepArt). The generative AI model analyzes the prompt text and generates design data for the corresponding 3D avatar. This data includes information such as the avatar's shape, color, and movement patterns. The server generates the 3D avatar using 3D modeling software (e.g., Blender) based on the design data. The generated 3D avatar is temporarily stored on the server.

[0520] Step 3:

[0521] The server sends the temporarily saved 3D avatar data to the device. The device analyzes the received 3D avatar design data and displays a preview to the user. This display includes the overall image and main features of the avatar. The user can check the avatar and request changes or adjustments as necessary.

[0522] Step 4:

[0523] To start a remote meeting or live broadcast, a user begins capturing video using a camera. The device acquires video from the camera in real time and analyzes it using computer vision technologies such as OpenCV and MediaPipe. This analysis detects facial features and body movements. Analyzed data includes the user's facial expressions, eye movements, and hand movements.

[0524] Step 5:

[0525] The device then uses the analysis results to adjust each part of the 3D avatar in real time. For example, if the user raises their hand, the device detects that movement and adjusts the 3D avatar's hand so that it also raises. Similarly, if the user smiles, the avatar's face also changes to a smile. The device uses this data to update the 3D avatar's movements in real time.

[0526] Step 6:

[0527] Users set up a 3D avatar as a virtual camera in remote conferencing or live streaming software (e.g., Zoom or YouTube Live). The device then uses virtual camera software such as OBS Virtual Camera to transmit a live image of the avatar, which is updated in real time. When the user speaks or moves, their actions are reflected in the avatar in real time, allowing for natural communication with viewers and participants.

[0528] (Application example 1)

[0529] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0530] There is a demand for virtual stores to provide users with a realistic shopping experience, guided in real time. However, existing systems lack avatar guidance functions that synchronize with the user's movements, and users have difficulty in easily customizing and using 3D avatars. In addition, to provide more attractive virtual store guidance, it is desirable for the avatar to synchronize with the user's movements and facial expressions in real time.

[0531] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0532] In this invention, the server includes: means for generating a 3D avatar based on prompts received from the user; means for capturing camera footage of the user in real time and analyzing the footage to detect the user's facial expressions and movements; means for reflecting the detected facial expressions and movements in the 3D avatar so that the avatar moves in response to the user's movements; means for displaying the moving 3D avatar on the user's device; and means for the user to move with the avatar in a virtual space and for the avatar to provide guidance as the user moves. This allows the user to easily create a customized 3D avatar based on their own movements and prompts, enabling a shopping experience in which they are guided through a virtual store in real time.

[0533] A "prompt" is a word or phrase entered by a user to indicate an instruction or request.

[0534] A "3D avatar" is a three-dimensional virtual character that is generated based on the user's prompts and camera footage, and moves in response to the user's movements and facial expressions.

[0535] "User's camera footage" refers to footage captured in real time by the camera of the device used by the user.

[0536] "Analysis of camera footage" is the process of detecting the user's facial expressions and movements from camera footage acquired in real time and evaluating their features and posture.

[0537] "Real-time" refers to a state in which operations and actions are reflected instantly with almost no delay.

[0538] "Moving 3D avatar video" refers to a 3D avatar that moves in real time in response to the user's facial expressions and movements, and is displayed as an image or video.

[0539] "Virtual space" refers to an imaginary environment or space generated using computer graphics or virtual reality technology.

[0540] A "virtual store" is a virtual shopping environment created using the Internet and virtual reality technology that has the same functions as a physical store.

[0541] "Means for providing guidance" refers to the function of the avatar to provide information to the user and guide the user to their destination when the user moves within the virtual space.

[0542] A "device" is an electronic device used by a user, such as a smartphone, tablet, or computer.

[0543] System implementation

[0544] This system generates a 3D avatar based on prompts received from the user, analyzes the user's camera footage in real time to detect the user's facial expressions and movements, and reflects those movements in the 3D avatar. Specifically, the following hardware and software are required:

[0545] Hardware: devices such as smartphones or tablets, cameras

[0546] Software: 3D modeling engine (Unity, Unreal Engine, etc.), facial recognition software (OpenCV, Dlib, etc.)

[0547] Program processing explanation

[0548] 1. Prompt input and avatar generation

[0549] A user uses a smartphone app to input a prompt such as "Nordic-style guide character." For example, the user types "Nordic-style guide."

[0550] The device sends this prompt to the server, which uses the generative AI model to generate a corresponding 3D avatar, which is then sent from the server to the device and displayed to the user.

[0551] 2. Analysis of camera footage

[0552] As users begin to move around the virtual store through the camera, the device captures camera footage and uses facial recognition software such as OpenCV or Dlib to detect facial features and body poses in real time.

[0553] 3. Movement and Avatar Movement Links

[0554] The detected facial expressions and movements are then reflected in a 3D avatar in real time based on the user's movements: for example, if the user raises their hand, the avatar will do the same.

[0555] 4. Real-time store information

[0556] The device provides a mechanism for a moving 3D avatar to be displayed on the user's device. As the user moves through the virtual space, the avatar walks ahead of the user, providing real-time guidance and explaining the features of various products and stores.

[0557] Specific examples

[0558] 1. Example of prompt input:

[0559] For example, a user enters "Scandinavian style guide."

[0560] 2. Avatar generation example:

[0561] The server receives the prompt, generates a Nordic-style guide character using a 3D modeling engine, and sends it to the user's device.

[0562] 3. Camera analysis example:

[0563] As users begin to move around the virtual store using their device's camera, the device analyzes the camera footage and detects facial features and body posture in real time.

[0564] 4. Avatar reflection example:

[0565] Based on the detection results, the 3D avatar moves in sync with the user's movements. For example, when the user smiles, the avatar also smiles.

[0566] 5. Store location example:

[0567] As the user moves around the virtual space with the device, the avatar explains product information and makes comments based on the product the user is looking at.

[0568] This system allows users to easily generate 3D avatars, further enhancing the shopping experience in virtual stores.

[0569] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0570] Step 1:

[0571] Prompt input and submit

[0572] The user starts the smartphone app and inputs a prompt for the 3D avatar they want to create, such as a "Nordic-style guide character." The input prompt is sent to the server by the device.

[0573] Input: User prompt (e.g. "Nordic style guide")

[0574] Output: Send prompt to server

[0575] Specific operation: The user enters a prompt using text input and presses the send button. The device sends this prompt to the server as an HTTP request.

[0576] Step 2:

[0577] 3D avatar generation

[0578] The server analyzes the received prompt using a generative AI model to generate a 3D avatar with the specified characteristics. The generated 3D avatar is temporarily stored on the server and then sent to the device as data.

[0579] Input: prompt (e.g. "Nordic style guide")

[0580] Output: Generated 3D avatar

[0581] How it works: The server uses a generative AI model to analyze the prompt and generates a 3D avatar using Unity or Unreal Engine, which is then stored on the server and sent to the device as an HTTP response.

[0582] Step 3:

[0583] Displaying 3D avatars

[0584] The device displays the 3D avatar received from the server.

[0585] Input: 3D avatar data sent from the server

[0586] Output: 3D avatar displayed on device

[0587] Specific operation: The device deserializes the received 3D avatar data and displays it to the user using a Unity viewer or similar.

[0588] Step 4:

[0589] Camera footage capture and analysis

[0590] As the user moves around the virtual store, the device captures and analyzes camera footage in real time, using OpenCV and Dlib to detect facial features and body poses.

[0591] Input: User's camera video

[0592] Output: Analysis results of user's facial expressions and movements

[0593] How it works: The device's camera captures video in real time, and then uses libraries such as OpenCV and Dlib to analyze facial features and body movements. The analysis results are output as data showing the user's current facial expressions and movements.

[0594] Step 5:

[0595] Avatar movement reflection

[0596] The device then uses the analysis results to update the 3D avatar's movements in real time, adjusting the avatar to perform similar movements based on the user's movements.

[0597] Input: Analyzed user facial expression and movement data

[0598] Output: A moving 3D avatar

[0599] Specific operation: Based on the analysis results, each joint and facial expression of the 3D avatar is updated in real time using a game engine such as Unity.

[0600] Step 6:

[0601] Guidance within the virtual store

[0602] As users move through the virtual space, a 3D avatar on the device acts as a real-time guide.

[0603] Input: User movement data in virtual space

[0604] Output: Guidance information using 3D avatars

[0605] Specific operation: The device uses the user's location and movement data to provide a 3D avatar with product information and directions. For example, when the user stands in front of a nearby product, the avatar will explain the product's features.

[0606] Step 7:

[0607] Display on user devices

[0608] The terminal displays a moving 3D avatar and real-time information on the user's device, and the user can view and interact with the avatar's guidance through their own device.

[0609] Input: A moving 3D avatar and related information

[0610] Output: What is displayed on the user's device

[0611] Specific operation: The terminal renders the display data and displays it in real time on the user's device, allowing the user to experience the virtual store through interaction with a 3D avatar on the device.

[0612] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0613] System Overview

[0614] This invention is a system that allows users to input prompts, generate original 3D avatars based on those prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar. It also has the ability to recognize the user's emotions using an emotion engine and reflect those emotions in the 3D avatar.

[0615] Program processing explanation (Example)

[0616] Avatar generation

[0617] A user enters a prompt into the system, and the device sends the prompt to the server. For example, if a user enters a prompt such as "a blue character with deer antlers," the device sends this information to the server. The server receives the prompt and uses a 3D modeling engine to generate a corresponding 3D avatar. This generated avatar is temporarily stored on the server and then sent as data to the user's device. The device receives this data and displays the generated avatar to the user.

[0618] Camera video and audio analysis

[0619] When a user starts a remote meeting or live broadcast using a camera and microphone connected to their device, video and audio are captured in real time. The device acquires and analyzes the camera video and audio data. Specifically, the device detects facial features and body posture, and simultaneously analyzes the audio data to recognize the user's emotional state. For example, when a user smiles or moves their hands, computer vision and voice recognition technologies are applied to recognize these movements in real time.

[0620] Emotion recognition by emotion engine

[0621] The device passes the captured video and audio data to the emotion engine, which recognizes the user's emotions in real time. The emotion engine detects positive (happiness, satisfaction) and negative (anger, sadness) emotions from facial and audio data. For example, when a user speaks with a smile, the emotion engine recognizes that the user is "happy" from their voice and facial expression.

[0622] Linking avatar movements and emotions

[0623] Based on the analysis results and the recognition results of the emotion engine, the device links each part of the generated 3D avatar with its emotional expression. For example, if the user is smiling and happy, the 3D avatar's face will also smile and show a happy expression. Similarly, if the user is angry, the avatar's face will show an angry expression. In this way, the user's movements and emotions are reflected directly in the 3D avatar in real time.

[0624] Real-time video streaming

[0625] The user connects the generated 3D avatar to remote conferencing software or live streaming software. Specifically, the 3D avatar can be set as a virtual camera in video conferencing software such as Zoom or streaming software such as YouTube Live. The device sends video of the 3D avatar, which moves and expresses emotions in real time, to the remote conferencing software or streaming software via this virtual camera. When the user speaks or moves, the avatar also expresses the same movements and emotions, allowing viewers and participants to communicate with the user through the avatar.

[0626] Specific examples

[0627] 1. Prompt Input: The user types "a wizard-looking character."

[0628] 2. Avatar generation: The server receives the prompt, generates a wizard-like 3D avatar, and sends it to the user's device.

[0629] 3. Video and audio analysis: The user initiates a remote meeting via a camera and microphone, and the device analyzes the user's facial expressions and voice.

[0630] 4. Emotion recognition: The emotion engine recognizes emotions from the user's video and audio, detecting, for example, a "happy" state.

[0631] 5. Avatar Reflection: Based on the analysis results and emotion recognition results, a 3D avatar expresses the user's movements and emotions in real time.

[0632] 6. Distribution: The user sets the generated 3D avatar in remote conferencing software, and the avatar acts on behalf of the user during the meeting and expresses emotions.

[0633] Thus, the present invention provides a system that generates a 3D avatar that reflects a user's movements and emotions in real time and can be used in remote meetings and live streaming.

[0634] The processing flow will be explained below.

[0635] Program processing flow

[0636] Avatar generation process

[0637] Step 1:

[0638] The user types a prompt into the system, for example, "a blue character with deer antlers."

[0639] Step 2:

[0640] The terminal sends the user's prompt data to the server.

[0641] Step 3:

[0642] The server receives the prompt and parses and understands its contents.

[0643] Step 4:

[0644] The server's 3D modeling engine generates a 3D avatar based on the prompts.

[0645] Step 5:

[0646] The server stores the generated 3D avatar data and sends it to the user's device.

[0647] Step 6:

[0648] The terminal displays the received 3D avatar data and allows the user to confirm it.

[0649] Camera video and audio analysis processing

[0650] Step 1:

[0651] The user connects a camera and microphone to their computer and launches remote conferencing software or live streaming software.

[0652] Step 2:

[0653] The device acquires video data in real time from the camera and audio data from the microphone.

[0654] Step 3:

[0655] The device analyzes the captured camera footage.

[0656] Step 4:

[0657] The device detects the user's facial features and body posture, for example, identifying the user's eye and mouth movements and hand positions.

[0658] Step 5:

[0659] The device analyzes the acquired voice data and recognizes the user's emotional state from the voice.

[0660] Emotion recognition processing using an emotion engine

[0661] Step 1:

[0662] The device passes the acquired video and audio data to the emotion engine.

[0663] Step 2:

[0664] The emotion engine analyzes the user's facial expressions from video data and recognizes emotions by analyzing the tone and speaking style from audio data.

[0665] Step 3:

[0666] For example, if a user speaks with a smile, the emotion engine will recognize from this data that the user is "happy."

[0667] Avatar movement and emotion link processing

[0668] Step 1:

[0669] The device moves each part of the 3D avatar based on the analysis results and the recognition results of the emotion engine.

[0670] Step 2:

[0671] For example, if the user is smiling and "happy," the 3D avatar's face will also smile and move to show a happy expression. Similarly, if the user moves their hand, the avatar's hand movement will reflect that movement.

[0672] Step 3:

[0673] The device repeats this real-time processing, constantly updating the 3D avatar to match the user's movements and emotions.

[0674] Real-time video distribution processing

[0675] Step 1:

[0676] The user connects the generated 3D avatar to remote conferencing software or live streaming software, for example, by setting it as a virtual camera on Zoom or YouTube Live.

[0677] Step 2:

[0678] The device maintains a link between the movements and emotions of the 3D avatar.

[0679] Step 3:

[0680] The device transmits images of a 3D avatar that moves and expresses emotions in real time via a virtual camera to remote conferencing software or live streaming software.

[0681] Step 4:

[0682] Users communicate through 3D avatars during remote meetings and live broadcasts. For example, when a user speaks, the 3D avatar moves its mouth in the same way and changes its facial expression depending on the recognized emotion.

[0683] In this way, the system reflects the user's movements and emotions through each step in real time onto a 3D avatar, allowing the footage to be used for remote meetings and live streaming.

[0684] Example 2

[0685] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0686] When users use 3D avatars in remote meetings or live streaming, conventional systems often fail to accurately reflect the user's movements and facial expressions. Furthermore, the 3D avatars generated based on prompts may not match the user's preferences, making customization difficult. This reduces the quality of the user experience and makes efficient communication difficult.

[0687] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for generating a 3D avatar based on a prompt received from the user, a means for acquiring the user's camera video and audio in real time and analyzing the video and audio to detect the user's facial expressions and movements, a means for recognizing the user's emotions using an emotion recognition engine, a means for reflecting the detected facial expressions, movements, and recognized emotions in the 3D avatar so that the avatar moves in accordance with the user's movements and movements, and a means for transmitting the video and audio of the moving 3D avatar to a remote conference or live streaming. This makes it possible to provide a customizable 3D avatar based on the prompt, reflecting the user's movements and movements in real time.

[0688] A "prompt" is text information that indicates a request or instruction that a user inputs to a system.

[0689] A "3D avatar" is a character or icon that can be displayed in three-dimensional space and operates in real time by reflecting the user's movements and emotions.

[0690] "Camera footage" refers to real-time video data captured by a camera connected to a user's terminal.

[0691] "Audio Data" refers to real-time audio data captured by a microphone connected to a user's terminal.

[0692] "Analyzing" refers to the process of analyzing the acquired data and extracting information such as feature points, posture, and emotions.

[0693] An "emotion recognition engine" refers to software or algorithms for detecting a user's emotional state from facial expression data and voice data.

[0694] A "remote conference" is a conference conducted in real time by multiple participants using video and audio over the Internet.

[0695] "Live streaming" refers to a form of broadcasting in which video and audio are transmitted in real time to a large number of viewers via the Internet.

[0696] "Customizable" refers to the ability to change the appearance and behavior of the generated 3D avatar based on user specifications.

[0697] "Detecting facial expressions and movements" refers to the process of recognizing facial features and body posture from camera footage and extracting that information.

[0698] This invention is a system that allows users to input prompts, generate original 3D avatars based on the prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar. It also has the function of recognizing the user's emotions using an emotion engine and reflecting them in the 3D avatar.

[0699] Hardware and Software Configuration

[0700] The server is a computer with high-performance computing power, and has Blender or Unity installed as a 3D modeling engine. The server also receives prompts from users and provides an environment for running programs to generate 3D avatars. Specifically, the following software is used:

[0701] 3D modeling engine: Blender, Unity

[0702] The device is connected to an interface for users to input prompts, and hardware (camera and microphone) for capturing camera images and audio. The device also has software (OpenCV, Dlib, Google Cloud Speech-to-Text API) for analyzing camera images and audio data in real time. The device also has an emotion engine (Affectiva SDK) installed for emotion recognition.

[0703] Example of operation

[0704] 1. Prompt input: The user inputs "a wizard-like character." This prompt is sent from the terminal to the server.

[0705] 2. Avatar generation: The server receives the prompt and generates a wizard-like 3D avatar using Blender or Unity. The generated 3D avatar is sent to the user's device.

[0706] 3. Video and audio analysis: When a user starts a remote meeting through a camera and microphone, the device captures and analyzes camera video and audio data in real time.

[0707] 4. Emotion Recognition: The emotion engine (Affectiva SDK) recognizes the user's emotions based on video and audio data. For example, if the user is smiling, the system will recognize that the user is "happy."

[0708] 5. Avatar reflection: Based on the analysis results and emotion recognition results, the device moves each part of a 3D avatar created using Blender or Unity, reflecting emotional expressions in real time.

[0709] 6. Broadcasting: Users can set the generated 3D avatar in remote conferencing or broadcasting software such as Zoom or YouTube Live, and control the avatar in real time through a virtual camera. Viewers and participants can communicate with the user through the avatar.

[0710] This invention provides a 3D avatar that reflects the user's movements and emotions in real time, significantly improving the user experience in remote meetings and live streaming.

[0711] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0712] Step 1:

[0713] The user inputs a prompt. The input is in the form of text, such as a request for a "magic-looking character." This input prompt is then sent to the server. Here, the user uses the system's interface to enter the prompt text into a text field.

[0714] Input: User prompt (e.g., "A character that looks like a wizard")

[0715] Output: Data containing a prompt (e.g., {"prompt": "A wizard-like character"})

[0716] Step 2:

[0717] The terminal sends the user's prompt to the server via a protocol such as an HTTP request, which passes the prompt to the server, where the terminal receives it and prepares the data in the appropriate format for transmission to the server.

[0718] Input: Data containing a prompt (e.g., {"prompt": "A wizard-like character"})

[0719] Output: Status of completion of sending to server

[0720] Step 3:

[0721] The server receives the prompt and generates a 3D avatar using a 3D modeling engine such as Blender or Unity. The server analyzes the prompt and generates an avatar based on the corresponding 3D model template.

[0722] Input: Prompt text (e.g., {"prompt": "A wizard-like character"})

[0723] Output: Generated 3D avatar data (e.g. 3D modeling data file)

[0724] Step 4:

[0725] The server sends the generated 3D avatar to the user's device. The generated 3D avatar data is appropriately compressed or formatted and transferred to the user's device.

[0726] Input: Generated 3D avatar data (e.g., 3D modeling data file)

[0727] Output: Transmission completed status to user terminal

[0728] Step 5:

[0729] The user starts broadcasting using the camera and microphone. The user starts the remote conferencing software or live broadcasting software and enables the camera and microphone. This prepares the video and audio to be captured in real time.

[0730] Input: Connecting and enabling cameras and microphones

[0731] Output: Real-time video and audio stream data

[0732] Step 6:

[0733] The device captures and analyzes camera images and audio data in real time, detects facial features and posture using OpenCV and Dlib, and converts the audio data into text using the Google Cloud Speech-to-Text API.

[0734] Input: Real-time video and audio stream data

[0735] Output: Analysis results (e.g., facial feature points, body posture, voice text data)

[0736] Step 7:

[0737] The data acquired by the device is passed to the emotion recognition engine to recognize the user's emotions. The Affectiva SDK is used to detect the user's emotional state from video and audio data.

[0738] Input: Analysis results (e.g., facial feature points, body posture, voice text data)

[0739] Output: Emotion recognition results (e.g., joy, sadness, anger)

[0740] Step 8:

[0741] The device updates the 3D avatar based on the analysis and emotion recognition results. Using Blender or Unity, the 3D avatar moves in real time and changes its facial expression according to the user's movements and emotions.

[0742] Input: Analysis results and emotion recognition results

[0743] Output: 3D avatar data updated in real time

[0744] Step 9:

[0745] The device transmits the 3D avatar's movements to remote conferencing or live streaming software, and the updated 3D avatar video is sent as a live stream using software such as OBS Virtual Camera.

[0746] Input: 3D avatar data updated in real time

[0747] Output: Stream data to remote conference and live streaming software

[0748] (Application example 2)

[0749] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal."

[0750] Conventional 3D avatar generation systems are limited to reflecting a user's facial expressions and movements, making it difficult to recognize a user's emotions in real time and reflect them in the 3D avatar. Furthermore, there is a lack of a way to express a user's movements and emotions more naturally during remote meetings and live streaming. For these reasons, there is a demand for improved user experience and more efficient communication.

[0751] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[0752] In this invention, the server includes means for generating a 3D model based on prompts received from the user, means for capturing camera footage of the user in real time and analyzing the footage to detect the user's facial expressions and movements, means for recognizing the user's emotional state based on the analyzed voice data, and means for reflecting the recognized emotions in the 3D model. This allows the user's movements and emotions to be reflected in the 3D avatar in real time, enabling more natural and interactive communication in remote meetings and live streaming.

[0753] A "prompt" is a text instruction that a user enters to generate a particular 3D model.

[0754] A "3D model" is an object in three-dimensional space generated using computer graphics.

[0755] "User's facial expression" refers to the emotional expression shown by the user's facial muscles and posture.

[0756] "User movement" refers to the movement or change in pose of a part or the whole of the user's body.

[0757] "Voice data" is information that is generated by detecting a user's speech and digitizing it.

[0758] "Emotional state" refers to the psychological state or emotion that a user expresses through their voice or facial expression.

[0759] A "remote conference" is a conference held over the Internet with multiple participants in remote locations.

[0760] "Live streaming" refers to the transmission of video and audio to viewers in real time via the Internet.

[0761] A "server" is a computer system that provides various services and data over a network.

[0762] "Camera footage" is video footage data captured by a camera.

[0763] "Analysis" is the process of breaking down acquired data into elements and examining them in detail.

[0764] "Recognition" is the process by which a system understands specific information from inputs such as a user's facial expressions or voice.

[0765] "Reflection" refers to reflecting the acquired information in the movements and expressions of the 3D model.

[0766] The system embodying this invention generates a 3D model based on prompts entered by the user and includes a set of means for reflecting the user's facial expressions and movements in the model. The system also recognizes the user's emotional state and reflects it in the 3D model, enabling more natural and interactive communication in remote meetings and live broadcasts.

[0767] The system is implemented using the following hardware and software: The hardware includes the user's device (e.g., smartphone, tablet) and the server. The software includes a 3D model generation engine (e.g., Blender + Python script), a speech recognition API (e.g., Google Cloud Speech-to-Text API), and an emotion recognition API (e.g., Microsoft Azure Emotion API).

[0768] Program processing explanation

[0769] 1. Receiving prompts and generating 3D models

[0770] A user types a prompt into an application on their device, such as "a cheerful store clerk in colorful clothing."

[0771] The server analyzes the received prompt and generates the specified 3D model using a 3D model generation engine, which then stores the model on the server and sends it to the user's device.

[0772] 2. Video and audio analysis

[0773] Users can use their device's camera and microphone to start remote meetings and live broadcasts.

[0774] The device captures camera footage and audio data in real time and uses a speech recognition API to analyze the audio data, for example, when a user says, "What are the features of this smartphone?"

[0775] 3. Recognizing emotional states

[0776] The device sends the captured video to an emotion recognition API, which recognizes the user's emotions in real time. For example, if a user is smiling while talking, the emotion recognition API will recognize the user's state as "happy."

[0777] 4. Reflecting emotions and actions

[0778] The server updates the 3D model based on the analyzed voice and emotion data. Specifically, if the voice analysis results identify a description of the "smartphone's features" and the emotion recognition results indicate "happiness," the 3D model will also explain with a smile.

[0779] 5. Real-time video transmission

[0780] The device transmits real-time video of the generated 3D model to remote conferencing or live streaming applications, where it can be configured as a virtual camera and used in remote conferencing or live streaming software (e.g., Zoom or YouTube Live).

[0781] Specific examples

[0782] A user types in "cheerful sales clerks in colorful clothing."

[0783] The server receives the prompt, generates the corresponding 3D model, and sends it to the user's device.

[0784] When a user asks a question using the camera and microphone, the device analyzes the video and audio, and the emotion recognition API recognizes the user's emotion of joy.

[0785] The server updates the 3D model based on the analysis and recognition results and answers the user's questions with a smile.

[0786] A user connects to a remote conferencing app, and the generated 3D model acts and expresses emotions on behalf of the user during the meeting.

[0787] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[0788] Step 1:

[0789] The user enters a prompt into an application on the device, such as "a cheerful store clerk in colorful clothing." This prompt is sent from the user's device to the server as an HTTP POST request.

[0790] Step 2:

[0791] The server analyzes the received prompt sentence and generates the specified 3D model using a 3D model generation engine (e.g. Blender + Python script). The generated 3D model is saved on the server and the generated data is sent to the user's device. As an output, 3D model data is generated and sent to the user's device.

[0792] Step 3:

[0793] A user can use an application installed on the device to start a remote meeting or live broadcast using the camera and microphone, and the device acquires the real-time camera video and audio data captured at that time. As output, the camera video and audio data are obtained.

[0794] Step 4:

[0795] The device analyzes the captured camera video and audio data. When analyzing the camera video, a computer vision algorithm (e.g., OpenCV) is used to detect facial features and body posture. When analyzing the audio data, a speech recognition API (e.g., Google Cloud Speech-to-Text API) is used. This allows the user's facial expressions, movements, and speech content to be analyzed.

[0796] Step 5:

[0797] The device uses an emotion recognition API (e.g., Microsoft Azure Emotion API) to recognize the user's emotional state based on the analyzed voice data and camera image data. For example, if the user is smiling while speaking, it will be recognized as "happy." Emotional information is obtained as output.

[0798] Step 6:

[0799] The server updates the 3D model based on the analyzed facial expression, motion, emotion, and voice data. Specifically, it changes the 3D model's facial expressions and movements in real time to reflect the user's actions and emotions. The updated 3D model data is obtained as the output.

[0800] Step 7:

[0801] The user's device transmits real-time video of the generated 3D model to a remote meeting or live streaming platform (e.g., Zoom, YouTube Live). This video is set up as a virtual camera and connected to the remote meeting or streaming software. The output is a video of the 3D model that moves and expresses emotions in real time during the remote meeting or live streaming.

[0802] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[0803] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0804] In the above embodiment, an example in which the specific processing is performed by the data processing device 12 has been given, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0805] [Third embodiment]

[0806] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0807] 5, the data processing system 310 includes the data processing device 12 and a headset terminal 314. An example of the data processing device 12 is a server.

[0808] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[0809] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0810] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[0811] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[0812] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[0813] Fig. 6 shows an example of the main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0814] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0815] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0816] In the headset type terminal 314, a reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0817] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server" and the headset type terminal 314 will be referred to as the "terminal."

[0818] System Overview

[0819] This invention is a system that allows users to input prompts, generate original 3D avatars based on those prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar, allowing the avatar to move in accordance with the user's movements.

[0820] Program processing explanation (Example)

[0821] Avatar generation

[0822] The user enters a prompt, which the device sends to the system. For example, if the user enters a prompt such as "a blue character with deer antlers," the device sends this information to the server. The server receives the prompt and uses a 3D modeling engine to generate a corresponding 3D avatar. The generated avatar is temporarily stored on the server and then sent as data to the user's device. The device receives this data and displays the generated avatar to the user.

[0823] Camera footage analysis

[0824] When a user starts a remote meeting or live broadcast using a camera connected to their device, the video is captured in real time. The device analyzes the camera video and detects facial features and body movements. For example, when a user smiles or moves their hands, computer vision technology is applied to recognize these movements in real time.

[0825] Avatar Action Link

[0826] The device then moves each part of the 3D avatar it generates based on the analysis results. For example, if the user smiles, the 3D avatar's face will also move to smile. Similarly, if the user moves their hand, the corresponding movement is transmitted to the avatar. In this way, the user's movements are directly linked to the 3D avatar.

[0827] Real-time video streaming

[0828] Users connect their 3D avatar to remote conferencing or live streaming software. Specifically, they can set the 3D avatar as a virtual camera in video conferencing software such as Zoom or streaming software such as YouTube Live. The device transmits video of the 3D avatar moving in real time through this virtual camera. When the user speaks or moves, the avatar also moves in the same way, allowing viewers and participants to communicate with the user through the avatar.

[0829] Specific examples

[0830] 1. Prompt Input: The user types "a wizard-looking character."

[0831] 2. Avatar generation: The server receives the prompt, generates a wizard-like 3D avatar, and sends it to the user's device.

[0832] 3. Camera analysis: The user starts the meeting through the camera, and the device analyzes the user's facial expressions and movements.

[0833] 4. Avatar Reflection: Based on the analysis results, the 3D avatar moves in accordance with the user's movements. For example, if the user raises their hand, the avatar will raise their hand as well.

[0834] 5. Broadcasting: The user sets up the generated 3D avatar in Zoom, and the avatar acts on the user's behalf during the meeting.

[0835] Thus, the present invention provides a system that allows users to easily generate 3D avatars and use them in remote meetings and live streaming.

[0836] The processing flow will be explained below.

[0837] Program processing flow

[0838] Avatar generation process

[0839] Step 1:

[0840] The user types a prompt into the system, for example, "a blue character with deer antlers."

[0841] Step 2:

[0842] The terminal sends the user's prompt data to the server.

[0843] Step 3:

[0844] The server receives the prompt and parses and understands its contents.

[0845] Step 4:

[0846] The server's 3D modeling engine generates a 3D avatar based on the prompts.

[0847] Step 5:

[0848] The server stores the generated 3D avatar data and sends it to the user's device.

[0849] Step 6:

[0850] The terminal displays the received 3D avatar data and allows the user to confirm it.

[0851] Camera image analysis processing

[0852] Step 1:

[0853] The user connects the camera to their computer and launches the remote conferencing software or live streaming software.

[0854] Step 2:

[0855] The device acquires video data from the camera in real time.

[0856] Step 3:

[0857] The device analyzes the captured camera footage.

[0858] Step 4:

[0859] The device detects the user's facial features and body posture, for example, identifying the user's eye and mouth movements and hand positions.

[0860] Avatar movement link processing

[0861] Step 1:

[0862] The device moves each part of the 3D avatar based on the analysis results.

[0863] Step 2:

[0864] For example, if the user smiles, the 3D avatar's face will also smile. Similarly, if the user moves their hand, the avatar's hand will also move in a similar way.

[0865] Step 3:

[0866] The device repeats this real-time processing, constantly updating the 3D avatar in accordance with the user's movements.

[0867] Real-time video distribution processing

[0868] Step 1:

[0869] The user connects the generated 3D avatar to remote conferencing software or live streaming software, for example, by setting it as a virtual camera on Zoom or YouTube Live.

[0870] Step 2:

[0871] The device continues to maintain a link to the 3D avatar's movements.

[0872] Step 3:

[0873] The device transmits images of a 3D avatar moving in real time via a virtual camera to remote conferencing software or live streaming software.

[0874] Step 4:

[0875] Users communicate through 3D avatars during remote meetings and live broadcasts. For example, when a user speaks, the 3D avatar's mouth moves in the same way.

[0876] In this way, the system reflects the user's movements through each step in real time onto a 3D avatar, allowing the footage to be used for remote meetings and live streaming.

[0877] Example 1

[0878] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0879] In modern remote meetings and live streaming, it is difficult for users to easily create and use original 3D avatars that reflect their own facial expressions and movements in real time. Another problem is that existing technologies lack the ability to accurately detect user movements through analysis of camera footage and then coordinate avatar movements in real time. By resolving these issues, users can enjoy more natural and intuitive remote communication.

[0880] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0881] In this invention, the server includes means for generating a 3D avatar based on a prompt received from the user, means for the server to analyze the prompt using a generation AI model and generate a corresponding 3D avatar, means for acquiring a camera image of the user in real time and analyzing the image to detect the user's facial expressions and movements, means for reflecting the detected facial expressions and movements in the 3D avatar so that the avatar moves in response to the user's movements, and means for transmitting a video of the moving 3D avatar to a remote conference or live streaming. This allows users to easily generate a 3D avatar, and the avatar moves in real time in coordination with the user's movements, enabling more effective remote communication.

[0882] A "prompt" is a sentence or phrase that a user enters to indicate the features or elements required to generate a 3D avatar.

[0883] A "generative AI model" is a type of artificial intelligence technology that analyzes prompts received from users and generates corresponding 3D avatars.

[0884] A "3D avatar" is a three-dimensional virtual character generated based on a user's characteristics and preferences.

[0885] The "server" is a computing system that receives prompts, generates a 3D avatar using a generative AI model, and transmits the data to the user's device.

[0886] "Camera video" refers to video data acquired in real time from a camera device used by a user.

[0887] "Real-time analysis" is a technology that instantly processes camera footage and detects the user's facial expressions and movements.

[0888] "Means for detecting facial expressions and movements" refers to computer vision technology for recognizing the user's facial features and body movements.

[0889] A "virtual camera" is a technology that allows software-generated images to be treated as if they were real camera images.

[0890] A "remote conference" is a conference in which multiple users participate from remote locations via the Internet.

[0891] "Live streaming" refers to the distribution of video and audio to viewers in real time via the Internet.

[0892] The present invention provides a system for generating a 3D avatar based on prompts input by a user, and for the avatar to move in real time in response to the user's actions. The system includes the following means.

[0893] 1. Enter and submit the prompt

[0894] It starts with the user entering a prompt into the terminal interface, for example, "a red dragon with wings." Once the prompt is entered, the terminal sends the information to the system.

[0895] 2. Prompt Processing and Avatar Creation

[0896] The device sends the prompt to a server, which uses a generative AI model (such as DALLE-2 or DeepArt) to analyze the prompt and generate a corresponding 3D avatar. For example, if a user enters the prompt "a red dragon with wings," the server can generate the avatar using a 3D modeling tool like Blender.

[0897] 3. Save and send your avatar

[0898] The server temporarily stores the generated 3D avatar and sends the data to the device. The device receives the data and displays it as a preview to the user. An initial preview image of the generated avatar is displayed on the device screen.

[0899] 4. Real-time analysis of camera footage

[0900] A user starts a remote conference or live broadcast using a camera connected to their device. The device uses an application (such as OpenCV or MediaPipe) to analyze the camera image in real time and detect facial features and body movements. When the user smiles, the change in facial expression is detected in real time.

[0901] 5. Avatar Action Link

[0902] The device updates the behavior of the 3D avatar in real time based on the analysis results. For example, if the user raises their hand, the device detects that movement and mirrors the avatar's hand by raising it. Similarly, facial expressions are updated simultaneously; if the user smiles, the avatar also smiles.

[0903] 6. Real-time video distribution

[0904] The user sets the generated 3D avatar as a virtual camera in remote conferencing software such as Zoom or YouTube Live. The device then uses virtual camera software such as OBS Virtual Camera to transmit the avatar's image, which is updated in real time. When the user speaks or moves, their movements are reflected in the avatar and displayed to viewers and participants in real time. For example, if a user waves their hand during a meeting, the avatar will also wave.

[0905] In this way, a 3D avatar generated from user-entered prompts can be linked to the user's movements in real time and used in remote meetings and live broadcasts.

[0906] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0907] Step 1:

[0908] The user enters a prompt (e.g., "a red dragon with wings") into the terminal interface. The entered prompt is sent to the system by the terminal. This prompt becomes input data to the system. The terminal sends the prompt to the server in JSON format, for example.

[0909] Step 2:

[0910] The server analyzes the received prompt text. First, it inputs the input prompt text into a generative AI model (e.g., DALLE-2 or DeepArt). The generative AI model analyzes the prompt text and generates design data for the corresponding 3D avatar. This data includes information such as the avatar's shape, color, and movement patterns. The server generates the 3D avatar using 3D modeling software (e.g., Blender) based on the design data. The generated 3D avatar is temporarily stored on the server.

[0911] Step 3:

[0912] The server sends the temporarily saved 3D avatar data to the device. The device analyzes the received 3D avatar design data and displays a preview to the user. This display includes the overall image and main features of the avatar. The user can check the avatar and request changes or adjustments as necessary.

[0913] Step 4:

[0914] To start a remote meeting or live broadcast, a user begins capturing video using a camera. The device acquires video from the camera in real time and analyzes it using computer vision technologies such as OpenCV and MediaPipe. This analysis detects facial features and body movements. Analyzed data includes the user's facial expressions, eye movements, and hand movements.

[0915] Step 5:

[0916] The device then uses the analysis results to adjust each part of the 3D avatar in real time. For example, if the user raises their hand, the device detects that movement and adjusts the 3D avatar's hand so that it also raises. Similarly, if the user smiles, the avatar's face also changes to a smile. The device uses this data to update the 3D avatar's movements in real time.

[0917] Step 6:

[0918] Users set up a 3D avatar as a virtual camera in remote conferencing or live streaming software (e.g., Zoom or YouTube Live). The device then uses virtual camera software such as OBS Virtual Camera to transmit a live image of the avatar, which is updated in real time. When the user speaks or moves, their actions are reflected in the avatar in real time, allowing for natural communication with viewers and participants.

[0919] (Application example 1)

[0920] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0921] There is a demand for virtual stores to provide users with a realistic shopping experience, guided in real time. However, existing systems lack avatar guidance functions that synchronize with the user's movements, and users have difficulty in easily customizing and using 3D avatars. In addition, to provide more attractive virtual store guidance, it is desirable for the avatar to synchronize with the user's movements and facial expressions in real time.

[0922] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0923] In this invention, the server includes: means for generating a 3D avatar based on prompts received from the user; means for capturing camera footage of the user in real time and analyzing the footage to detect the user's facial expressions and movements; means for reflecting the detected facial expressions and movements in the 3D avatar so that the avatar moves in response to the user's movements; means for displaying the moving 3D avatar on the user's device; and means for the user to move with the avatar in a virtual space and for the avatar to provide guidance as the user moves. This allows the user to easily create a customized 3D avatar based on their own movements and prompts, enabling a shopping experience in which they are guided through a virtual store in real time.

[0924] A "prompt" is a word or phrase entered by a user to indicate an instruction or request.

[0925] A "3D avatar" is a three-dimensional virtual character that is generated based on the user's prompts and camera footage, and moves in response to the user's movements and facial expressions.

[0926] "User's camera footage" refers to footage captured in real time by the camera of the device used by the user.

[0927] "Analysis of camera footage" is the process of detecting the user's facial expressions and movements from camera footage acquired in real time and evaluating their features and posture.

[0928] "Real-time" refers to a state in which operations and actions are reflected instantly with almost no delay.

[0929] "Moving 3D avatar video" refers to a 3D avatar that moves in real time in response to the user's facial expressions and movements, and is displayed as an image or video.

[0930] "Virtual space" refers to an imaginary environment or space generated using computer graphics or virtual reality technology.

[0931] A "virtual store" is a virtual shopping environment created using the Internet and virtual reality technology that has the same functions as a physical store.

[0932] "Means for providing guidance" refers to the function of the avatar to provide information to the user and guide the user to their destination when the user moves within the virtual space.

[0933] A "device" is an electronic device used by a user, such as a smartphone, tablet, or computer.

[0934] System implementation

[0935] This system generates a 3D avatar based on prompts received from the user, analyzes the user's camera footage in real time to detect the user's facial expressions and movements, and reflects those movements in the 3D avatar. Specifically, the following hardware and software are required:

[0936] Hardware: devices such as smartphones or tablets, cameras

[0937] Software: 3D modeling engine (Unity, Unreal Engine, etc.), facial recognition software (OpenCV, Dlib, etc.)

[0938] Program processing explanation

[0939] 1. Prompt input and avatar generation

[0940] A user uses a smartphone app to input a prompt such as "Nordic-style guide character." For example, the user types "Nordic-style guide."

[0941] The device sends this prompt to the server, which uses the generative AI model to generate a corresponding 3D avatar, which is then sent from the server to the device and displayed to the user.

[0942] 2. Analysis of camera footage

[0943] As users begin to move around the virtual store through the camera, the device captures camera footage and uses facial recognition software such as OpenCV or Dlib to detect facial features and body poses in real time.

[0944] 3. Movement and Avatar Movement Links

[0945] The detected facial expressions and movements are then reflected in a 3D avatar in real time based on the user's movements: for example, if the user raises their hand, the avatar will do the same.

[0946] 4. Real-time store information

[0947] The device provides a mechanism for a moving 3D avatar to be displayed on the user's device. As the user moves through the virtual space, the avatar walks ahead of the user, providing real-time guidance and explaining the features of various products and stores.

[0948] Specific examples

[0949] 1. Example of prompt input:

[0950] For example, a user enters "Scandinavian style guide."

[0951] 2. Avatar generation example:

[0952] The server receives the prompt, generates a Nordic-style guide character using a 3D modeling engine, and sends it to the user's device.

[0953] 3. Camera analysis example:

[0954] As users begin to move around the virtual store using their device's camera, the device analyzes the camera footage and detects facial features and body posture in real time.

[0955] 4. Avatar reflection example:

[0956] Based on the detection results, the 3D avatar moves in sync with the user's movements. For example, when the user smiles, the avatar also smiles.

[0957] 5. Store location example:

[0958] As the user moves around the virtual space with the device, the avatar explains product information and makes comments based on the product the user is looking at.

[0959] This system allows users to easily generate 3D avatars, further enhancing the shopping experience in virtual stores.

[0960] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[0961] Step 1:

[0962] Prompt input and submit

[0963] The user starts the smartphone app and inputs a prompt for the 3D avatar they want to create, such as a "Nordic-style guide character." The input prompt is sent to the server by the device.

[0964] Input: User prompt (e.g. "Nordic style guide")

[0965] Output: Send prompt to server

[0966] Specific operation: The user enters a prompt using text input and presses the send button. The device sends this prompt to the server as an HTTP request.

[0967] Step 2:

[0968] 3D avatar generation

[0969] The server analyzes the received prompt using a generative AI model to generate a 3D avatar with the specified characteristics. The generated 3D avatar is temporarily stored on the server and then sent to the device as data.

[0970] Input: prompt (e.g. "Nordic style guide")

[0971] Output: Generated 3D avatar

[0972] How it works: The server uses a generative AI model to analyze the prompt and generates a 3D avatar using Unity or Unreal Engine, which is then stored on the server and sent to the device as an HTTP response.

[0973] Step 3:

[0974] Displaying 3D avatars

[0975] The device displays the 3D avatar received from the server.

[0976] Input: 3D avatar data sent from the server

[0977] Output: 3D avatar displayed on device

[0978] Specific operation: The device deserializes the received 3D avatar data and displays it to the user using a Unity viewer or similar.

[0979] Step 4:

[0980] Camera footage capture and analysis

[0981] As the user moves around the virtual store, the device captures and analyzes camera footage in real time, using OpenCV and Dlib to detect facial features and body poses.

[0982] Input: User's camera video

[0983] Output: Analysis results of user's facial expressions and movements

[0984] How it works: The device's camera captures video in real time, and then uses libraries such as OpenCV and Dlib to analyze facial features and body movements. The analysis results are output as data showing the user's current facial expressions and movements.

[0985] Step 5:

[0986] Avatar movement reflection

[0987] The device then uses the analysis results to update the 3D avatar's movements in real time, adjusting the avatar to perform similar movements based on the user's movements.

[0988] Input: Analyzed user facial expression and movement data

[0989] Output: A moving 3D avatar

[0990] Specific operation: Based on the analysis results, each joint and facial expression of the 3D avatar is updated in real time using a game engine such as Unity.

[0991] Step 6:

[0992] Guidance within the virtual store

[0993] As users move through the virtual space, a 3D avatar on the device acts as a real-time guide.

[0994] Input: User movement data in virtual space

[0995] Output: Guidance information using 3D avatars

[0996] Specific operation: The device uses the user's location and movement data to provide a 3D avatar with product information and directions. For example, when the user stands in front of a nearby product, the avatar will explain the product's features.

[0997] Step 7:

[0998] Display on user devices

[0999] The terminal displays a moving 3D avatar and real-time information on the user's device, and the user can view and interact with the avatar's guidance through their own device.

[1000] Input: A moving 3D avatar and related information

[1001] Output: What is displayed on the user's device

[1002] Specific operation: The terminal renders the display data and displays it in real time on the user's device, allowing the user to experience the virtual store through interaction with a 3D avatar on the device.

[1003] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1004] System Overview

[1005] This invention is a system that allows users to input prompts, generate original 3D avatars based on those prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar. It also has the ability to recognize the user's emotions using an emotion engine and reflect those emotions in the 3D avatar.

[1006] Program processing explanation (Example)

[1007] Avatar generation

[1008] A user enters a prompt into the system, and the device sends the prompt to the server. For example, if a user enters a prompt such as "a blue character with deer antlers," the device sends this information to the server. The server receives the prompt and uses a 3D modeling engine to generate a corresponding 3D avatar. This generated avatar is temporarily stored on the server and then sent as data to the user's device. The device receives this data and displays the generated avatar to the user.

[1009] Camera video and audio analysis

[1010] When a user starts a remote meeting or live broadcast using a camera and microphone connected to their device, video and audio are captured in real time. The device acquires and analyzes the camera video and audio data. Specifically, the device detects facial features and body posture, and simultaneously analyzes the audio data to recognize the user's emotional state. For example, when a user smiles or moves their hands, computer vision and voice recognition technologies are applied to recognize these movements in real time.

[1011] Emotion recognition by emotion engine

[1012] The device passes the captured video and audio data to the emotion engine, which recognizes the user's emotions in real time. The emotion engine detects positive (happiness, satisfaction) and negative (anger, sadness) emotions from facial and audio data. For example, when a user speaks with a smile, the emotion engine recognizes that the user is "happy" from their voice and facial expression.

[1013] Linking avatar movements and emotions

[1014] Based on the analysis results and the recognition results of the emotion engine, the device links each part of the generated 3D avatar with its emotional expression. For example, if the user is smiling and happy, the 3D avatar's face will also smile and show a happy expression. Similarly, if the user is angry, the avatar's face will show an angry expression. In this way, the user's movements and emotions are reflected directly in the 3D avatar in real time.

[1015] Real-time video streaming

[1016] The user connects the generated 3D avatar to remote conferencing software or live streaming software. Specifically, the 3D avatar can be set as a virtual camera in video conferencing software such as Zoom or streaming software such as YouTube Live. The device sends video of the 3D avatar, which moves and expresses emotions in real time, to the remote conferencing software or streaming software via this virtual camera. When the user speaks or moves, the avatar also expresses the same movements and emotions, allowing viewers and participants to communicate with the user through the avatar.

[1017] Specific examples

[1018] 1. Prompt Input: The user types "a wizard-looking character."

[1019] 2. Avatar generation: The server receives the prompt, generates a wizard-like 3D avatar, and sends it to the user's device.

[1020] 3. Video and audio analysis: The user initiates a remote meeting via a camera and microphone, and the device analyzes the user's facial expressions and voice.

[1021] 4. Emotion recognition: The emotion engine recognizes emotions from the user's video and audio, detecting, for example, a "happy" state.

[1022] 5. Avatar Reflection: Based on the analysis results and emotion recognition results, a 3D avatar expresses the user's movements and emotions in real time.

[1023] 6. Distribution: The user sets the generated 3D avatar in remote conferencing software, and the avatar acts on behalf of the user during the meeting and expresses emotions.

[1024] Thus, the present invention provides a system that generates a 3D avatar that reflects a user's movements and emotions in real time and can be used in remote meetings and live streaming.

[1025] The processing flow will be explained below.

[1026] Program processing flow

[1027] Avatar generation process

[1028] Step 1:

[1029] The user types a prompt into the system, for example, "a blue character with deer antlers."

[1030] Step 2:

[1031] The terminal sends the user's prompt data to the server.

[1032] Step 3:

[1033] The server receives the prompt and parses and understands its contents.

[1034] Step 4:

[1035] The server's 3D modeling engine generates a 3D avatar based on the prompts.

[1036] Step 5:

[1037] The server stores the generated 3D avatar data and sends it to the user's device.

[1038] Step 6:

[1039] The terminal displays the received 3D avatar data and allows the user to confirm it.

[1040] Camera video and audio analysis processing

[1041] Step 1:

[1042] The user connects a camera and microphone to their computer and launches remote conferencing software or live streaming software.

[1043] Step 2:

[1044] The device acquires video data in real time from the camera and audio data from the microphone.

[1045] Step 3:

[1046] The device analyzes the captured camera footage.

[1047] Step 4:

[1048] The device detects the user's facial features and body posture, for example, identifying the user's eye and mouth movements and hand positions.

[1049] Step 5:

[1050] The device analyzes the acquired voice data and recognizes the user's emotional state from the voice.

[1051] Emotion recognition processing using an emotion engine

[1052] Step 1:

[1053] The device passes the acquired video and audio data to the emotion engine.

[1054] Step 2:

[1055] The emotion engine analyzes the user's facial expressions from video data and recognizes emotions by analyzing the tone and speaking style from audio data.

[1056] Step 3:

[1057] For example, if a user speaks with a smile, the emotion engine will recognize from this data that the user is "happy."

[1058] Avatar movement and emotion link processing

[1059] Step 1:

[1060] The device moves each part of the 3D avatar based on the analysis results and the recognition results of the emotion engine.

[1061] Step 2:

[1062] For example, if the user is smiling and "happy," the 3D avatar's face will also smile and move to show a happy expression. Similarly, if the user moves their hand, the avatar's hand movement will reflect that movement.

[1063] Step 3:

[1064] The device repeats this real-time processing, constantly updating the 3D avatar to match the user's movements and emotions.

[1065] Real-time video distribution processing

[1066] Step 1:

[1067] The user connects the generated 3D avatar to remote conferencing software or live streaming software, for example, by setting it as a virtual camera on Zoom or YouTube Live.

[1068] Step 2:

[1069] The device maintains a link between the movements and emotions of the 3D avatar.

[1070] Step 3:

[1071] The device transmits images of a 3D avatar that moves and expresses emotions in real time via a virtual camera to remote conferencing software or live streaming software.

[1072] Step 4:

[1073] Users communicate through 3D avatars during remote meetings and live broadcasts. For example, when a user speaks, the 3D avatar moves its mouth in the same way and changes its facial expression depending on the recognized emotion.

[1074] In this way, the system reflects the user's movements and emotions through each step in real time onto a 3D avatar, allowing the footage to be used for remote meetings and live streaming.

[1075] Example 2

[1076] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1077] When users use 3D avatars in remote meetings or live streaming, conventional systems often fail to accurately reflect the user's movements and facial expressions. Furthermore, the 3D avatars generated based on prompts may not match the user's preferences, making customization difficult. This reduces the quality of the user experience and makes efficient communication difficult.

[1078] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for generating a 3D avatar based on a prompt received from the user, a means for acquiring the user's camera video and audio in real time and analyzing the video and audio to detect the user's facial expressions and movements, a means for recognizing the user's emotions using an emotion recognition engine, a means for reflecting the detected facial expressions, movements, and recognized emotions in the 3D avatar so that the avatar moves in accordance with the user's movements and movements, and a means for transmitting the video and audio of the moving 3D avatar to a remote conference or live streaming. This makes it possible to provide a customizable 3D avatar based on the prompt, reflecting the user's movements and movements in real time.

[1079] A "prompt" is text information that indicates a request or instruction that a user inputs to a system.

[1080] A "3D avatar" is a character or icon that can be displayed in three-dimensional space and operates in real time by reflecting the user's movements and emotions.

[1081] "Camera footage" refers to real-time video data captured by a camera connected to a user's terminal.

[1082] "Audio Data" refers to real-time audio data captured by a microphone connected to a user's terminal.

[1083] "Analyzing" refers to the process of analyzing the acquired data and extracting information such as feature points, posture, and emotions.

[1084] An "emotion recognition engine" refers to software or algorithms for detecting a user's emotional state from facial expression data and voice data.

[1085] A "remote conference" is a conference conducted in real time by multiple participants using video and audio over the Internet.

[1086] "Live streaming" refers to a form of broadcasting in which video and audio are transmitted in real time to a large number of viewers via the Internet.

[1087] "Customizable" refers to the ability to change the appearance and behavior of the generated 3D avatar based on user specifications.

[1088] "Detecting facial expressions and movements" refers to the process of recognizing facial features and body posture from camera footage and extracting that information.

[1089] This invention is a system that allows users to input prompts, generate original 3D avatars based on the prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar. It also has the function of recognizing the user's emotions using an emotion engine and reflecting them in the 3D avatar.

[1090] Hardware and Software Configuration

[1091] The server is a computer with high-performance computing power, and has Blender or Unity installed as a 3D modeling engine. The server also receives prompts from users and provides an environment for running programs to generate 3D avatars. Specifically, the following software is used:

[1092] 3D modeling engine: Blender, Unity

[1093] The device is connected to an interface for users to input prompts, and hardware (camera and microphone) for capturing camera images and audio. The device also has software (OpenCV, Dlib, Google Cloud Speech-to-Text API) for analyzing camera images and audio data in real time. The device also has an emotion engine (Affectiva SDK) installed for emotion recognition.

[1094] Example of operation

[1095] 1. Prompt input: The user inputs "a wizard-like character." This prompt is sent from the terminal to the server.

[1096] 2. Avatar generation: The server receives the prompt and generates a wizard-like 3D avatar using Blender or Unity. The generated 3D avatar is sent to the user's device.

[1097] 3. Video and audio analysis: When a user starts a remote meeting through a camera and microphone, the device captures and analyzes camera video and audio data in real time.

[1098] 4. Emotion Recognition: The emotion engine (Affectiva SDK) recognizes the user's emotions based on video and audio data. For example, if the user is smiling, the system will recognize that the user is "happy."

[1099] 5. Avatar reflection: Based on the analysis results and emotion recognition results, the device moves each part of a 3D avatar created using Blender or Unity, reflecting emotional expressions in real time.

[1100] 6. Broadcasting: Users can set the generated 3D avatar in remote conferencing or broadcasting software such as Zoom or YouTube Live, and control the avatar in real time through a virtual camera. Viewers and participants can communicate with the user through the avatar.

[1101] This invention provides a 3D avatar that reflects the user's movements and emotions in real time, significantly improving the user experience in remote meetings and live streaming.

[1102] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1103] Step 1:

[1104] The user inputs a prompt. The input is in the form of text, such as a request for a "magic-looking character." This input prompt is then sent to the server. Here, the user uses the system's interface to enter the prompt text into a text field.

[1105] Input: User prompt (e.g., "A character that looks like a wizard")

[1106] Output: Data containing a prompt (e.g., {"prompt": "A wizard-like character"})

[1107] Step 2:

[1108] The terminal sends the user's prompt to the server via a protocol such as an HTTP request, which passes the prompt to the server, where the terminal receives it and prepares the data in the appropriate format for transmission to the server.

[1109] Input: Data containing a prompt (e.g., {"prompt": "A wizard-like character"})

[1110] Output: Status of completion of sending to server

[1111] Step 3:

[1112] The server receives the prompt and generates a 3D avatar using a 3D modeling engine such as Blender or Unity. The server analyzes the prompt and generates an avatar based on the corresponding 3D model template.

[1113] Input: Prompt text (e.g., {"prompt": "A wizard-like character"})

[1114] Output: Generated 3D avatar data (e.g. 3D modeling data file)

[1115] Step 4:

[1116] The server sends the generated 3D avatar to the user's device. The generated 3D avatar data is appropriately compressed or formatted and transferred to the user's device.

[1117] Input: Generated 3D avatar data (e.g., 3D modeling data file)

[1118] Output: Transmission completed status to user terminal

[1119] Step 5:

[1120] The user starts broadcasting using the camera and microphone. The user starts the remote conferencing software or live broadcasting software and enables the camera and microphone. This prepares the video and audio to be captured in real time.

[1121] Input: Connecting and enabling cameras and microphones

[1122] Output: Real-time video and audio stream data

[1123] Step 6:

[1124] The device captures and analyzes camera images and audio data in real time, detects facial features and posture using OpenCV and Dlib, and converts the audio data into text using the Google Cloud Speech-to-Text API.

[1125] Input: Real-time video and audio stream data

[1126] Output: Analysis results (e.g., facial feature points, body posture, voice text data)

[1127] Step 7:

[1128] The data acquired by the device is passed to the emotion recognition engine to recognize the user's emotions. The Affectiva SDK is used to detect the user's emotional state from video and audio data.

[1129] Input: Analysis results (e.g., facial feature points, body posture, voice text data)

[1130] Output: Emotion recognition results (e.g., joy, sadness, anger)

[1131] Step 8:

[1132] The device updates the 3D avatar based on the analysis and emotion recognition results. Using Blender or Unity, the 3D avatar moves in real time and changes its facial expression according to the user's movements and emotions.

[1133] Input: Analysis results and emotion recognition results

[1134] Output: 3D avatar data updated in real time

[1135] Step 9:

[1136] The device transmits the 3D avatar's movements to remote conferencing or live streaming software, and the updated 3D avatar video is sent as a live stream using software such as OBS Virtual Camera.

[1137] Input: 3D avatar data updated in real time

[1138] Output: Stream data to remote conference and live streaming software

[1139] (Application example 2)

[1140] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[1141] Conventional 3D avatar generation systems are limited to reflecting a user's facial expressions and movements, making it difficult to recognize a user's emotions in real time and reflect them in the 3D avatar. Furthermore, there is a lack of a way to express a user's movements and emotions more naturally during remote meetings and live streaming. For these reasons, there is a demand for improved user experience and more efficient communication.

[1142] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1143] In this invention, the server includes means for generating a 3D model based on prompts received from the user, means for capturing camera footage of the user in real time and analyzing the footage to detect the user's facial expressions and movements, means for recognizing the user's emotional state based on the analyzed voice data, and means for reflecting the recognized emotions in the 3D model. This allows the user's movements and emotions to be reflected in the 3D avatar in real time, enabling more natural and interactive communication in remote meetings and live streaming.

[1144] A "prompt" is a text instruction that a user enters to generate a particular 3D model.

[1145] A "3D model" is an object in three-dimensional space generated using computer graphics.

[1146] "User's facial expression" refers to the emotional expression shown by the user's facial muscles and posture.

[1147] "User movement" refers to the movement or change in pose of a part or the whole of the user's body.

[1148] "Voice data" is information that is generated by detecting a user's speech and digitizing it.

[1149] "Emotional state" refers to the psychological state or emotion that a user expresses through their voice or facial expression.

[1150] A "remote conference" is a conference held over the Internet with multiple participants in remote locations.

[1151] "Live streaming" refers to the transmission of video and audio to viewers in real time via the Internet.

[1152] A "server" is a computer system that provides various services and data over a network.

[1153] "Camera footage" is video footage data captured by a camera.

[1154] "Analysis" is the process of breaking down acquired data into elements and examining them in detail.

[1155] "Recognition" is the process by which a system understands specific information from inputs such as a user's facial expressions or voice.

[1156] "Reflection" refers to reflecting the acquired information in the movements and expressions of the 3D model.

[1157] The system embodying this invention generates a 3D model based on prompts entered by the user and includes a set of means for reflecting the user's facial expressions and movements in the model. The system also recognizes the user's emotional state and reflects it in the 3D model, enabling more natural and interactive communication in remote meetings and live broadcasts.

[1158] The system is implemented using the following hardware and software: The hardware includes the user's device (e.g., smartphone, tablet) and the server. The software includes a 3D model generation engine (e.g., Blender + Python script), a speech recognition API (e.g., Google Cloud Speech-to-Text API), and an emotion recognition API (e.g., Microsoft Azure Emotion API).

[1159] Program processing explanation

[1160] 1. Receiving prompts and generating 3D models

[1161] A user types a prompt into an application on their device, such as "a cheerful store clerk in colorful clothing."

[1162] The server analyzes the received prompt and generates the specified 3D model using a 3D model generation engine, which then stores the model on the server and sends it to the user's device.

[1163] 2. Video and audio analysis

[1164] Users can use their device's camera and microphone to start remote meetings and live broadcasts.

[1165] The device captures camera footage and audio data in real time and uses a speech recognition API to analyze the audio data, for example, when a user says, "What are the features of this smartphone?"

[1166] 3. Recognizing emotional states

[1167] The device sends the captured video to an emotion recognition API, which recognizes the user's emotions in real time. For example, if a user is smiling while talking, the emotion recognition API will recognize the user's state as "happy."

[1168] 4. Reflecting emotions and actions

[1169] The server updates the 3D model based on the analyzed voice and emotion data. Specifically, if the voice analysis results identify a description of the "smartphone's features" and the emotion recognition results indicate "happiness," the 3D model will also explain with a smile.

[1170] 5. Real-time video transmission

[1171] The device transmits real-time video of the generated 3D model to remote conferencing or live streaming applications, where it can be configured as a virtual camera and used in remote conferencing or live streaming software (e.g., Zoom or YouTube Live).

[1172] Specific examples

[1173] A user types in "cheerful sales clerks in colorful clothing."

[1174] The server receives the prompt, generates the corresponding 3D model, and sends it to the user's device.

[1175] When a user asks a question using the camera and microphone, the device analyzes the video and audio, and the emotion recognition API recognizes the user's emotion of joy.

[1176] The server updates the 3D model based on the analysis and recognition results and answers the user's questions with a smile.

[1177] A user connects to a remote conferencing app, and the generated 3D model acts and expresses emotions on behalf of the user during the meeting.

[1178] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1179] Step 1:

[1180] The user enters a prompt into an application on the device, such as "a cheerful store clerk in colorful clothing." This prompt is sent from the user's device to the server as an HTTP POST request.

[1181] Step 2:

[1182] The server analyzes the received prompt sentence and generates the specified 3D model using a 3D model generation engine (e.g. Blender + Python script). The generated 3D model is saved on the server and the generated data is sent to the user's device. As an output, 3D model data is generated and sent to the user's device.

[1183] Step 3:

[1184] A user can use an application installed on the device to start a remote meeting or live broadcast using the camera and microphone, and the device acquires the real-time camera video and audio data captured at that time. As output, the camera video and audio data are obtained.

[1185] Step 4:

[1186] The device analyzes the captured camera video and audio data. When analyzing the camera video, a computer vision algorithm (e.g., OpenCV) is used to detect facial features and body posture. When analyzing the audio data, a speech recognition API (e.g., Google Cloud Speech-to-Text API) is used. This allows the user's facial expressions, movements, and speech content to be analyzed.

[1187] Step 5:

[1188] The device uses an emotion recognition API (e.g., Microsoft Azure Emotion API) to recognize the user's emotional state based on the analyzed voice data and camera image data. For example, if the user is smiling while speaking, it will be recognized as "happy." Emotional information is obtained as output.

[1189] Step 6:

[1190] The server updates the 3D model based on the analyzed facial expression, motion, emotion, and voice data. Specifically, it changes the 3D model's facial expressions and movements in real time to reflect the user's actions and emotions. The updated 3D model data is obtained as the output.

[1191] Step 7:

[1192] The user's device transmits real-time video of the generated 3D model to a remote meeting or live streaming platform (e.g., Zoom, YouTube Live). This video is set up as a virtual camera and connected to the remote meeting or streaming software. The output is a video of the 3D model that moves and expresses emotions in real time during the remote meeting or live streaming.

[1193] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires audio indicating a user input regarding the result of the specific processing. The control unit 46A transmits audio data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the audio data.

[1194] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1195] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[1196] [Fourth embodiment]

[1197] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[1198] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[1199] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a WAN (Wide Area Network) and / or a LAN (Local Area Network).

[1200] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[1201] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs audio in accordance with instructions from the processor 46.

[1202] Camera 42 is a small digital camera equipped with an optical system including a lens, aperture, and shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (for example, an imaging range defined by an angle of view equivalent to the field of vision of a typical healthy person).

[1203] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for the exchange of various information between the processor 46 and the processor 28 via the network 54. The exchange of various information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is carried out in a secure state.

[1204] The control object 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, and feet. The posture and gestures of the robot 414 are controlled by controlling the motors of the arms, hands, and feet. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[1205] Fig. 8 shows an example of the main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[1206] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32 and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[1207] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[1208] In the robot 414, the processor 46 performs the reception output process. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50 and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[1209] Next, a description will be given of the specific processing performed by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1210] System Overview

[1211] This invention is a system that allows users to input prompts, generate original 3D avatars based on those prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar, allowing the avatar to move in accordance with the user's movements.

[1212] Program processing explanation (Example)

[1213] Avatar generation

[1214] The user enters a prompt, which the device sends to the system. For example, if the user enters a prompt such as "a blue character with deer antlers," the device sends this information to the server. The server receives the prompt and uses a 3D modeling engine to generate a corresponding 3D avatar. The generated avatar is temporarily stored on the server and then sent as data to the user's device. The device receives this data and displays the generated avatar to the user.

[1215] Camera footage analysis

[1216] When a user starts a remote meeting or live broadcast using a camera connected to their device, the video is captured in real time. The device analyzes the camera video and detects facial features and body movements. For example, when a user smiles or moves their hands, computer vision technology is applied to recognize these movements in real time.

[1217] Avatar Action Link

[1218] The device then moves each part of the 3D avatar it generates based on the analysis results. For example, if the user smiles, the 3D avatar's face will also move to smile. Similarly, if the user moves their hand, the corresponding movement is transmitted to the avatar. In this way, the user's movements are directly linked to the 3D avatar.

[1219] Real-time video streaming

[1220] Users connect their 3D avatar to remote conferencing or live streaming software. Specifically, they can set the 3D avatar as a virtual camera in video conferencing software such as Zoom or streaming software such as YouTube Live. The device transmits video of the 3D avatar moving in real time through this virtual camera. When the user speaks or moves, the avatar also moves in the same way, allowing viewers and participants to communicate with the user through the avatar.

[1221] Specific examples

[1222] 1. Prompt Input: The user types "a wizard-looking character."

[1223] 2. Avatar generation: The server receives the prompt, generates a wizard-like 3D avatar, and sends it to the user's device.

[1224] 3. Camera analysis: The user starts the meeting through the camera, and the device analyzes the user's facial expressions and movements.

[1225] 4. Avatar Reflection: Based on the analysis results, the 3D avatar moves in accordance with the user's movements. For example, if the user raises their hand, the avatar will raise their hand as well.

[1226] 5. Broadcasting: The user sets up the generated 3D avatar in Zoom, and the avatar acts on the user's behalf during the meeting.

[1227] Thus, the present invention provides a system that allows users to easily generate 3D avatars and use them in remote meetings and live streaming.

[1228] The processing flow will be explained below.

[1229] Program processing flow

[1230] Avatar generation process

[1231] Step 1:

[1232] The user types a prompt into the system, for example, "a blue character with deer antlers."

[1233] Step 2:

[1234] The terminal sends the user's prompt data to the server.

[1235] Step 3:

[1236] The server receives the prompt and parses and understands its contents.

[1237] Step 4:

[1238] The server's 3D modeling engine generates a 3D avatar based on the prompts.

[1239] Step 5:

[1240] The server stores the generated 3D avatar data and sends it to the user's device.

[1241] Step 6:

[1242] The terminal displays the received 3D avatar data and allows the user to confirm it.

[1243] Camera image analysis processing

[1244] Step 1:

[1245] The user connects the camera to their computer and launches the remote conferencing software or live streaming software.

[1246] Step 2:

[1247] The device acquires video data from the camera in real time.

[1248] Step 3:

[1249] The device analyzes the captured camera footage.

[1250] Step 4:

[1251] The device detects the user's facial features and body posture, for example, identifying the user's eye and mouth movements and hand positions.

[1252] Avatar movement link processing

[1253] Step 1:

[1254] The device moves each part of the 3D avatar based on the analysis results.

[1255] Step 2:

[1256] For example, if the user smiles, the 3D avatar's face will also smile. Similarly, if the user moves their hand, the avatar's hand will also move in a similar way.

[1257] Step 3:

[1258] The device repeats this real-time processing, constantly updating the 3D avatar in accordance with the user's movements.

[1259] Real-time video distribution processing

[1260] Step 1:

[1261] The user connects the generated 3D avatar to remote conferencing software or live streaming software, for example, by setting it as a virtual camera on Zoom or YouTube Live.

[1262] Step 2:

[1263] The device continues to maintain a link to the 3D avatar's movements.

[1264] Step 3:

[1265] The device transmits images of a 3D avatar moving in real time via a virtual camera to remote conferencing software or live streaming software.

[1266] Step 4:

[1267] Users communicate through 3D avatars during remote meetings and live broadcasts. For example, when a user speaks, the 3D avatar's mouth moves in the same way.

[1268] In this way, the system reflects the user's movements through each step in real time onto a 3D avatar, allowing the footage to be used for remote meetings and live streaming.

[1269] Example 1

[1270] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1271] In modern remote meetings and live streaming, it is difficult for users to easily create and use original 3D avatars that reflect their own facial expressions and movements in real time. Another problem is that existing technologies lack the ability to accurately detect user movements through analysis of camera footage and then coordinate avatar movements in real time. By resolving these issues, users can enjoy more natural and intuitive remote communication.

[1272] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[1273] In this invention, the server includes means for generating a 3D avatar based on a prompt received from the user, means for the server to analyze the prompt using a generation AI model and generate a corresponding 3D avatar, means for acquiring a camera image of the user in real time and analyzing the image to detect the user's facial expressions and movements, means for reflecting the detected facial expressions and movements in the 3D avatar so that the avatar moves in response to the user's movements, and means for transmitting a video of the moving 3D avatar to a remote conference or live streaming. This allows users to easily generate a 3D avatar, and the avatar moves in real time in coordination with the user's movements, enabling more effective remote communication.

[1274] A "prompt" is a sentence or phrase that a user enters to indicate the features or elements required to generate a 3D avatar.

[1275] A "generative AI model" is a type of artificial intelligence technology that analyzes prompts received from users and generates corresponding 3D avatars.

[1276] A "3D avatar" is a three-dimensional virtual character generated based on a user's characteristics and preferences.

[1277] The "server" is a computing system that receives prompts, generates a 3D avatar using a generative AI model, and transmits the data to the user's device.

[1278] "Camera video" refers to video data acquired in real time from a camera device used by a user.

[1279] "Real-time analysis" is a technology that instantly processes camera footage and detects the user's facial expressions and movements.

[1280] "Means for detecting facial expressions and movements" refers to computer vision technology for recognizing the user's facial features and body movements.

[1281] A "virtual camera" is a technology that allows software-generated images to be treated as if they were real camera images.

[1282] A "remote conference" is a conference in which multiple users participate from remote locations via the Internet.

[1283] "Live streaming" refers to the distribution of video and audio to viewers in real time via the Internet.

[1284] The present invention provides a system for generating a 3D avatar based on prompts input by a user, and for the avatar to move in real time in response to the user's actions. The system includes the following means.

[1285] 1. Enter and submit the prompt

[1286] It starts with the user entering a prompt into the terminal interface, for example, "a red dragon with wings." Once the prompt is entered, the terminal sends the information to the system.

[1287] 2. Prompt Processing and Avatar Creation

[1288] The device sends the prompt to a server, which uses a generative AI model (such as DALLE-2 or DeepArt) to analyze the prompt and generate a corresponding 3D avatar. For example, if a user enters the prompt "a red dragon with wings," the server can generate the avatar using a 3D modeling tool like Blender.

[1289] 3. Save and send your avatar

[1290] The server temporarily stores the generated 3D avatar and sends the data to the device. The device receives the data and displays it as a preview to the user. An initial preview image of the generated avatar is displayed on the device screen.

[1291] 4. Real-time analysis of camera footage

[1292] A user starts a remote conference or live broadcast using a camera connected to their device. The device uses an application (such as OpenCV or MediaPipe) to analyze the camera image in real time and detect facial features and body movements. When the user smiles, the change in facial expression is detected in real time.

[1293] 5. Avatar Action Link

[1294] The device updates the behavior of the 3D avatar in real time based on the analysis results. For example, if the user raises their hand, the device detects that movement and mirrors the avatar's hand by raising it. Similarly, facial expressions are updated simultaneously; if the user smiles, the avatar also smiles.

[1295] 6. Real-time video distribution

[1296] The user sets the generated 3D avatar as a virtual camera in remote conferencing software such as Zoom or YouTube Live. The device then uses virtual camera software such as OBS Virtual Camera to transmit the avatar's image, which is updated in real time. When the user speaks or moves, their movements are reflected in the avatar and displayed to viewers and participants in real time. For example, if a user waves their hand during a meeting, the avatar will also wave.

[1297] In this way, a 3D avatar generated from user-entered prompts can be linked to the user's movements in real time and used in remote meetings and live broadcasts.

[1298] The flow of the identification process in the first embodiment will be described with reference to FIG.

[1299] Step 1:

[1300] The user enters a prompt (e.g., "a red dragon with wings") into the terminal interface. The entered prompt is sent to the system by the terminal. This prompt becomes input data to the system. The terminal sends the prompt to the server in JSON format, for example.

[1301] Step 2:

[1302] The server analyzes the received prompt text. First, it inputs the input prompt text into a generative AI model (e.g., DALLE-2 or DeepArt). The generative AI model analyzes the prompt text and generates design data for the corresponding 3D avatar. This data includes information such as the avatar's shape, color, and movement patterns. The server generates the 3D avatar using 3D modeling software (e.g., Blender) based on the design data. The generated 3D avatar is temporarily stored on the server.

[1303] Step 3:

[1304] The server sends the temporarily saved 3D avatar data to the device. The device analyzes the received 3D avatar design data and displays a preview to the user. This display includes the overall image and main features of the avatar. The user can check the avatar and request changes or adjustments as necessary.

[1305] Step 4:

[1306] To start a remote meeting or live broadcast, a user begins capturing video using a camera. The device acquires video from the camera in real time and analyzes it using computer vision technologies such as OpenCV and MediaPipe. This analysis detects facial features and body movements. Analyzed data includes the user's facial expressions, eye movements, and hand movements.

[1307] Step 5:

[1308] The device then uses the analysis results to adjust each part of the 3D avatar in real time. For example, if the user raises their hand, the device detects that movement and adjusts the 3D avatar's hand so that it also raises. Similarly, if the user smiles, the avatar's face also changes to a smile. The device uses this data to update the 3D avatar's movements in real time.

[1309] Step 6:

[1310] Users set up a 3D avatar as a virtual camera in remote conferencing or live streaming software (e.g., Zoom or YouTube Live). The device then uses virtual camera software such as OBS Virtual Camera to transmit a live image of the avatar, which is updated in real time. When the user speaks or moves, their actions are reflected in the avatar in real time, allowing for natural communication with viewers and participants.

[1311] (Application example 1)

[1312] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1313] There is a demand for virtual stores to provide users with a realistic shopping experience, guided in real time. However, existing systems lack avatar guidance functions that synchronize with the user's movements, and users have difficulty in easily customizing and using 3D avatars. In addition, to provide more attractive virtual store guidance, it is desirable for the avatar to synchronize with the user's movements and facial expressions in real time.

[1314] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[1315] In this invention, the server includes: means for generating a 3D avatar based on prompts received from the user; means for capturing camera footage of the user in real time and analyzing the footage to detect the user's facial expressions and movements; means for reflecting the detected facial expressions and movements in the 3D avatar so that the avatar moves in response to the user's movements; means for displaying the moving 3D avatar on the user's device; and means for the user to move with the avatar in a virtual space and for the avatar to provide guidance as the user moves. This allows the user to easily create a customized 3D avatar based on their own movements and prompts, enabling a shopping experience in which they are guided through a virtual store in real time.

[1316] A "prompt" is a word or phrase entered by a user to indicate an instruction or request.

[1317] A "3D avatar" is a three-dimensional virtual character that is generated based on the user's prompts and camera footage, and moves in response to the user's movements and facial expressions.

[1318] "User's camera footage" refers to footage captured in real time by the camera of the device used by the user.

[1319] "Analysis of camera footage" is the process of detecting the user's facial expressions and movements from camera footage acquired in real time and evaluating their features and posture.

[1320] "Real-time" refers to a state in which operations and actions are reflected instantly with almost no delay.

[1321] "Moving 3D avatar video" refers to a 3D avatar that moves in real time in response to the user's facial expressions and movements, and is displayed as an image or video.

[1322] "Virtual space" refers to an imaginary environment or space generated using computer graphics or virtual reality technology.

[1323] A "virtual store" is a virtual shopping environment created using the Internet and virtual reality technology that has the same functions as a physical store.

[1324] "Means for providing guidance" refers to the function of the avatar to provide information to the user and guide the user to their destination when the user moves within the virtual space.

[1325] A "device" is an electronic device used by a user, such as a smartphone, tablet, or computer.

[1326] System implementation

[1327] This system generates a 3D avatar based on prompts received from the user, analyzes the user's camera footage in real time to detect the user's facial expressions and movements, and reflects those movements in the 3D avatar. Specifically, the following hardware and software are required:

[1328] Hardware: devices such as smartphones or tablets, cameras

[1329] Software: 3D modeling engine (Unity, Unreal Engine, etc.), facial recognition software (OpenCV, Dlib, etc.)

[1330] Program processing explanation

[1331] 1. Prompt input and avatar generation

[1332] A user uses a smartphone app to input a prompt such as "Nordic-style guide character." For example, the user types "Nordic-style guide."

[1333] The device sends this prompt to the server, which uses the generative AI model to generate a corresponding 3D avatar, which is then sent from the server to the device and displayed to the user.

[1334] 2. Analysis of camera footage

[1335] As users begin to move around the virtual store through the camera, the device captures camera footage and uses facial recognition software such as OpenCV or Dlib to detect facial features and body poses in real time.

[1336] 3. Movement and Avatar Movement Links

[1337] The detected facial expressions and movements are then reflected in a 3D avatar in real time based on the user's movements: for example, if the user raises their hand, the avatar will do the same.

[1338] 4. Real-time store information

[1339] The device provides a mechanism for a moving 3D avatar to be displayed on the user's device. As the user moves through the virtual space, the avatar walks ahead of the user, providing real-time guidance and explaining the features of various products and stores.

[1340] Specific examples

[1341] 1. Example of prompt input:

[1342] For example, a user enters "Scandinavian style guide."

[1343] 2. Avatar generation example:

[1344] The server receives the prompt, generates a Nordic-style guide character using a 3D modeling engine, and sends it to the user's device.

[1345] 3. Camera analysis example:

[1346] As users begin to move around the virtual store using their device's camera, the device analyzes the camera footage and detects facial features and body posture in real time.

[1347] 4. Avatar reflection example:

[1348] Based on the detection results, the 3D avatar moves in sync with the user's movements. For example, when the user smiles, the avatar also smiles.

[1349] 5. Store location example:

[1350] As the user moves around the virtual space with the device, the avatar explains product information and makes comments based on the product the user is looking at.

[1351] This system allows users to easily generate 3D avatars, further enhancing the shopping experience in virtual stores.

[1352] The flow of the specific processing in the application example 1 will be described with reference to FIG.

[1353] Step 1:

[1354] Prompt input and submit

[1355] The user starts the smartphone app and inputs a prompt for the 3D avatar they want to create, such as a "Nordic-style guide character." The input prompt is sent to the server by the device.

[1356] Input: User prompt (e.g. "Nordic style guide")

[1357] Output: Send prompt to server

[1358] Specific operation: The user enters a prompt using text input and presses the send button. The device sends this prompt to the server as an HTTP request.

[1359] Step 2:

[1360] 3D avatar generation

[1361] The server analyzes the received prompt using a generative AI model to generate a 3D avatar with the specified characteristics. The generated 3D avatar is temporarily stored on the server and then sent to the device as data.

[1362] Input: prompt (e.g. "Nordic style guide")

[1363] Output: Generated 3D avatar

[1364] How it works: The server uses a generative AI model to analyze the prompt and generates a 3D avatar using Unity or Unreal Engine, which is then stored on the server and sent to the device as an HTTP response.

[1365] Step 3:

[1366] Displaying 3D avatars

[1367] The device displays the 3D avatar received from the server.

[1368] Input: 3D avatar data sent from the server

[1369] Output: 3D avatar displayed on device

[1370] Specific operation: The device deserializes the received 3D avatar data and displays it to the user using a Unity viewer or similar.

[1371] Step 4:

[1372] Camera footage capture and analysis

[1373] As the user moves around the virtual store, the device captures and analyzes camera footage in real time, using OpenCV and Dlib to detect facial features and body poses.

[1374] Input: User's camera video

[1375] Output: Analysis results of user's facial expressions and movements

[1376] How it works: The device's camera captures video in real time, and then uses libraries such as OpenCV and Dlib to analyze facial features and body movements. The analysis results are output as data showing the user's current facial expressions and movements.

[1377] Step 5:

[1378] Avatar movement reflection

[1379] The device then uses the analysis results to update the 3D avatar's movements in real time, adjusting the avatar to perform similar movements based on the user's movements.

[1380] Input: Analyzed user facial expression and movement data

[1381] Output: A moving 3D avatar

[1382] Specific operation: Based on the analysis results, each joint and facial expression of the 3D avatar is updated in real time using a game engine such as Unity.

[1383] Step 6:

[1384] Guidance within the virtual store

[1385] As users move through the virtual space, a 3D avatar on the device acts as a real-time guide.

[1386] Input: User movement data in virtual space

[1387] Output: Guidance information using 3D avatars

[1388] Specific operation: The device uses the user's location and movement data to provide a 3D avatar with product information and directions. For example, when the user stands in front of a nearby product, the avatar will explain the product's features.

[1389] Step 7:

[1390] Display on user devices

[1391] The terminal displays a moving 3D avatar and real-time information on the user's device, and the user can view and interact with the avatar's guidance through their own device.

[1392] Input: A moving 3D avatar and related information

[1393] Output: What is displayed on the user's device

[1394] Specific operation: The terminal renders the display data and displays it in real time on the user's device, allowing the user to experience the virtual store through interaction with a 3D avatar on the device.

[1395] Furthermore, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[1396] System Overview

[1397] This invention is a system that allows users to input prompts, generate original 3D avatars based on those prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar. It also has the ability to recognize the user's emotions using an emotion engine and reflect those emotions in the 3D avatar.

[1398] Program processing explanation (Example)

[1399] Avatar generation

[1400] A user enters a prompt into the system, and the device sends the prompt to the server. For example, if a user enters a prompt such as "a blue character with deer antlers," the device sends this information to the server. The server receives the prompt and uses a 3D modeling engine to generate a corresponding 3D avatar. This generated avatar is temporarily stored on the server and then sent as data to the user's device. The device receives this data and displays the generated avatar to the user.

[1401] Camera video and audio analysis

[1402] When a user starts a remote meeting or live broadcast using a camera and microphone connected to their device, video and audio are captured in real time. The device acquires and analyzes the camera video and audio data. Specifically, the device detects facial features and body posture, and simultaneously analyzes the audio data to recognize the user's emotional state. For example, when a user smiles or moves their hands, computer vision and voice recognition technologies are applied to recognize these movements in real time.

[1403] Emotion recognition by emotion engine

[1404] The device passes the captured video and audio data to the emotion engine, which recognizes the user's emotions in real time. The emotion engine detects positive (happiness, satisfaction) and negative (anger, sadness) emotions from facial and audio data. For example, when a user speaks with a smile, the emotion engine recognizes that the user is "happy" from their voice and facial expression.

[1405] Linking avatar movements and emotions

[1406] Based on the analysis results and the recognition results of the emotion engine, the device links each part of the generated 3D avatar with its emotional expression. For example, if the user is smiling and happy, the 3D avatar's face will also smile and show a happy expression. Similarly, if the user is angry, the avatar's face will show an angry expression. In this way, the user's movements and emotions are reflected directly in the 3D avatar in real time.

[1407] Real-time video streaming

[1408] The user connects the generated 3D avatar to remote conferencing software or live streaming software. Specifically, the 3D avatar can be set as a virtual camera in video conferencing software such as Zoom or streaming software such as YouTube Live. The device sends video of the 3D avatar, which moves and expresses emotions in real time, to the remote conferencing software or streaming software via this virtual camera. When the user speaks or moves, the avatar also expresses the same movements and emotions, allowing viewers and participants to communicate with the user through the avatar.

[1409] Specific examples

[1410] 1. Prompt Input: The user types "a wizard-looking character."

[1411] 2. Avatar generation: The server receives the prompt, generates a wizard-like 3D avatar, and sends it to the user's device.

[1412] 3. Video and audio analysis: The user initiates a remote meeting via a camera and microphone, and the device analyzes the user's facial expressions and voice.

[1413] 4. Emotion recognition: The emotion engine recognizes emotions from the user's video and audio, detecting, for example, a "happy" state.

[1414] 5. Avatar Reflection: Based on the analysis results and emotion recognition results, a 3D avatar expresses the user's movements and emotions in real time.

[1415] 6. Distribution: The user sets the generated 3D avatar in remote conferencing software, and the avatar acts on behalf of the user during the meeting and expresses emotions.

[1416] Thus, the present invention provides a system that generates a 3D avatar that reflects a user's movements and emotions in real time and can be used in remote meetings and live streaming.

[1417] The processing flow will be explained below.

[1418] Program processing flow

[1419] Avatar generation process

[1420] Step 1:

[1421] The user types a prompt into the system, for example, "a blue character with deer antlers."

[1422] Step 2:

[1423] The terminal sends the user's prompt data to the server.

[1424] Step 3:

[1425] The server receives the prompt and parses and understands its contents.

[1426] Step 4:

[1427] The server's 3D modeling engine generates a 3D avatar based on the prompts.

[1428] Step 5:

[1429] The server stores the generated 3D avatar data and sends it to the user's device.

[1430] Step 6:

[1431] The terminal displays the received 3D avatar data and allows the user to confirm it.

[1432] Camera video and audio analysis processing

[1433] Step 1:

[1434] The user connects a camera and microphone to their computer and launches remote conferencing software or live streaming software.

[1435] Step 2:

[1436] The device acquires video data in real time from the camera and audio data from the microphone.

[1437] Step 3:

[1438] The device analyzes the captured camera footage.

[1439] Step 4:

[1440] The device detects the user's facial features and body posture, for example, identifying the user's eye and mouth movements and hand positions.

[1441] Step 5:

[1442] The device analyzes the acquired voice data and recognizes the user's emotional state from the voice.

[1443] Emotion recognition processing using an emotion engine

[1444] Step 1:

[1445] The device passes the acquired video and audio data to the emotion engine.

[1446] Step 2:

[1447] The emotion engine analyzes the user's facial expressions from video data and recognizes emotions by analyzing the tone and speaking style from audio data.

[1448] Step 3:

[1449] For example, if a user speaks with a smile, the emotion engine will recognize from this data that the user is "happy."

[1450] Avatar movement and emotion link processing

[1451] Step 1:

[1452] The device moves each part of the 3D avatar based on the analysis results and the recognition results of the emotion engine.

[1453] Step 2:

[1454] For example, if the user is smiling and "happy," the 3D avatar's face will also smile and move to show a happy expression. Similarly, if the user moves their hand, the avatar's hand movement will reflect that movement.

[1455] Step 3:

[1456] The device repeats this real-time processing, constantly updating the 3D avatar to match the user's movements and emotions.

[1457] Real-time video distribution processing

[1458] Step 1:

[1459] The user connects the generated 3D avatar to remote conferencing software or live streaming software, for example, by setting it as a virtual camera on Zoom or YouTube Live.

[1460] Step 2:

[1461] The device maintains a link between the movements and emotions of the 3D avatar.

[1462] Step 3:

[1463] The device transmits images of a 3D avatar that moves and expresses emotions in real time via a virtual camera to remote conferencing software or live streaming software.

[1464] Step 4:

[1465] Users communicate through 3D avatars during remote meetings and live broadcasts. For example, when a user speaks, the 3D avatar moves its mouth in the same way and changes its facial expression depending on the recognized emotion.

[1466] In this way, the system reflects the user's movements and emotions through each step in real time onto a 3D avatar, allowing the footage to be used for remote meetings and live streaming.

[1467] Example 2

[1468] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1469] When users use 3D avatars in remote meetings or live streaming, conventional systems often fail to accurately reflect the user's movements and facial expressions. Furthermore, the 3D avatars generated based on prompts may not match the user's preferences, making customization difficult. This reduces the quality of the user experience and makes efficient communication difficult.

[1470] The identification process by the identification processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for generating a 3D avatar based on a prompt received from the user, a means for acquiring the user's camera video and audio in real time and analyzing the video and audio to detect the user's facial expressions and movements, a means for recognizing the user's emotions using an emotion recognition engine, a means for reflecting the detected facial expressions, movements, and recognized emotions in the 3D avatar so that the avatar moves in accordance with the user's movements and movements, and a means for transmitting the video and audio of the moving 3D avatar to a remote conference or live streaming. This makes it possible to provide a customizable 3D avatar based on the prompt, reflecting the user's movements and movements in real time.

[1471] A "prompt" is text information that indicates a request or instruction that a user inputs to a system.

[1472] A "3D avatar" is a character or icon that can be displayed in three-dimensional space and operates in real time by reflecting the user's movements and emotions.

[1473] "Camera footage" refers to real-time video data captured by a camera connected to a user's terminal.

[1474] "Audio Data" refers to real-time audio data captured by a microphone connected to a user's terminal.

[1475] "Analyzing" refers to the process of analyzing the acquired data and extracting information such as feature points, posture, and emotions.

[1476] An "emotion recognition engine" refers to software or algorithms for detecting a user's emotional state from facial expression data and voice data.

[1477] A "remote conference" is a conference conducted in real time by multiple participants using video and audio over the Internet.

[1478] "Live streaming" refers to a form of broadcasting in which video and audio are transmitted in real time to a large number of viewers via the Internet.

[1479] "Customizable" refers to the ability to change the appearance and behavior of the generated 3D avatar based on user specifications.

[1480] "Detecting facial expressions and movements" refers to the process of recognizing facial features and body posture from camera footage and extracting that information.

[1481] This invention is a system that allows users to input prompts, generate original 3D avatars based on the prompts, and use them in remote meetings and live streaming. The system analyzes the user's camera footage in real time and reflects the analysis results in the 3D avatar. It also has the function of recognizing the user's emotions using an emotion engine and reflecting them in the 3D avatar.

[1482] Hardware and Software Configuration

[1483] The server is a computer with high-performance computing power, and has Blender or Unity installed as a 3D modeling engine. The server also receives prompts from users and provides an environment for running programs to generate 3D avatars. Specifically, the following software is used:

[1484] 3D modeling engine: Blender, Unity

[1485] The device is connected to an interface for users to input prompts, and hardware (camera and microphone) for capturing camera images and audio. The device also has software (OpenCV, Dlib, Google Cloud Speech-to-Text API) for analyzing camera images and audio data in real time. The device also has an emotion engine (Affectiva SDK) installed for emotion recognition.

[1486] Example of operation

[1487] 1. Prompt input: The user inputs "a wizard-like character." This prompt is sent from the terminal to the server.

[1488] 2. Avatar generation: The server receives the prompt and generates a wizard-like 3D avatar using Blender or Unity. The generated 3D avatar is sent to the user's device.

[1489] 3. Video and audio analysis: When a user starts a remote meeting through a camera and microphone, the device captures and analyzes camera video and audio data in real time.

[1490] 4. Emotion Recognition: The emotion engine (Affectiva SDK) recognizes the user's emotions based on video and audio data. For example, if the user is smiling, the system will recognize that the user is "happy."

[1491] 5. Avatar reflection: Based on the analysis results and emotion recognition results, the device moves each part of a 3D avatar created using Blender or Unity, reflecting emotional expressions in real time.

[1492] 6. Broadcasting: Users can set the generated 3D avatar in remote conferencing or broadcasting software such as Zoom or YouTube Live, and control the avatar in real time through a virtual camera. Viewers and participants can communicate with the user through the avatar.

[1493] This invention provides a 3D avatar that reflects the user's movements and emotions in real time, significantly improving the user experience in remote meetings and live streaming.

[1494] The flow of the identification process in the second embodiment will be described with reference to FIG.

[1495] Step 1:

[1496] The user inputs a prompt. The input is in the form of text, such as a request for a "magic-looking character." This input prompt is then sent to the server. Here, the user uses the system's interface to enter the prompt text into a text field.

[1497] Input: User prompt (e.g., "A character that looks like a wizard")

[1498] Output: Data containing a prompt (e.g., {"prompt": "A wizard-like character"})

[1499] Step 2:

[1500] The terminal sends the user's prompt to the server via a protocol such as an HTTP request, which passes the prompt to the server, where the terminal receives it and prepares the data in the appropriate format for transmission to the server.

[1501] Input: Data containing a prompt (e.g., {"prompt": "A wizard-like character"})

[1502] Output: Status of completion of sending to server

[1503] Step 3:

[1504] The server receives the prompt and generates a 3D avatar using a 3D modeling engine such as Blender or Unity. The server analyzes the prompt and generates an avatar based on the corresponding 3D model template.

[1505] Input: Prompt text (e.g., {"prompt": "A wizard-like character"})

[1506] Output: Generated 3D avatar data (e.g. 3D modeling data file)

[1507] Step 4:

[1508] The server sends the generated 3D avatar to the user's device. The generated 3D avatar data is appropriately compressed or formatted and transferred to the user's device.

[1509] Input: Generated 3D avatar data (e.g., 3D modeling data file)

[1510] Output: Transmission completed status to user terminal

[1511] Step 5:

[1512] The user starts broadcasting using the camera and microphone. The user starts the remote conferencing software or live broadcasting software and enables the camera and microphone. This prepares the video and audio to be captured in real time.

[1513] Input: Connecting and enabling cameras and microphones

[1514] Output: Real-time video and audio stream data

[1515] Step 6:

[1516] The device captures and analyzes camera images and audio data in real time, detects facial features and posture using OpenCV and Dlib, and converts the audio data into text using the Google Cloud Speech-to-Text API.

[1517] Input: Real-time video and audio stream data

[1518] Output: Analysis results (e.g., facial feature points, body posture, voice text data)

[1519] Step 7:

[1520] The data acquired by the device is passed to the emotion recognition engine to recognize the user's emotions. The Affectiva SDK is used to detect the user's emotional state from video and audio data.

[1521] Input: Analysis results (e.g., facial feature points, body posture, voice text data)

[1522] Output: Emotion recognition results (e.g., joy, sadness, anger)

[1523] Step 8:

[1524] The device updates the 3D avatar based on the analysis and emotion recognition results. Using Blender or Unity, the 3D avatar moves in real time and changes its facial expression according to the user's movements and emotions.

[1525] Input: Analysis results and emotion recognition results

[1526] Output: 3D avatar data updated in real time

[1527] Step 9:

[1528] The device transmits the 3D avatar's movements to remote conferencing or live streaming software, and the updated 3D avatar video is sent as a live stream using software such as OBS Virtual Camera.

[1529] Input: 3D avatar data updated in real time

[1530] Output: Stream data to remote conference and live streaming software

[1531] (Application example 2)

[1532] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[1533] Conventional 3D avatar generation systems are limited to reflecting a user's facial expressions and movements, making it difficult to recognize a user's emotions in real time and reflect them in the 3D avatar. Furthermore, there is a lack of a way to express a user's movements and emotions more naturally during remote meetings and live streaming. For these reasons, there is a demand for improved user experience and more efficient communication.

[1534] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means.

[1535] In this invention, the server includes means for generating a 3D model based on prompts received from the user, means for capturing camera footage of the user in real time and analyzing the footage to detect the user's facial expressions and movements, means for recognizing the user's emotional state based on the analyzed voice data, and means for reflecting the recognized emotions in the 3D model. This allows the user's movements and emotions to be reflected in the 3D avatar in real time, enabling more natural and interactive communication in remote meetings and live streaming.

[1536] A "prompt" is a text instruction that a user enters to generate a particular 3D model.

[1537] A "3D model" is an object in three-dimensional space generated using computer graphics.

[1538] "User's facial expression" refers to the emotional expression shown by the user's facial muscles and posture.

[1539] "User movement" refers to the movement or change in pose of a part or the whole of the user's body.

[1540] "Voice data" is information that is generated by detecting a user's speech and digitizing it.

[1541] "Emotional state" refers to the psychological state or emotion that a user expresses through their voice or facial expression.

[1542] A "remote conference" is a conference held over the Internet with multiple participants in remote locations.

[1543] "Live streaming" refers to the transmission of video and audio to viewers in real time via the Internet.

[1544] A "server" is a computer system that provides various services and data over a network.

[1545] "Camera footage" is video footage data captured by a camera.

[1546] "Analysis" is the process of breaking down acquired data into elements and examining them in detail.

[1547] "Recognition" is the process by which a system understands specific information from inputs such as a user's facial expressions or voice.

[1548] "Reflection" refers to reflecting the acquired information in the movements and expressions of the 3D model.

[1549] The system embodying this invention generates a 3D model based on prompts entered by the user and includes a set of means for reflecting the user's facial expressions and movements in the model. The system also recognizes the user's emotional state and reflects it in the 3D model, enabling more natural and interactive communication in remote meetings and live broadcasts.

[1550] The system is implemented using the following hardware and software: The hardware includes the user's device (e.g., smartphone, tablet) and the server. The software includes a 3D model generation engine (e.g., Blender + Python script), a speech recognition API (e.g., Google Cloud Speech-to-Text API), and an emotion recognition API (e.g., Microsoft Azure Emotion API).

[1551] Program processing explanation

[1552] 1. Receiving prompts and generating 3D models

[1553] A user types a prompt into an application on their device, such as "a cheerful store clerk in colorful clothing."

[1554] The server analyzes the received prompt and generates the specified 3D model using a 3D model generation engine, which then stores the model on the server and sends it to the user's device.

[1555] 2. Video and audio analysis

[1556] Users can use their device's camera and microphone to start remote meetings and live broadcasts.

[1557] The device captures camera footage and audio data in real time and uses a speech recognition API to analyze the audio data, for example, when a user says, "What are the features of this smartphone?"

[1558] 3. Recognizing emotional states

[1559] The device sends the captured video to an emotion recognition API, which recognizes the user's emotions in real time. For example, if a user is smiling while talking, the emotion recognition API will recognize the user's state as "happy."

[1560] 4. Reflecting emotions and actions

[1561] The server updates the 3D model based on the analyzed voice and emotion data. Specifically, if the voice analysis results identify a description of the "smartphone's features" and the emotion recognition results indicate "happiness," the 3D model will also explain with a smile.

[1562] 5. Real-time video transmission

[1563] The device transmits real-time video of the generated 3D model to remote conferencing or live streaming applications, where it can be configured as a virtual camera and used in remote conferencing or live streaming software (e.g., Zoom or YouTube Live).

[1564] Specific examples

[1565] A user types in "cheerful sales clerks in colorful clothing."

[1566] The server receives the prompt, generates the corresponding 3D model, and sends it to the user's device.

[1567] When a user asks a question using the camera and microphone, the device analyzes the video and audio, and the emotion recognition API recognizes the user's emotion of joy.

[1568] The server updates the 3D model based on the analysis and recognition results and answers the user's questions with a smile.

[1569] A user connects to a remote conferencing app, and the generated 3D model acts and expresses emotions on behalf of the user during the meeting.

[1570] The flow of the specific processing in the application example 2 will be described with reference to FIG.

[1571] Step 1:

[1572] The user enters a prompt into an application on the device, such as "a cheerful store clerk in colorful clothing." This prompt is sent from the user's device to the server as an HTTP POST request.

[1573] Step 2:

[1574] The server analyzes the received prompt sentence and generates the specified 3D model using a 3D model generation engine (e.g. Blender + Python script). The generated 3D model is saved on the server and the generated data is sent to the user's device. As an output, 3D model data is generated and sent to the user's device.

[1575] Step 3:

[1576] A user can use an application installed on the device to start a remote meeting or live broadcast using the camera and microphone, and the device acquires the real-time camera video and audio data captured at that time. As output, the camera video and audio data are obtained.

[1577] Step 4:

[1578] The device analyzes the captured camera video and audio data. When analyzing the camera video, a computer vision algorithm (e.g., OpenCV) is used to detect facial features and body posture. When analyzing the audio data, a speech recognition API (e.g., Google Cloud Speech-to-Text API) is used. This allows the user's facial expressions, movements, and speech content to be analyzed.

[1579] Step 5:

[1580] The device uses an emotion recognition API (e.g., Microsoft Azure Emotion API) to recognize the user's emotional state based on the analyzed voice data and camera image data. For example, if the user is smiling while speaking, it will be recognized as "happy." Emotional information is obtained as output.

[1581] Step 6:

[1582] The server updates the 3D model based on the analyzed facial expression, motion, emotion, and voice data. Specifically, it changes the 3D model's facial expressions and movements in real time to reflect the user's actions and emotions. The updated 3D model data is obtained as the output.

[1583] Step 7:

[1584] The user's device transmits real-time video of the generated 3D model to a remote meeting or live streaming platform (e.g., Zoom, YouTube Live). This video is set up as a virtual camera and connected to the remote meeting or streaming software. The output is a video of the 3D model that moves and expresses emotions in real time during the remote meeting or live streaming.

[1585] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input regarding the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[1586] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (Internet Search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by performing deep learning on a neural network. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating voice, text data indicating text, and image data indicating an image is also input. The data generation model 58 performs inference on the input inference data in accordance with the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[1587] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the robot 414.

[1588] The emotion identification model 59 as an emotion engine may determine the user's emotion according to a specific mapping. Specifically, the emotion identification model 59 may determine the user's emotion according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the robot's emotion, and the identification processing unit 290 may perform identification processing using the robot's emotion.

[1589] FIG. 9 is a diagram illustrating an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. Emotions closer to the center of the concentric circles are more primitive. Emotions representing states and actions arising from a state of mind are arranged on the outer edges of the concentric circles. The concept of emotion includes both affect and mental states. Emotions generally generated from reactions occurring in the brain are arranged on the left side of the concentric circles. Emotions generally induced by situational judgment are arranged on the right side of the concentric circles. Emotions generally generated from reactions occurring in the brain and induced by situational judgment are arranged on the upper and lower sides of the concentric circles. Furthermore, the emotion of "pleasure" is arranged on the upper side of the concentric circles, and the emotion of "discomfort" is arranged on the lower side. In this way, in the emotion map 400, multiple emotions are mapped based on the structure by which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[1590] These emotions are distributed in the 3 o'clock direction on emotion map 400, and typically fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[1591] The inside of emotion map 400 represents what is going on in the mind, and the outside of emotion map 400 represents behavior, so the further you go outside emotion map 400, the more visible the emotions become (the more they are expressed in behavior).

[1592] Human emotions are based on various balances, such as posture and blood sugar levels. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. Emotions can also be created for robots, automobiles, and motorcycles, based on various balances, such as posture and remaining battery life. When these balances deviate from the ideal, a state of discomfort is indicated, and when they approach the ideal, a state of pleasure is indicated. An emotion map can be generated, for example, based on Dr. Mitsuyoshi's emotion map (Research on Voice Emotion Recognition and Emotional Brain Physiological Signal Analysis Systems, Tokushima University, Doctoral Dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). The left half of the emotion map lists emotions belonging to the "reaction" domain, where sensation is dominant. The right half of the emotion map lists emotions belonging to the "situation" domain, where situational awareness is dominant.

[1593] The emotion map defines two emotions that promote learning. One is a negative emotion on the situation side, around the middle of "repentance" or "reflection." In other words, this occurs when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is a positive emotion on the response side, around "desire." In other words, this occurs when the robot experiences positive feelings such as "I want more" or "I want to know more."

[1594] The emotion identification model 59 inputs user input into a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the user's emotion. This neural network is pre-trained based on multiple pieces of training data that are combinations of user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in FIG. 10. FIG. 10 shows an example in which multiple emotions, "relieved," "calm," and "reassuring," have similar emotion values.

[1595] The system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, but the system according to the present disclosure is not necessarily implemented on a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program running on a personal computer or an application running on a smartphone, etc. The method according to the present disclosure may be provided to users in the form of SaaS (Software as a Service).

[1596] In the above embodiment, an example was given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and data may be generated in the external device in accordance with input data.

[1597] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable, computer-readable, non-transitory storage medium such as a USB (Universal Serial Bus) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[1598] Alternatively, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 in response to a request from the data processing device 12.

[1599] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[1600] The hardware resource for executing a specific process can be any of the following processors: An example of a processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing a specific process by executing software, i.e., a program. Another example of a processor is a dedicated electrical circuit, such as an FPGA (Field-Programmable Gate Array), a PLD (Programmable Logic Device), or an ASIC (Application Specific Integrated Circuit), which is a processor with a circuit configuration designed specifically for executing a specific process. Each processor has built-in or connected memory, and each processor uses the memory to execute the specific process.

[1601] The hardware resource that executes the specific processing may be configured with one of these various processors, or may be configured with a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific processing may be a single processor.

[1602] As an example of a system configured with a single processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a system that uses a processor that realizes the functions of an entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[1603] Furthermore, the hardware structure of these various processors can be, more specifically, an electric circuit that combines circuit elements such as semiconductor devices. The specific processing described above is merely an example. Therefore, it goes without saying that unnecessary steps may be deleted, new steps may be added, or the processing order may be rearranged, without departing from the spirit of the invention.

[1604] The above-described description and illustrations are a detailed explanation of the parts related to the technology of the present disclosure and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, functions, actions, and effects is an explanation of an example of the configuration, functions, actions, and effects of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above-described description and illustrations within the scope of the gist of the technology of the present disclosure. Furthermore, to avoid confusion and facilitate understanding of the parts related to the technology of the present disclosure, the above-described description and illustrations omit explanations of common technical knowledge that do not require particular explanation to enable the implementation of the technology of the present disclosure.

[1605] All publications, patent applications, and technical standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or technical standard was specifically and individually indicated to be incorporated by reference.

[1606] The following is further disclosed regarding the above embodiment.

[1607] (Claim 1)

[1608] means for generating a 3D avatar based on prompts received from a user;

[1609] A means for acquiring a camera image of a user in real time and analyzing the image to detect the user's facial expressions and movements;

[1610] A means for reflecting the detected facial expressions and movements in a 3D avatar so that the avatar moves in response to the user's movements;

[1611] A means for transmitting the moving 3D avatar image to a remote conference or live streaming

[1612] A system including:

[1613] (Claim 2)

[1614] 10. The system of claim 1, further comprising means for allowing the generated 3D avatar to be customizable based on user specifications.

[1615] (Claim 3)

[1616] 2. The system of claim 1, further comprising means for detecting facial features and body posture in camera image analysis.

[1617] "Example 1"

[1618] (Claim 1)

[1619] means for generating a 3D avatar based on prompts received from a user;

[1620] a means for the server to analyze the prompt using a generative AI model and generate a corresponding 3D avatar;

[1621] A means for acquiring a camera image of a user in real time and analyzing the image to detect the user's facial expressions and movements;

[1622] A means for reflecting the detected facial expressions and movements in a 3D avatar so that the avatar moves in response to the user's movements;

[1623] A means for transmitting the moving 3D avatar image to a remote conference or live streaming

[1624] A system including:

[1625] (Claim 2)

[1626] 10. The system of claim 1, further comprising means for allowing the generated 3D avatar to be customizable based on user specifications.

[1627] (Claim 3)

[1628] 2. The system according to claim 1, further comprising means for detecting facial features and whole body posture in camera image analysis.

[1629] "Application Example 1"

[1630] (Claim 1)

[1631] means for generating a 3D avatar based on prompts received from a user;

[1632] A means for acquiring a camera image of a user in real time and analyzing the image to detect the user's facial expressions and movements;

[1633] A means for reflecting the detected facial expressions and movements in a 3D avatar so that the avatar moves in response to the user's movements;

[1634] a means for displaying the animated 3D avatar on the user's device;

[1635] A means for allowing a user to move with an avatar in a virtual space, and for the avatar to provide guidance in accordance with the user's movement;

[1636] A system including:

[1637] (Claim 2)

[1638] 10. The system of claim 1, further comprising means for allowing the generated 3D avatar to be customizable based on user specifications.

[1639] (Claim 3)

[1640] 2. The system of claim 1, further comprising means for detecting facial features and body posture in camera image analysis.

[1641] "Example 2: Combining Emotion Engines"

[1642] (Claim 1)

[1643] means for generating a 3D avatar based on prompts received from a user;

[1644] a means for capturing a camera image and voice of a user in real time and analyzing the image and voice to detect the user's facial expressions and movements;

[1645] means for recognizing a user's emotion using an emotion recognition engine;

[1646] A means for reflecting the detected facial expressions, movements, and recognized emotions in a 3D avatar so that the avatar behaves in accordance with the user's movements and emotions;

[1647] A means for transmitting the video and audio of the moving 3D avatar to a remote conference or live broadcast.

[1648] A system including:

[1649] (Claim 2)

[1650] 10. The system of claim 1, further comprising means for allowing the generated 3D avatar to be customizable based on user specifications.

[1651] (Claim 3)

[1652] 2. The system of claim 1, further comprising means for detecting facial features and body posture in camera image analysis.

[1653] "Application example 2 when combining emotion engines"

[1654] (Claim 1)

[1655] means for generating a 3D model based on prompts received from a user;

[1656] A means for acquiring a camera image of a user in real time and analyzing the image to detect the user's facial expressions and movements;

[1657] A means for reflecting the detected facial expressions and movements in a 3D model so that the model moves in response to the user's movements;

[1658] means for recognizing the emotional state of the user based on the analyzed voice data;

[1659] A means to reflect the recognized emotions in the 3D model,

[1660] A means for transmitting the video of the moving and emotional 3D model to a remote conference or live streaming.

[1661] A system including:

[1662] (Claim 2)

[1663] 10. The system of claim 1, further comprising means for allowing the generated 3D model to be customizable based on user specifications.

[1664] (Claim 3)

[1665] 2. The system of claim 1, further comprising means for detecting facial features and body posture in camera image analysis. [Explanation of symbols]

[1666] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. means for generating a 3D avatar based on prompts received from a user; A means for acquiring a camera image of a user in real time and analyzing the image to detect the user's facial expressions and movements; A means for reflecting the detected facial expressions and movements in a 3D avatar so that the avatar moves in response to the user's movements; A means for transmitting the moving 3D avatar image to a remote conference or live streaming A system including:

2. 10. The system of claim 1, further comprising means for allowing the generated 3D avatar to be customizable based on user specifications.

3. 2. The system according to claim 1, further comprising means for detecting facial features and body posture in the camera image analysis.

Images