system

Abstract

To provide a system.SOLUTION: A system in an SNS application for enabling at least one of social interactions and business meetings in a virtual space includes: means for receiving a user's input, recognizing the user's emotion using an emotion engine, and generating at least one of presentation support and discussion moderation in real time via a generative AI; and means for presenting at least one of the generated support and moderation.SELECTED DRAWING: Figure 1

Description

[Technical field]

[0001] The technology of the present disclosure relates to a system. [Background technology]

[0002] Patent document 1 discloses a persona chatbot control method performed by at least one processor, the method including the steps of receiving a user utterance, adding the user utterance to a prompt including a description and related instruction sentence regarding the chatbot character, encoding the prompt, and inputting the encoded prompt into a language model to generate a chatbot utterance in response to the user utterance. [Prior art documents] [Patent documents]

[0003] [Patent Document 1] JP 2022-180282 A Summary of the Invention [Problem to be solved by the invention]

[0004] Traditional social and business meetings held in virtual spaces lacked real-time presentation support and discussion moderation, making it difficult for participants to exchange opinions and share information smoothly. [Means for solving the problem]

[0005] The present invention solves the above problems by providing a means for receiving user input and generating presentation support and discussion moderation in real time via generative AI. In addition, the generative AI provides support and moderation taking into account the user's emotions, making it easier for participants to exchange opinions and share information. By using the above means, the conventional problems are solved and more effective social interactions and business meetings in virtual spaces are realized.

[0006] "SNS Application" means a social networking service that enables social interactions and business meetings in a virtual space.

[0007] "Generative AI" refers to natural language processing models that analyze user input and generate presentation support and discussion moderation in real time.

[0008] "Presentation support and discussion moderation" refers to providing appropriate information and advice based on user input, and helping to facilitate the exchange of opinions and discussion.

[0009] "Emotion-aware support and moderation" refers to providing appropriate responses and support to the user's emotions and expressions, thereby providing a more personalized conversational experience. [Brief description of the drawings]

[0010] [Figure 1] 1 is a conceptual diagram showing an example of the configuration of a data processing system according to a first embodiment. [Diagram 2] 1 is a conceptual diagram showing an example of main functions of a data processing device and a smart device according to a first embodiment. FIG. [Diagram 3] FIG. 11 is a conceptual diagram showing an example of the configuration of a data processing system according to a second embodiment. [Figure 4] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and smart glasses according to a second embodiment. [Diagram 5]FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a third embodiment. [Figure 6] FIG. 11 is a conceptual diagram showing an example of main functions of a data processing device and a headset-type terminal according to a third embodiment. [Figure 7] FIG. 13 is a conceptual diagram showing an example of the configuration of a data processing system according to a fourth embodiment. [Figure 8] FIG. 13 is a conceptual diagram showing an example of main functions of a data processing device and a robot according to a fourth embodiment. [Figure 9] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 10] 1 shows an emotion map onto which multiple emotions are mapped. [Figure 11] 4 is a sequence diagram showing a process flow of the data processing system according to the first embodiment. FIG. [Figure 12] 11 is a sequence diagram showing a process flow of the data processing system in application example 1. FIG. [Figure 13] FIG. 11 is a sequence diagram showing the flow of processing of the data processing system according to the second embodiment when an emotion engine is combined. [Figure 14] FIG. 11 is a sequence diagram showing the flow of processing in the data processing system in application example 2 when combined with an emotion engine. DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0011] Hereinafter, an example of an embodiment of a system according to the technology of the present disclosure will be described with reference to the accompanying drawings.

[0012] First, the terms used in the following description will be explained.

[0013] In the following embodiments, a signed processor (hereinafter simply referred to as a "processor") may be one arithmetic device or a combination of multiple arithmetic devices. Furthermore, a processor may be one type of arithmetic device or a combination of multiple types of arithmetic devices. Examples of arithmetic devices include a CPU (Central Processing Unit), a GPU (Graphics Processing Unit), a GPGPU (General-Purpose computing on Graphics Processing Units), an APU (Accelerated Processing Unit), etc.

[0014] In the following embodiments, a signed RAM (Random Access Memory) is a memory in which information is temporarily stored and is used as a working memory by the processor.

[0015] In the following embodiments, the coded storage is one or more non-volatile storage devices that store various programs, various parameters, etc. Examples of non-volatile storage devices include flash memory (SSD (Solid State Drive)), magnetic disks (e.g., hard disks), and magnetic tapes.

[0016] In the following embodiments, a communication I / F (Interface) with a code is an interface including a communication processor and an antenna. The communication I / F controls communication between multiple computers. An example of a communication standard applied to the communication I / F is a wireless communication standard including 5G (5th Generation Mobile Communication System), Wi-Fi (registered trademark), Bluetooth (registered trademark), etc.

[0017] In the following embodiments, "A and / or B" is synonymous with "at least one of A and B." In other words, "A and / or B" means that it may be only A, only B, or a combination of A and B. In addition, in this specification, the same idea as "A and / or B" is also applied when three or more things are expressed by connecting them with "and / or."

[0018] [First embodiment]

[0019] FIG. 1 shows an example of the configuration of a data processing system 10 according to the first embodiment.

[0020] 1, a data processing system 10 includes a data processing device 12 and a smart device 14. An example of the data processing device 12 is a server.

[0021] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0022] The smart device 14 includes a computer 36, a reception device 38, an output device 40, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The reception device 38, the output device 40, and the camera 42 are also connected to the bus 52.

[0023] The reception device 38 includes a touch panel 38A and a microphone 38B, and receives user input. The touch panel 38A detects contact with a pointer (e.g., a pen or a finger) to receive user input by touch. The microphone 38B detects the user's voice to receive user input by voice. A control unit 46A (see also FIG. 2), which will be described later, transmits data indicating the user input received by the touch panel 38A and the microphone 38B to the data processing device 12. In the data processing device 12, a specific processing unit 290 (see also FIG. 2), which will be described later, acquires the data indicating the user input.

[0024] The output device 40 includes a display 40A and a speaker 40B, and presents data to a user by outputting the data in a form of expression that the user can perceive (e.g., voice and / or text). The display 40A displays visible information such as text and images according to instructions from the processor 46. The speaker 40B outputs voice according to instructions from the processor 46. The camera 42 is a compact digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor.

[0025] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54.

[0026] FIG. 2 shows an example of main functions of the data processing device 12 and the smart device 14.

[0027] As shown in Fig. 2, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32. The specific process program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific process program 56 from the storage 32, and executes the read specific process program 56 on the RAM 30. The specific process is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific process program 56 executed on the RAM 30.

[0028] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0029] In the smart device 14, the processor 46 performs the reception output processing. The storage 50 stores a reception output program 60. The reception output program 60 is used together with the specific processing program 56 by the data processing system 10. The processor 46 reads out the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output processing is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0030] Next, a description will be given of the specific processing by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal".

[0031] The embodiment for carrying out the present invention comprises the following elements.

[0032] 1. Server: Receives user input and processes it to send it to the generative AI. It also receives support and moderation generated by the generative AI and processes it to send it to the terminal.

[0033] 2. Terminal: Displays the information entered by the user, and displays support and moderation received from the generative AI. It also detects the user's dialogue and actions and transmits them to the server.

[0034] 3. Generative AI: Analyzes user input and generates presentation support and discussion moderation in real time. It also takes into account the user's emotions to provide appropriate support and moderation.

[0035] 4. Virtual space: A virtual space for users to participate in, where events such as workshops and seminars are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0036] [Specific example]

[0037] When a user wishes to hold a business meeting in a virtual space, the invention can be implemented in the following manner.

[0038] 1. The user enters a self-introduction, suggestions, requests, etc., and sends them to the server via the terminal.

[0039] 2. The server receives the user's input and sends it to the generative AI.

[0040] 3. Generative AI analyses user input and generates presentation support and discussion moderation in real time.

[0041] 4. The generated support and moderation are sent to the terminal via the server.

[0042] 5. The device displays the support and moderation received and provides it to the user.

[0043] 6. Users can conduct business meetings in the virtual space while referring to the displayed support and moderation.

[0044] In this manner, users can smoothly conduct social gatherings and business meetings in the virtual space. Users can also participate in workshops and seminars in the virtual space and deepen their interactions with other participants.

[0045] The process flow will be explained below.

[0046] Step 1: The user inputs a self-introduction, proposal, request, etc., and sends it to the server via the terminal.

[0047] Step 2: The server receives the user's input and sends it to the generative AI.

[0048] Step 3: The generative AI analyzes user input and generates presentation support and discussion moderation in real time.

[0049] Step 4: The support and moderation generated by the generative AI are sent to the device via the server.

[0050] Step 5: The device displays the support and moderation received and provides it to the user.

[0051] Step 6: The user conducts a business meeting in the virtual space while referring to the displayed support and moderation.

[0052] Step 7: User interactions and actions are detected on the device and sent to the server.

[0053] Step 8: The server sends the user's interactions and actions to the generative AI, which updates them in real time.

[0054] Step 9: Updated support and moderation is sent to the device via the server.

[0055] Step 10: The terminal displays the received update information and provides it to the user.

[0056] Step 11: The user continues the business meeting in the virtual space while referring to the displayed updated information.

[0057] Example 1

[0058] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0059] In conventional social and business meetings in virtual spaces, users often lacked support to effectively present and discuss in real time. As a result, the quality and efficiency of meetings declined, and the user experience was impaired. In addition, it was difficult to properly analyze the user's input information and provide support and moderation that took the user's emotions into account. This meant that users could not receive appropriate feedback or support, making smooth communication difficult.

[0060] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0061] In this invention, the server includes means for receiving a user's input and generating at least one of a presentation support and a discussion moderation in real time via a generative artificial intelligence, means for transmitting at least one of the support and the moderation to the user's terminal, and means for displaying at least one of the support and the moderation on the user's terminal, thereby enabling the user to effectively socialize or hold a business meeting in a virtual space in real time. In addition, the generative artificial intelligence analyzes the user's input and provides appropriate support and moderation taking into account the user's emotions, improving the user's experience and making communication proceed smoothly.

[0062] "User input" refers to the input of a user's self-introduction, suggestions, opinions, requests, and requests to the other party through a terminal.

[0063] A "generative artificial intelligence" is an artificial intelligence model that analyzes user input and generates presentation support and / or discussion moderation in real time.

[0064] "Generating in real time" means selecting, generating, and outputting presentation support and discussion moderation almost instantly after receiving user input.

[0065] "Presentation support" refers to providing information and materials that enable users to give presentations effectively and smoothly.

[0066] "Discussion moderation" refers to providing advice, questions, feedback, etc. to facilitate discussions among users.

[0067] "Sending to the user's device" means transferring the generated support and moderation content as data from the server to a device such as a PC or smartphone used by the user.

[0068] "Displaying on the user's device" means presenting the submitted support and moderation content on the screen of the device (PC, smartphone, etc.) used by the user.

[0069] "Virtual social and business meetings" refers to activities in which users communicate with other users in a virtual environment created on the Internet, including business meetings and social gatherings.

[0070] The present invention relates to a system for smoothly conducting social interactions and business meetings in a virtual space. The configuration and operation of this system will be described in detail below.

[0071] System Configuration

[0072] The system consists of the following main elements:

[0073] 1. Server

[0074] The server uses a cloud server equipped with a high-performance CPU and memory (e.g. AWS (registered trademark) EC2 (registered trademark)). It receives user input and processes it to send it to the generative AI. It also receives support and moderation generated by the generative AI and processes it to send it to the terminal.

[0075] 2. Terminal

[0076] This refers to the PC or smartphone used by the user (e.g. Windows (registered trademark) PC, iPhone (registered trademark), ANDROID (registered trademark) device). The terminal displays the user's input and displays support and moderation received from the generative AI. It also detects the user's dialogue and actions and processes them to send them to the server.

[0077] 3. Generative Artificial Intelligence

[0078] Use specialized artificial intelligence tools (e.g., OpenAI ChatGPT, Google BERT). Analyze user input and generate presentation support and discussion moderation in real time. Additionally, provide appropriate support and moderation by taking into account user sentiment.

[0079] 4. Virtual Space

[0080] Use an online event platform (e.g. VRChat (registered trademark), Zoom (registered trademark)). This is a place where users can participate and events such as workshops and seminars are held.

[0081] Example of system operation

[0082] A specific example of operation is shown below.

[0083] 1. User Input The user types into the terminal, "I would like to share my ideas for promoting a new product."

[0084] 2. Server Processing The server receives user input and performs formatting and validation of the data. Send the generative artificial intelligence a prompt: "Generate promotional ideas for a new product."

[0085] 3. Analysis and Generation of Generative Artificial Intelligence Generative artificial intelligence analyses user input and generates relevant ideas and presentation materials in real time. Example prompt: "Generate promotional ideas for a new product."

[0086] 4. Sending the generated results The server transmits the generated ideas and materials to the user's terminal.

[0087] 5. Display of terminal The user's terminal displays the generated promotion ideas and materials on the screen. Users can refer to the displayed information and proceed with the meeting in the virtual space.

[0088] How to use

[0089] When a user holds a business meeting in a virtual space, he or she uses this system in the following steps.

[0090] 1. The user inputs a self-introduction, suggestions, requests, etc. through the terminal. 2. The server receives the user's input and sends it to the generative artificial intelligence. 3. Generative AI analyses user input and generates the necessary support and moderation in real time. 4. The generated support and moderation are sent to the user's device via the server. 5. The device displays the support and moderation received and provides it to the user. 6. The user proceeds with the meeting in the virtual space while referring to the displayed support and moderation.

[0091] This allows users to smoothly conduct social and business meetings in a virtual space and to communicate efficiently with other participants.

[0092] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0093] Program processing steps and detailed explanation

[0094] Step 1: Users use the terminal to input information such as self-introductions, suggestions, and requests. (Input) Text information entered by the user (e.g., "I would like to share my ideas for promoting new products") (Output) User input data that the terminal sends to the server (Specific operation) The user enters an idea into the text box on the terminal and clicks the send button.

[0095] Step 2: The server receives the data input by the user and performs initial processing, checking the format and content of the data and converting it into a format for sending to the generative artificial intelligence. (Input) User input data (Output) Formatted data (format for sending to generative artificial intelligence) (Specific operation) The server checks the string and unifies the format, and also handles errors if necessary.

[0096] Step 3: The server sends the formatted data to the generative AI, which uses prompts to guide the AI ​​in its proper analysis. (Input) Formatted data, prompt (e.g. "Generate promotional ideas for a new product") (Output) Generated ideas and materials (generated by generative artificial intelligence) (Specific operation) When the server sends a prompt sentence to the generative artificial intelligence, it uses an HTTP request and calls the API.

[0097] Step 4: Generative artificial intelligence analyzes user input based on prompt text and generates relevant information and materials. (Input) Prompt statement, user input data (Output) Support materials for generated presentations and moderation of discussions (Specific operation) Generative AI analyzes the user's requests and generates related ideas and specific presentation materials.

[0098] Step 5: The server receives the data generated by the generative artificial intelligence and transmits it to the user's device. (Input) Generated supporting materials and moderation data (Output) Data sent to the user's terminal (Specific Operation) The server checks the format of the received data again and sends the data to the user's terminal.

[0099] Step 6: The terminal displays supporting materials and moderation data received from the server. (Input) Data sent from the server (Output) User-visible screen display (Specific Operation) The terminal displays the received data at a specified position on the screen, making it easy for the user to view.

[0100] Step 7: Users can conduct business meetings and discussions in the virtual space while referring to the supporting materials and moderation data displayed on their devices. (Input) Support materials and moderation data displayed on the terminal (Output) Use in user meetings and discussions (Specific operation) The user uses tools in the virtual space to use the generated materials and hold a discussion. For example, the user gives a presentation in a virtual conference room and exchanges opinions with other participants.

[0101] (Application example 1)

[0102] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0103] We aim to improve the user experience not only in social interactions and business meetings in virtual spaces, but also in virtual stores. In particular, it is required to provide appropriate product recommendations and purchasing support in real time when users have questions or suggestions about products. However, current systems lack support that can immediately respond to user requests, resulting in a decrease in user satisfaction and loss of purchasing opportunities.

[0104] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0105] In this invention, the server includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation in real time via generative AI, a means for presenting the support and the moderation, a means for generating at least one of product recommendations and purchasing support based on the user's questions and suggestions in the virtual environment, and a means for presenting the recommendations and support. This allows the user to obtain appropriate information in real time in the virtual space or virtual store, making it possible to smoothly select and purchase products.

[0106] A "virtual space" is a three-dimensional virtual space generated using computer technology, and is an environment in which users can participate interactively.

[0107] "Generative AI" is an artificial intelligence technology that analyzes user input and generates appropriate responses and support information in real time.

[0108] "Presentation support" is a function that provides users with assistance with creating materials, organizing key points, and giving advice on how to speak when giving a presentation.

[0109] "Discussion moderation" is a function that organizes topics, promotes dialogue, provides feedback on ideas, and more, to ensure that discussions between users proceed smoothly.

[0110] "Product recommendation" is a function that selects and suggests the most suitable products based on the user's interests and requests.

[0111] "Purchase support" is a function that provides the user with the necessary information and procedural assistance when selecting and purchasing a product.

[0112] The "server" is a central processing unit that receives user input, transmits it to the generative AI, and transmits the generated information to the terminal.

[0113] A "terminal" is a device on which a user inputs information and on which information provided by generative AI is displayed. Examples of such devices include smartphones, smart glasses, and head-mounted displays.

[0114] "User input" refers to information such as a self-introduction, a suggestion, an opinion, a request, a question, or a request to a conversation partner that a user gives to the system.

[0115] This invention is a system for smoothly conducting social and business meetings in a virtual space and improving the user experience in a virtual store. The system mainly consists of a server, a terminal, a generative AI, and a virtual space, and operates as follows.

[0116] server

[0117] The server plays a central role in receiving user input and sending it to the generative AI. Specifically, it analyzes the information entered by the user through the terminal in real time and transfers it to the generative AI. It also sends support, moderation, product recommendations, and purchasing support generated by the generative AI to the terminal.

[0118] Terminal

[0119] A terminal is a device that allows users to input and display information. This includes smartphones, smart glasses, and head-mounted displays. Through the terminal, users input suggestions, questions, requests, etc., and receive and display responses from generative AI.

[0120] Generative AI

[0121] The generative AI analyzes user input and generates at least one of the following in real time: presentation support, discussion moderation, product recommendations, and purchasing support. This generated information is sent to the terminal via the server and provided to the user.

[0122] Virtual Space

[0123] A virtual world is a three-dimensional virtual space in which users can participate interactively, where they can socialize, hold business meetings, and enjoy shopping in virtual stores.

[0124] Specific examples of hardware and software used

[0125] Hardware: Servers, smartphones, smart glasses, head-mounted displays

[0126] Software: OpenAI® generative AI models (e.g. GPT® models), virtual environment software

[0127] Examples

[0128] Example 1: Use in business meetings

[0129] When a user holds a business meeting in a virtual space and proposes a new project, the generative AI provides real-time advice on the proposal. For example, if the user asks, "How can I make my presentation materials more visually appealing?" the generative AI will provide specific advice such as, "Add a graph and use color to indicate the points you want to emphasize."

[0130] Example 2: Use in a virtual store

[0131] When a user asks "What is a recommended gift for Father's Day?" through smart glasses in a virtual store, generative AI will generate a list of related products and suggest them to the user. For example, it will make a recommendation such as "This luxury watch has an elegant design and is popular with men."

[0132] Examples of prompt statements

[0133] Prompt: "Virtual storefront question: 'What would you recommend as a gift for Father's Day?' Generate an answer."

[0134] This allows the system to provide users with the most appropriate information in real time in the virtual space or virtual store, and to respond quickly to questions and suggestions.

[0135] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0136] Step 1: The user inputs information through the terminal. Specifically, a user uses a smartphone or smart glasses to input information such as self-introduction, suggestions, questions, requests, etc. in text or voice format. The input information is sent from the terminal to the server. The input data is in text format, and the output is the received data on the server side.

[0137] Step 2: The server receives the user's input and prepares it to be sent to the generative AI. Specifically, the server analyzes the input data it receives and formats it to send an appropriate prompt to the generative AI. Data processing involves analyzing the user's input, extracting the necessary information, and converting it into a prompt. As an output, a prompt is generated in a format that the generative AI can understand.

[0138] Step 3: The server sends the generated prompt to the generation AI. Specifically, the server sends the generated prompt text to the generative AI (e.g., GPT (registered trademark) model) via API. This causes the generative AI to start processing according to the user's input. The input data is the prompt text, and the response content is obtained from the generative AI as output.

[0139] Step 4: A generative AI analyzes the prompt and generates an appropriate response in real time. Specifically, generative AI performs natural language processing based on the prompt sentence and generates responses such as appropriate presentation support, discussion moderation, product recommendations, and purchasing support. Data calculation involves analysis and generation using a multi-layer neural network. The output data is the response content.

[0140] Step 5: The server receives the response from the generative AI and sends it to the terminal. Specifically, the server formats the response received from the generative AI in a format that is easy for the user to understand. After that, a communication process is performed to send it to the terminal. The input data is the response from the generative AI, and the output data is the data that is ready to be sent to the terminal.

[0141] Step 6: The terminal receives the response from the server and displays it to the user. Specifically, the terminal displays the response received from the server on the screen. The user can proceed with their activities in the virtual space based on the displayed information. Data processing involves converting the received data into a display format. The response information that can be viewed by the user is obtained as the output.

[0142] Step 7: The user takes action based on the generated response. Specifically, the user refers to the received information to advance a presentation or discussion in the virtual space, select a product in a virtual store, and complete a purchase procedure. The input data is the displayed response content, and the specific user action occurs as the output.

[0143] Furthermore, an emotion engine that estimates the emotion of the user may be combined. That is, the identification processing unit 290 may estimate the emotion of the user using the emotion identification model 59, and perform identification processing using the emotion of the user.

[0144] The embodiment for carrying out the present invention comprises the following elements.

[0145] 1. Server: Receives user input and processes it using a combination of generative AI and an emotion engine. The emotion engine recognizes the user's emotions and provides appropriate support and moderation based on them.

[0146] 2. Terminal: Displays the information entered by the user and the emotional information received from the emotion engine, and displays support and moderation received from the generative AI. It also detects the user's dialogue and behavior and transmits them to the server.

[0147] 3. Generative AI: Analyzes user input and generates presentation support and discussion moderation in real time, taking into account the emotional information received from the emotion engine to provide more appropriate support and moderation.

[0148] 4. Emotion Engine: An engine that combines technologies to recognize user emotions, using facial recognition, voice analysis, natural language processing, etc.

[0149] 5. Virtual space: A virtual space for users to participate in, where events such as workshops and seminars are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0150] [Specific example]

[0151] When a user wishes to hold a business meeting in a virtual space, the invention can be implemented in the following manner.

[0152] 1. The user enters a self-introduction, suggestions, requests, etc., and sends them to the server via the terminal.

[0153] 2. The server receives user input and processes it using a combination of generative AI and an emotion engine. The emotion engine recognizes the user's emotions and provides appropriate support or moderation based on them.

[0154] 3. Generative AI analyzes user input and generates presentation support and discussion moderation in real time, taking into account the emotional information received from the emotion engine to provide more appropriate support and moderation.

[0155] 4. The generated support and moderation are sent to the terminal via the server.

[0156] 5. The device displays the support and moderation received and provides it to the user. It also detects the user's interactions and actions and transmits them to the server.

[0157] 6. Users can conduct business meetings in the virtual space while referring to the displayed support and moderation.

[0158] In this manner, users can smoothly conduct social and business meetings in virtual space with support that takes into account emotions. Users can also participate in workshops and seminars in virtual space and deepen their interactions with other participants.

[0159] The process flow will be explained below.

[0160] Step 1: The user inputs a self-introduction, proposal, request, etc., and sends it to the server via the terminal.

[0161] Step 2: The server receives the user's input and processes it to recognize the user's emotion using the emotion engine.

[0162] Step 3: The emotion engine combines technologies such as facial recognition, voice analysis, and natural language processing to recognize the user's emotions and send the results to the server.

[0163] Step 4: The server sends user input and emotion information to the generative AI to generate presentation support and discussion moderation in real time.

[0164] Step 5: The generative AI analyzes the user's input and emotional information and generates appropriate support or moderation.

[0165] Step 6: The generated support and moderation are sent to the terminal via the server.

[0166] Step 7: The device displays the support and moderation received and provides it to the user.

[0167] Step 8: The user conducts a business meeting in the virtual space while referring to the displayed support and moderation.

[0168] Step 9: User interactions and actions are detected on the device and sent to the server.

[0169] Step 10: The server sends the user's interactions and actions to the generative AI for updates.

[0170] Step 11: Updated support and moderation is sent to the device via the server.

[0171] Step 12: The terminal displays the received update information and provides it to the user.

[0172] Step 13: The user continues the business meeting in the virtual space while referring to the displayed updated information.

[0173] Example 2

[0174] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart device 14 is referred to as a "terminal."

[0175] Conventional social networking services and business meeting tools provide information without considering the user's emotional state, making it difficult to provide appropriate support according to the user's emotions. As a result, users end up engaging in activities in virtual space while feeling tension or anxiety, which leads to problems such as reduced performance and satisfaction.

[0176] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation in real time via a generative AI and an emotion engine, a means for presenting at least one of the support and the moderation, a means for providing support based on the user's emotion information using an emotion engine that analyzes the user's emotion, a means for transmitting the generated support information and moderation to the user's terminal and displaying them on the user's terminal, and a means for the user to interact in the virtual space. This makes it possible to provide appropriate support according to the user's emotion and to smoothly carry out activities in the virtual space.

[0177] "User input" refers to information that a user of the system inputs via a terminal, and includes at least one of a self-introduction, a suggestion, an opinion, a request, and a request to the other party in the conversation.

[0178] "Generative AI" is artificial intelligence that analyzes user input and generates presentation support and discussion moderation in real time.

[0179] An "emotion engine" is a system that combines technologies such as facial recognition, voice analysis, and natural language processing to recognize a user's emotions and provide information about those emotions.

[0180] "Presentation support" is a function that provides information and materials necessary for a user to give a presentation in a virtual space.

[0181] "Discussion moderation" is a function that provides appropriate instructions and adjustments to facilitate smooth discussions and dialogue between users.

[0182] A "virtual space" is a virtual space on the Internet in which users can participate, and where events such as workshops, seminars, and business meetings are held.

[0183] A "terminal" is a device through which a user enters information and views the offered support and moderation, such as a computer or smartphone.

[0184] The "server" is a computer system that receives information sent by users, analyzes it, generates the necessary support information and moderation via a generative AI and emotion engine, and sends it to the user's device.

[0185] "Emotion information" is data about the user's emotional state that the emotion engine obtains through facial recognition, voice analysis, natural language processing, and the like.

[0186] The system for implementing the present invention is composed of the following elements: a server, a terminal, a generative AI, an emotion engine, and a virtual space.

[0187] server

[0188] The server has the function of receiving user input and generating at least one of presentation support and discussion moderation in real time via generative AI and an emotion engine. It also generates support information that takes into account the user's emotion information based on the analysis results of the emotion engine. The server is responsible for receiving and analyzing the information sent by the user, and sending the generated support information and moderation to the user's device. A secure communication protocol (e.g., HTTPS) is used in this process.

[0189] Terminal

[0190] The terminal is a device used by the user to input information and check support information and moderation. The user inputs information through the terminal, and the information is securely transmitted to the server. The terminal displays the support information and moderation transmitted from the server in real time and provides it to the user. It also includes a function to detect the user's interactions and actions and transmit such information to the server.

[0191] Generative AI

[0192] Generative AI analyzes user input and generates presentation support and discussion moderation in real time. Generative AI uses technologies such as natural language processing, speech recognition, and image analysis. In this process, it takes into account emotional information, such as when the user is nervous, to provide appropriate support.

[0193] Emotion Engine

[0194] The emotion engine uses facial recognition, voice analysis, and natural language processing to recognize the user's emotions. For example, it recognizes emotions such as joy, anger, and sadness from the user's facial expressions and tone of voice, and provides that emotional information to the generative AI.

[0195] Virtual Space

[0196] A virtual space is a virtual space on the Internet for users to participate in, where events such as workshops, seminars, business meetings, etc. are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0197] Examples

[0198] When a user wants to hold a business meeting in a virtual space, the system works as follows.

[0199] 1. The user types "I would like to give a presentation on our new product" through the terminal. 2. The server receives the information and passes it to the generative AI, while at the same time analyzing the user's emotional information using the emotion engine. 3. The emotion engine recognizes that the user is a little nervous and provides that information to the generative AI. 4. Generative AI will generate support information tailored to the user, such as slides that highlight the main points of a presentation or advice on how to relax. 5. The server sends the generated support information to the user's terminal. 6. The device displays the received support information and moderation to the user. 7. The user uses the received information to hold a business meeting in the virtual space.

[0200] Examples of prompt statements

[0201] "Please provide support information for a user to give a presentation on a new product. The user seems a bit nervous. Please generate appropriate advice based on the following information: Key points from the presentation Tips for relaxing

[0202] With the above configuration, the user can smoothly carry out activities in the virtual space while receiving appropriate support that takes into account the user's emotions.

[0203] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0204] Processing Steps

[0205] Step 1: The user enters information into the device The user inputs information such as self-introduction, proposals, requests, etc. via the terminal. This input includes, for example, the text "I would like to give a presentation on a new product." After inputting, a send button to proceed to the next step is displayed on the terminal. Input: User's text information (e.g. "I would like to give a presentation on a new product") Output: The encoded input data (e.g., data in JSON format).

[0206] Step 2: The device sends the input information to the server The terminal transmits the information input by the user to the server. This transmission is performed using a secure protocol (e.g. HTTPS). Input: The encoded input data Output: Data sent to the server

[0207] Step 3: The server receives the input information and analyzes it with the emotion engine The server passes the received user input to the emotion engine, which analyzes the user's emotions using facial recognition, voice analysis, and natural language processing to determine the user's emotional state. Input: Data sent to the server (e.g. user text and corresponding metadata) Data processing / computation: Emotion analysis (e.g., facial recognition, voice analysis, and assessment of emotional state through natural language processing) Output: Data about the user's emotional state (e.g., "The user is nervous")

[0208] Step 4: The server passes the analysis results to the generative AI The server passes the user's input information to the generative AI along with the analysis results obtained from the emotion engine. Input: Data about the user's emotional state, user input data Output: Data passed to the generative AI

[0209] Step 5: Generative AI generates supporting information and moderation The generative AI generates appropriate support information and discussion moderation based on the input information and emotional information received from the server. By taking emotional information into account, advice and materials appropriate for the user are created. Input: Data passed to the generative AI Data processing / computation: Generation of supporting information and moderation (e.g., natural language processing, data analysis) Output: Generated supporting information and moderation (e.g. slides to emphasize key points in the presentation, advice for relaxation)

[0210] Step 6: The server sends the generated results to the terminal. The server transmits the support information and moderation generated by the generative AI to the terminal. This transmission is also done using a secure communication protocol. Input: Generated supporting information and moderation Output: Data sent to the terminal

[0211] Step 7: The terminal displays the results to the user The device displays the received support information and moderation to the user, for example by displaying presentation slides or relaxation techniques on the screen. Input: Data sent to the terminal Output: Support information and moderation displayed to users

[0212] Step 8: Users use the results to take action in the virtual space Users can conduct business meetings and other activities in the virtual space by referring to the support information and moderation displayed on the terminal. Specifically, users can hold discussions with other participants in the virtual space and give presentations. Input: Support information and moderation displayed on the terminal Output: Activity in the virtual space (e.g., a successful presentation, a good discussion)

[0213] As described above, through the specific operations and data flows performed at each processing step, support is provided to enable the user to move smoothly in the virtual space.

[0214] (Application example 2)

[0215] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart device 14 will be referred to as a "terminal."

[0216] Conventional user experiences in social interactions, business meetings, and virtual stores in virtual spaces lacked support and responses based on the user's emotions, which led to insufficient customer satisfaction. In addition, real-time support based on the user's input was rarely provided, which often hindered efficient communication and shopping experiences. In particular, due to a lack of technology that properly recognized the user's emotions and provided support based on them, the user's experience tended to be one-way.

[0217] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server, in an application that enables socializing and business meetings in a virtual space, as well as shopping experiences in a virtual store, includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation, and in-store product guide in real time via a generative AI, a means including an emotion engine that recognizes the user's emotions and provides appropriate responses and support based on the emotions, and a means for presenting the support, the moderation, and at least one of the responses. This allows the user to receive appropriate support and moderation that take emotions into account in real time, making it possible to facilitate communication and improve customer satisfaction.

[0218] "Virtual space" refers to a three-dimensional virtual environment constructed using digital technology that enables real-time interaction.

[0219] "Socializing" refers to activities in which people communicate with each other and build social networks.

[0220] "Business meeting" refers to a conference or meeting held for business purposes.

[0221] A "virtual store" refers to a digital store that operates on the Internet or in a virtual space.

[0222] "Generative AI" refers to artificial intelligence that analyzes user input and generates appropriate responses and support in real time based on that input.

[0223] An "emotion engine" refers to a system that combines technologies to analyze a user's facial expressions, voice, and input content and recognize the user's emotions.

[0224] "Presentation support" refers to providing information, materials, and advice in real time to assist a user in making a presentation.

[0225] "Discussion moderation" refers to intervening and adjusting at the appropriate time to ensure that discussions and dialogue between users proceed smoothly.

[0226] "Product Introduction" refers to providing information about products and services within the Virtual Store.

[0227] "User input" refers to various information provided and operations performed by a user on the system.

[0228] "Response" refers to the reply or action that the system generates based on the user's input and emotions.

[0229] "Support" refers to information and advice that the system provides to assist the user in their activities.

[0230] This invention is an advanced system for supporting users' social interactions, business meetings, and shopping experiences in virtual stores in a virtual space. The system is realized mainly using a server, a terminal, a generative AI, an emotion engine, and a virtual space.

[0231] System Configuration

[0232] 1. Server The server receives user input and performs processing that combines generative AI and an emotion engine. Specifically, the server analyzes the input data sent by the user and generates appropriate support and responses in real time. The server uses cloud infrastructure such as Azure Cloud Services (registered trademark).

[0233] 2. Terminal The terminal refers to a device used by a user, and includes a smartphone, smart glasses, a head-mounted display, etc. The terminal collects user input and transmits the data to a server. It also presents support and responses sent from the server to the user. For example, Microsoft® HoloLens® 2 is used as a smart glass.

[0234] 3. Generative AI The generative AI analyzes the user's input and takes into account the emotional information obtained from the emotion engine to generate presentation support, discussion moderation, and / or product guides in the virtual store. ChatGPT (registered trademark) from OpenAI (registered trademark) is used for the generative AI.

[0235] 4. Emotion Engine The emotion engine uses technology to recognize emotions from the user's facial expressions, tone of voice, input text, etc. This uses the Affectiva SDK (registered trademark). The emotion engine sends the user's emotional information to the server, and the generative AI generates an appropriate response based on that information.

[0236] 5. Virtual Space A virtual world is a three-dimensional virtual environment that users can access to socialize, hold business meetings, or shop. The virtual world is built using Unity® 3D.

[0237] Processing flow

[0238] User Behavior When a user accesses a virtual store through smart glasses and browses products, their facial expressions and voices are collected by the device. For example, if a user is interested in a particular product, that behavioral data is sent to the server.

[0239] Analyzing data and generating responses The server analyzes the received data using a generative AI (ChatGPT®) and an emotion engine (Affectiva SDK®). Based on the user's emotions and input, the generative AI generates useful information such as detailed descriptions of products the user is interested in and other customer reviews.

[0240] Feedback to users The generated responses and supporting information are sent to the terminal in real time, and the user can check the information on the screen of the smart glasses. For example, a message such as "This product looks great on you. It has been highly rated by other customers" may be displayed.

[0241] Examples

[0242] As a specific use case, consider a scenario in which a user is choosing a dress in a virtual fashion store. As the user tries on the chosen dress, the emotion engine judges the user's satisfaction to be high based on the user's facial expressions and voice. Based on this information, the generative AI generates a support message such as "This dress is perfect for you and has been well received by many customers," and displays it on the user's smart glasses.

[0243] Example prompt:

[0244] When the user is choosing tableware in the virtual store, he / she looks a little confused. As a generative AI, please generate a support message that is appropriate for this situation.

[0245] As described above, the system provides emotion-based real-time support to enhance social interactions, business meetings, and shopping experiences in virtual spaces.

[0246] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0247] Step 1: A user accesses a virtual store in a virtual space through a device such as smart glasses. A 3D model of the virtual store is displayed on the user's interface. The user focuses on a specific product and selects it. At this stage, the device collects the user's facial expression data and voice data in real time. Input: User's gaze, facial expression data, voice data Output: 3D model display of virtual space, selected product data

[0248] Step 2: The device transmits the collected data of the user's facial expression and voice to the server. The transmitted data includes the user's facial features, tone of voice, selected product ID, etc. Input: User facial expression data, voice data, product ID from the device Output: Data packet sent to the server

[0249] Step 3: The server analyzes the received data. First, the emotion engine (Affectiva SDK®) analyzes facial features and tone of voice to recognize the user's emotional state (e.g., interest, confusion, satisfaction, etc.). Input: User's facial expression data, voice data Output: User's emotional state data

[0250] Step 4: The server passes the emotional state data and the selected product ID as input to a generative AI (ChatGPT (registered trademark)). The generative AI generates an appropriate support message or response based on the user's emotional state and the product information. Input: User's emotional state data, product ID Output: The generated support message or response.

[0251] Step 5: The generated support messages and responses are sent from the server to the terminal and displayed in real time on the user's smart glasses, for example, "This product looks great on you. Other customers have given it high marks." Input: Generated support message or response Output: Message displayed on the smart glasses

[0252] Step 6: If the user takes any further action (such as asking a question about a product or making a purchase), that information is also sent from the device to the server and is again analyzed by the generative AI and emotion engine, which then generates the next appropriate support or response in real time. Input: User's additional action data Output: The following support message or response:

[0253] Through these steps, the system can provide appropriate support and responses in real time based on the user's emotions, improving the social and shopping experience in the virtual space.

[0254] The specific processing unit 290 transmits the result of the specific processing to the smart device 14. In the smart device 14, the control unit 46A causes the output device 40 to output the result of the specific processing. The microphone 38B acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 38B to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0255] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0256] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the smart device 14.

[0257] [Second embodiment]

[0258] FIG. 3 shows an example of the configuration of a data processing system 210 according to the second embodiment.

[0259] 3, the data processing system 210 includes the data processing device 12 and smart glasses 214. An example of the data processing device 12 is a server.

[0260] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0261] The smart glasses 214 include a computer 36, a microphone 238, a speaker 240, a camera 42, and a communication I / F 44. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, and the camera 42 are also connected to the bus 52.

[0262] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0263] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0264] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0265] Fig. 4 shows an example of main functions of the data processing device 12 and the smart glasses 214. As shown in Fig. 4, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0266] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0267] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0268] In the smart glasses 214, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0269] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0270] The embodiment for carrying out the present invention comprises the following elements.

[0271] 1. Server: Receives user input and processes it to send it to the generative AI. It also receives support and moderation generated by the generative AI and processes it to send it to the terminal.

[0272] 2. Terminal: Displays the information entered by the user, and displays support and moderation received from the generative AI. It also detects the user's dialogue and actions and transmits them to the server.

[0273] 3. Generative AI: Analyzes user input and generates presentation support and discussion moderation in real time. It also takes into account the user's emotions to provide appropriate support and moderation.

[0274] 4. Virtual space: A virtual space for users to participate in, where events such as workshops and seminars are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0275] [Specific example]

[0276] When a user wishes to hold a business meeting in a virtual space, the invention can be implemented in the following manner.

[0277] 1. The user enters a self-introduction, suggestions, requests, etc., and sends them to the server via the terminal.

[0278] 2. The server receives the user's input and sends it to the generative AI.

[0279] 3. Generative AI analyses user input and generates presentation support and discussion moderation in real time.

[0280] 4. The generated support and moderation are sent to the terminal via the server.

[0281] 5. The device displays the support and moderation received and provides it to the user.

[0282] 6. Users can conduct business meetings in the virtual space while referring to the displayed support and moderation.

[0283] In this manner, users can smoothly conduct social gatherings and business meetings in the virtual space. Users can also participate in workshops and seminars in the virtual space and deepen their interactions with other participants.

[0284] The process flow will be explained below.

[0285] Step 1: The user inputs a self-introduction, proposal, request, etc., and sends it to the server via the terminal.

[0286] Step 2: The server receives the user's input and sends it to the generative AI.

[0287] Step 3: The generative AI analyzes user input and generates presentation support and discussion moderation in real time.

[0288] Step 4: The support and moderation generated by the generative AI are sent to the device via the server.

[0289] Step 5: The device displays the support and moderation received and provides it to the user.

[0290] Step 6: The user conducts a business meeting in the virtual space while referring to the displayed support and moderation.

[0291] Step 7: User interactions and actions are detected on the device and sent to the server.

[0292] Step 8: The server sends the user's interactions and actions to the generative AI, which updates them in real time.

[0293] Step 9: Updated support and moderation is sent to the device via the server.

[0294] Step 10: The terminal displays the received update information and provides it to the user.

[0295] Step 11: The user continues the business meeting in the virtual space while referring to the displayed updated information.

[0296] Example 1

[0297] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0298] In conventional social and business meetings in virtual spaces, users often lacked support to effectively present and discuss in real time. As a result, the quality and efficiency of meetings declined, and the user experience was impaired. In addition, it was difficult to properly analyze the user's input information and provide support and moderation that took the user's emotions into account. This meant that users could not receive appropriate feedback or support, making smooth communication difficult.

[0299] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0300] In this invention, the server includes means for receiving a user's input and generating at least one of a presentation support and a discussion moderation in real time via a generative artificial intelligence, means for transmitting at least one of the support and the moderation to the user's terminal, and means for displaying at least one of the support and the moderation on the user's terminal, thereby enabling the user to effectively socialize or hold a business meeting in a virtual space in real time. In addition, the generative artificial intelligence analyzes the user's input and provides appropriate support and moderation taking into account the user's emotions, improving the user's experience and making communication proceed smoothly.

[0301] "User input" refers to the input of a user's self-introduction, suggestions, opinions, requests, and requests to the other party through a terminal.

[0302] A "generative artificial intelligence" is an artificial intelligence model that analyzes user input and generates presentation support and / or discussion moderation in real time.

[0303] "Generating in real time" means selecting, generating, and outputting presentation support and discussion moderation almost instantly after receiving user input.

[0304] "Presentation support" refers to providing information and materials that enable users to give presentations effectively and smoothly.

[0305] "Discussion moderation" refers to providing advice, questions, feedback, etc. to facilitate discussions among users.

[0306] "Sending to the user's device" means transferring the generated support and moderation content as data from the server to a device such as a PC or smartphone used by the user.

[0307] "Displaying on the user's device" means presenting the submitted support and moderation content on the screen of the device (PC, smartphone, etc.) used by the user.

[0308] "Virtual social and business meetings" refers to activities in which users communicate with other users in a virtual environment created on the Internet, including business meetings and social gatherings.

[0309] The present invention relates to a system for smoothly conducting social interactions and business meetings in a virtual space. The configuration and operation of this system will be described in detail below.

[0310] System Configuration

[0311] The system consists of the following main elements:

[0312] 1. Server

[0313] The server uses a cloud server equipped with a high-performance CPU and memory (e.g. AWS (registered trademark) EC2 (registered trademark)). It receives user input and processes it to send it to the generative AI. It also receives support and moderation generated by the generative AI and processes it to send it to the terminal.

[0314] 2. Terminal

[0315] This refers to the PC or smartphone used by the user (e.g. Windows® PC, iPhone®, Android® device). The terminal displays the user's input and displays support and moderation received from the generative AI. It also detects the user's dialogue and actions and processes them to send them to the server.

[0316] 3. Generative Artificial Intelligence

[0317] Use specialized artificial intelligence tools (e.g., OpenAI ChatGPT, Google BERT). Analyze user input and generate presentation support and discussion moderation in real time. Additionally, provide appropriate support and moderation by taking into account user sentiment.

[0318] 4. Virtual Space

[0319] Use an online event platform (e.g. VRChat (registered trademark), Zoom (registered trademark)). This is a place where users can participate and events such as workshops and seminars are held.

[0320] Example of system operation

[0321] A specific example of operation is shown below.

[0322] 1. User Input The user types into the terminal, "I would like to share my ideas for promoting a new product."

[0323] 2. Server Processing The server receives user input and performs formatting and validation of the data. Send the generative artificial intelligence a prompt: "Generate promotional ideas for a new product."

[0324] 3. Analysis and Generation of Generative Artificial Intelligence Generative artificial intelligence analyses user input and generates relevant ideas and presentation materials in real time. Example prompt: "Generate promotional ideas for a new product."

[0325] 4. Sending the generated results The server transmits the generated ideas and materials to the user's terminal.

[0326] 5. Display of terminal The user's terminal displays the generated promotion ideas and materials on the screen. Users can refer to the displayed information and proceed with the meeting in the virtual space.

[0327] How to use

[0328] When a user holds a business meeting in a virtual space, he or she uses this system in the following steps.

[0329] 1. The user inputs a self-introduction, suggestions, requests, etc. through the terminal. 2. The server receives the user's input and sends it to the generative artificial intelligence. 3. Generative AI analyses user input and generates the necessary support and moderation in real time. 4. The generated support and moderation are sent to the user's device via the server. 5. The device displays the support and moderation received and provides it to the user. 6. The user proceeds with the meeting in the virtual space while referring to the displayed support and moderation.

[0330] This allows users to smoothly conduct social and business meetings in a virtual space and to communicate efficiently with other participants.

[0331] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0332] Program processing steps and detailed explanation

[0333] Step 1: Users use the terminal to input information such as self-introductions, suggestions, and requests. (Input) Text information entered by the user (e.g., "I would like to share my ideas for promoting new products") (Output) User input data that the terminal sends to the server (Specific operation) The user enters an idea into the text box on the terminal and clicks the send button.

[0334] Step 2: The server receives the data input by the user and performs initial processing, checking the format and content of the data and converting it into a format for sending to the generative artificial intelligence. (Input) User input data (Output) Formatted data (format for sending to generative artificial intelligence) (Specific operation) The server checks the string and unifies the format, and also handles errors if necessary.

[0335] Step 3: The server sends the formatted data to the generative AI, which uses prompts to guide the AI ​​in its proper analysis. (Input) Formatted data, prompt (e.g. "Generate promotional ideas for a new product") (Output) Generated ideas and materials (generated by generative artificial intelligence) (Specific operation) When the server sends a prompt sentence to the generative artificial intelligence, it uses an HTTP request and calls the API.

[0336] Step 4: Generative artificial intelligence analyzes user input based on prompt text and generates relevant information and materials. (Input) Prompt statement, user input data (Output) Support materials for generated presentations and moderation of discussions (Specific operation) Generative AI analyzes the user's requests and generates related ideas and specific presentation materials.

[0337] Step 5: The server receives the data generated by the generative artificial intelligence and transmits it to the user's device. (Input) Generated supporting materials and moderation data (Output) Data sent to the user's terminal (Specific Operation) The server checks the format of the received data again and sends the data to the user's terminal.

[0338] Step 6: The terminal displays supporting materials and moderation data received from the server. (Input) Data sent from the server (Output) User-visible screen display (Specific Operation) The terminal displays the received data at a specified position on the screen, making it easy for the user to view.

[0339] Step 7: Users can conduct business meetings and discussions in the virtual space while referring to the supporting materials and moderation data displayed on their devices. (Input) Support materials and moderation data displayed on the terminal (Output) Use in user meetings and discussions (Specific operation) The user uses tools in the virtual space to use the generated materials and hold a discussion. For example, the user gives a presentation in a virtual conference room and exchanges opinions with other participants.

[0340] (Application example 1)

[0341] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0342] We aim to improve the user experience not only in social interactions and business meetings in virtual spaces, but also in virtual stores. In particular, it is required to provide appropriate product recommendations and purchasing support in real time when users have questions or suggestions about products. However, current systems lack support that can immediately respond to user requests, resulting in a decrease in user satisfaction and loss of purchasing opportunities.

[0343] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0344] In this invention, the server includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation in real time via generative AI, a means for presenting the support and the moderation, a means for generating at least one of product recommendations and purchasing support based on the user's questions and suggestions in the virtual environment, and a means for presenting the recommendations and support. This allows the user to obtain appropriate information in real time in the virtual space or virtual store, making it possible to smoothly select and purchase products.

[0345] A "virtual space" is a three-dimensional virtual space generated using computer technology, and is an environment in which users can participate interactively.

[0346] "Generative AI" is an artificial intelligence technology that analyzes user input and generates appropriate responses and support information in real time.

[0347] "Presentation support" is a function that provides users with assistance with creating materials, organizing key points, and giving advice on how to speak when giving a presentation.

[0348] "Discussion moderation" is a function that organizes topics, promotes dialogue, provides feedback on ideas, and more, to ensure that discussions between users proceed smoothly.

[0349] "Product recommendation" is a function that selects and suggests the most suitable products based on the user's interests and requests.

[0350] "Purchase support" is a function that provides the user with the necessary information and procedural assistance when selecting and purchasing a product.

[0351] The "server" is a central processing unit that receives user input, transmits it to the generative AI, and transmits the generated information to the terminal.

[0352] A "terminal" is a device on which a user inputs information and on which information provided by generative AI is displayed. Examples of such devices include smartphones, smart glasses, and head-mounted displays.

[0353] "User input" refers to information such as a self-introduction, a suggestion, an opinion, a request, a question, or a request to a conversation partner that a user gives to the system.

[0354] This invention is a system for smoothly conducting social and business meetings in a virtual space and improving the user experience in a virtual store. The system mainly consists of a server, a terminal, a generative AI, and a virtual space, and operates as follows.

[0355] server

[0356] The server plays a central role in receiving user input and sending it to the generative AI. Specifically, it analyzes the information entered by the user through the terminal in real time and transfers it to the generative AI. It also sends support, moderation, product recommendations, and purchasing support generated by the generative AI to the terminal.

[0357] Terminal

[0358] A terminal is a device that allows users to input and display information. This includes smartphones, smart glasses, and head-mounted displays. Through the terminal, users input suggestions, questions, requests, etc., and receive and display responses from generative AI.

[0359] Generative AI

[0360] The generative AI analyzes user input and generates at least one of the following in real time: presentation support, discussion moderation, product recommendations, and purchasing support. This generated information is sent to the terminal via the server and provided to the user.

[0361] Virtual Space

[0362] A virtual world is a three-dimensional virtual space in which users can participate interactively, where they can socialize, hold business meetings, and enjoy shopping in virtual stores.

[0363] Specific examples of hardware and software used

[0364] Hardware: Servers, smartphones, smart glasses, head-mounted displays

[0365] Software: OpenAI® generative AI models (e.g. GPT® models), virtual environment software

[0366] Examples

[0367] Example 1: Use in business meetings

[0368] When a user holds a business meeting in a virtual space and proposes a new project, the generative AI provides real-time advice on the proposal. For example, if the user asks, "How can I make my presentation materials more visually appealing?" the generative AI will provide specific advice such as, "Add a graph and use color to indicate the points you want to emphasize."

[0369] Example 2: Use in a virtual store

[0370] When a user asks "What is a recommended gift for Father's Day?" through smart glasses in a virtual store, generative AI will generate a list of related products and suggest them to the user. For example, it will make a recommendation such as "This luxury watch has an elegant design and is popular with men."

[0371] Examples of prompt statements

[0372] Prompt: "Virtual storefront question: 'What would you recommend as a gift for Father's Day?' Generate an answer."

[0373] This allows the system to provide users with the most appropriate information in real time in the virtual space or virtual store, and to respond quickly to questions and suggestions.

[0374] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0375] Step 1: The user inputs information through the terminal. Specifically, a user uses a smartphone or smart glasses to input information such as self-introduction, suggestions, questions, requests, etc. in text or voice format. The input information is sent from the terminal to the server. The input data is in text format, and the output is the received data on the server side.

[0376] Step 2: The server receives the user's input and prepares it to be sent to the generative AI. Specifically, the server analyzes the input data it receives and formats it to send an appropriate prompt to the generative AI. Data processing involves analyzing the user's input, extracting the necessary information, and converting it into a prompt. As an output, a prompt is generated in a format that the generative AI can understand.

[0377] Step 3: The server sends the generated prompt to the generation AI. Specifically, the server sends the generated prompt text to the generative AI (e.g., GPT (registered trademark) model) via API. This causes the generative AI to start processing according to the user's input. The input data is the prompt text, and the response content is obtained from the generative AI as output.

[0378] Step 4: A generative AI analyzes the prompt and generates an appropriate response in real time. Specifically, generative AI performs natural language processing based on the prompt sentence and generates responses such as appropriate presentation support, discussion moderation, product recommendations, and purchasing support. Data calculation involves analysis and generation using a multi-layer neural network. The output data is the response content.

[0379] Step 5: The server receives the response from the generative AI and sends it to the terminal. Specifically, the server formats the response received from the generative AI in a format that is easy for the user to understand. After that, a communication process is performed to send it to the terminal. The input data is the response from the generative AI, and the output data is the data that is ready to be sent to the terminal.

[0380] Step 6: The terminal receives the response from the server and displays it to the user. Specifically, the terminal displays the response received from the server on the screen. The user can proceed with their activities in the virtual space based on the displayed information. Data processing involves converting the received data into a display format. The response information that can be viewed by the user is obtained as the output.

[0381] Step 7: The user takes action based on the generated response. Specifically, the user refers to the received information to advance a presentation or discussion in the virtual space, select a product in a virtual store, and complete a purchase procedure. The input data is the displayed response content, and the specific user action occurs as the output.

[0382] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0383] The embodiment for carrying out the present invention comprises the following elements.

[0384] 1. Server: Receives user input and processes it using a combination of generative AI and an emotion engine. The emotion engine recognizes the user's emotions and provides appropriate support and moderation based on them.

[0385] 2. Terminal: Displays the information entered by the user and the emotional information received from the emotion engine, and displays support and moderation received from the generative AI. It also detects the user's dialogue and behavior and transmits them to the server.

[0386] 3. Generative AI: Analyzes user input and generates presentation support and discussion moderation in real time, taking into account the emotional information received from the emotion engine to provide more appropriate support and moderation.

[0387] 4. Emotion Engine: An engine that combines technologies to recognize user emotions, using facial recognition, voice analysis, natural language processing, etc.

[0388] 5. Virtual space: A virtual space for users to participate in, where events such as workshops and seminars are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0389] [Specific example]

[0390] When a user wishes to hold a business meeting in a virtual space, the invention can be implemented in the following manner.

[0391] 1. The user enters a self-introduction, suggestions, requests, etc., and sends them to the server via the terminal.

[0392] 2. The server receives user input and processes it using a combination of generative AI and an emotion engine. The emotion engine recognizes the user's emotions and provides appropriate support or moderation based on them.

[0393] 3. Generative AI analyzes user input and generates presentation support and discussion moderation in real time, taking into account the emotional information received from the emotion engine to provide more appropriate support and moderation.

[0394] 4. The generated support and moderation are sent to the terminal via the server.

[0395] 5. The device displays the support and moderation received and provides it to the user. It also detects the user's interactions and actions and transmits them to the server.

[0396] 6. Users can conduct business meetings in the virtual space while referring to the displayed support and moderation.

[0397] In this manner, users can smoothly conduct social and business meetings in virtual space with support that takes into account emotions. Users can also participate in workshops and seminars in virtual space and deepen their interactions with other participants.

[0398] The process flow will be explained below.

[0399] Step 1: The user inputs a self-introduction, proposal, request, etc., and sends it to the server via the terminal.

[0400] Step 2: The server receives the user's input and processes it to recognize the user's emotion using the emotion engine.

[0401] Step 3: The emotion engine combines technologies such as facial recognition, voice analysis, and natural language processing to recognize the user's emotions and send the results to the server.

[0402] Step 4: The server sends user input and emotion information to the generative AI to generate presentation support and discussion moderation in real time.

[0403] Step 5: The generative AI analyzes the user's input and emotional information and generates appropriate support or moderation.

[0404] Step 6: The generated support and moderation are sent to the terminal via the server.

[0405] Step 7: The device displays the support and moderation received and provides it to the user.

[0406] Step 8: The user conducts a business meeting in the virtual space while referring to the displayed support and moderation.

[0407] Step 9: User interactions and actions are detected on the device and sent to the server.

[0408] Step 10: The server sends the user's interactions and actions to the generative AI for updates.

[0409] Step 11: Updated support and moderation is sent to the device via the server.

[0410] Step 12: The terminal displays the received update information and provides it to the user.

[0411] Step 13: The user continues the business meeting in the virtual space while referring to the displayed updated information.

[0412] Example 2

[0413] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the smart glasses 214 are referred to as a "terminal".

[0414] Conventional social networking services and business meeting tools provide information without considering the user's emotional state, making it difficult to provide appropriate support according to the user's emotions. As a result, users end up engaging in activities in virtual space while feeling tension or anxiety, which leads to problems such as reduced performance and satisfaction.

[0415] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation in real time via a generative AI and an emotion engine, a means for presenting at least one of the support and the moderation, a means for providing support based on the user's emotion information using an emotion engine that analyzes the user's emotion, a means for transmitting the generated support information and moderation to the user's terminal and displaying them on the user's terminal, and a means for the user to interact in the virtual space. This makes it possible to provide appropriate support according to the user's emotion and to smoothly carry out activities in the virtual space.

[0416] "User input" refers to information that a user of the system inputs via a terminal, and includes at least one of a self-introduction, a suggestion, an opinion, a request, and a request to the other party in the conversation.

[0417] "Generative AI" is artificial intelligence that analyzes user input and generates presentation support and discussion moderation in real time.

[0418] An "emotion engine" is a system that combines technologies such as facial recognition, voice analysis, and natural language processing to recognize a user's emotions and provide information about those emotions.

[0419] "Presentation support" is a function that provides information and materials necessary for a user to give a presentation in a virtual space.

[0420] "Discussion moderation" is a function that provides appropriate instructions and adjustments to facilitate smooth discussions and dialogue between users.

[0421] A "virtual space" is a virtual space on the Internet in which users can participate, and where events such as workshops, seminars, and business meetings are held.

[0422] A "terminal" is a device through which a user enters information and views the offered support and moderation, such as a computer or smartphone.

[0423] The "server" is a computer system that receives information sent by users, analyzes it, generates the necessary support information and moderation via a generative AI and emotion engine, and sends it to the user's device.

[0424] "Emotion information" is data about the user's emotional state that the emotion engine obtains through facial recognition, voice analysis, natural language processing, and the like.

[0425] The system for implementing the present invention is composed of the following elements: a server, a terminal, a generative AI, an emotion engine, and a virtual space.

[0426] server

[0427] The server has the function of receiving user input and generating at least one of presentation support and discussion moderation in real time via generative AI and an emotion engine. It also generates support information that takes into account the user's emotion information based on the analysis results of the emotion engine. The server is responsible for receiving and analyzing the information sent by the user, and sending the generated support information and moderation to the user's device. A secure communication protocol (e.g., HTTPS) is used in this process.

[0428] Terminal

[0429] The terminal is a device used by the user to input information and check support information and moderation. The user inputs information through the terminal, and the information is securely transmitted to the server. The terminal displays the support information and moderation transmitted from the server in real time and provides it to the user. It also includes a function to detect the user's interactions and actions and transmit such information to the server.

[0430] Generative AI

[0431] Generative AI analyzes user input and generates presentation support and discussion moderation in real time. Generative AI uses technologies such as natural language processing, speech recognition, and image analysis. In this process, it takes into account emotional information, such as when the user is nervous, to provide appropriate support.

[0432] Emotion Engine

[0433] The emotion engine uses facial recognition, voice analysis, and natural language processing to recognize the user's emotions. For example, it recognizes emotions such as joy, anger, and sadness from the user's facial expressions and tone of voice, and provides that emotional information to the generative AI.

[0434] Virtual Space

[0435] A virtual space is a virtual space on the Internet for users to participate in, where events such as workshops, seminars, business meetings, etc. are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0436] Examples

[0437] When a user wants to hold a business meeting in a virtual space, the system works as follows.

[0438] 1. The user types "I would like to give a presentation on our new product" through the terminal. 2. The server receives the information and passes it to the generative AI, while at the same time analyzing the user's emotional information using the emotion engine. 3. The emotion engine recognizes that the user is a little nervous and provides that information to the generative AI. 4. Generative AI will generate support information tailored to the user, such as slides that highlight the main points of a presentation or advice on how to relax. 5. The server sends the generated support information to the user's terminal. 6. The device displays the received support information and moderation to the user. 7. The user uses the received information to hold a business meeting in the virtual space.

[0439] Examples of prompt statements

[0440] "Please provide support information for a user to give a presentation on a new product. The user seems a bit nervous. Please generate appropriate advice based on the following information: Key points from the presentation Tips for relaxing

[0441] With the above configuration, the user can smoothly carry out activities in the virtual space while receiving appropriate support that takes into account the user's emotions.

[0442] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0443] Processing Steps

[0444] Step 1: The user enters information into the device The user inputs information such as self-introduction, proposals, requests, etc. via the terminal. This input includes, for example, the text "I would like to give a presentation on a new product." After inputting, a send button to proceed to the next step is displayed on the terminal. Input: User's text information (e.g. "I would like to give a presentation on a new product") Output: The encoded input data (e.g., data in JSON format).

[0445] Step 2: The device sends the input information to the server The terminal transmits the information input by the user to the server. This transmission is performed using a secure protocol (e.g. HTTPS). Input: The encoded input data Output: Data sent to the server

[0446] Step 3: The server receives the input information and analyzes it with the emotion engine The server passes the received user input to the emotion engine, which analyzes the user's emotions using facial recognition, voice analysis, and natural language processing to determine the user's emotional state. Input: Data sent to the server (e.g. user text and corresponding metadata) Data processing / computation: Emotion analysis (e.g., facial recognition, voice analysis, and assessment of emotional state through natural language processing) Output: Data about the user's emotional state (e.g., "The user is nervous")

[0447] Step 4: The server passes the analysis results to the generative AI The server passes the user's input information to the generative AI along with the analysis results obtained from the emotion engine. Input: Data about the user's emotional state, user input data Output: Data passed to the generative AI

[0448] Step 5: Generative AI generates supporting information and moderation The generative AI generates appropriate support information and discussion moderation based on the input information and emotional information received from the server. By taking emotional information into account, advice and materials appropriate for the user are created. Input: Data passed to the generative AI Data processing / computation: Generation of supporting information and moderation (e.g., natural language processing, data analysis) Output: Generated supporting information and moderation (e.g. slides to emphasize key points in the presentation, advice for relaxation)

[0449] Step 6: The server sends the generated results to the terminal. The server transmits the support information and moderation generated by the generative AI to the terminal. This transmission is also done using a secure communication protocol. Input: Generated supporting information and moderation Output: Data sent to the terminal

[0450] Step 7: The terminal displays the results to the user The device displays the received support information and moderation to the user, for example by displaying presentation slides or relaxation techniques on the screen. Input: Data sent to the terminal Output: Support information and moderation displayed to users

[0451] Step 8: Users use the results to take action in the virtual space Users can conduct business meetings and other activities in the virtual space by referring to the support information and moderation displayed on the terminal. Specifically, users can hold discussions with other participants in the virtual space and give presentations. Input: Support information and moderation displayed on the terminal Output: Activity in the virtual space (e.g., a successful presentation, a good discussion)

[0452] As described above, through the specific operations and data flows performed at each processing step, support is provided to enable the user to move smoothly in the virtual space.

[0453] (Application example 2)

[0454] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the smart glasses 214 will be referred to as a "terminal".

[0455] Conventional user experiences in social interactions, business meetings, and virtual stores in virtual spaces lacked support and responses based on the user's emotions, which led to insufficient customer satisfaction. In addition, real-time support based on the user's input was rarely provided, which often hindered efficient communication and shopping experiences. In particular, due to a lack of technology that properly recognized the user's emotions and provided support based on them, the user's experience tended to be one-way.

[0456] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server, in an application that enables socializing and business meetings in a virtual space, as well as shopping experiences in a virtual store, includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation, and in-store product guide in real time via a generative AI, a means including an emotion engine that recognizes the user's emotions and provides appropriate responses and support based on the emotions, and a means for presenting the support, the moderation, and at least one of the responses. This allows the user to receive appropriate support and moderation that take emotions into account in real time, making it possible to facilitate communication and improve customer satisfaction.

[0457] "Virtual space" refers to a three-dimensional virtual environment constructed using digital technology that enables real-time interaction.

[0458] "Socializing" refers to activities in which people communicate with each other and build social networks.

[0459] "Business meeting" refers to a conference or meeting held for business purposes.

[0460] A "virtual store" refers to a digital store that operates on the Internet or in a virtual space.

[0461] "Generative AI" refers to artificial intelligence that analyzes user input and generates appropriate responses and support in real time based on that input.

[0462] An "emotion engine" refers to a system that combines technologies to analyze a user's facial expressions, voice, and input content and recognize the user's emotions.

[0463] "Presentation support" refers to providing information, materials, and advice in real time to assist a user in making a presentation.

[0464] "Discussion moderation" refers to intervening and adjusting at the appropriate time to ensure that discussions and dialogue between users proceed smoothly.

[0465] "Product Introduction" refers to providing information about products and services within the Virtual Store.

[0466] "User input" refers to various information provided and operations performed by a user on the system.

[0467] "Response" refers to the reply or action that the system generates based on the user's input and emotions.

[0468] "Support" refers to information and advice that the system provides to assist the user in their activities.

[0469] This invention is an advanced system for supporting users' social interactions, business meetings, and shopping experiences in virtual stores in a virtual space. The system is realized mainly using a server, a terminal, a generative AI, an emotion engine, and a virtual space.

[0470] System Configuration

[0471] 1. Server The server receives user input and performs processing that combines generative AI and an emotion engine. Specifically, the server analyzes the input data sent by the user and generates appropriate support and responses in real time. The server uses cloud infrastructure such as Azure Cloud Services (registered trademark).

[0472] 2. Terminal The terminal refers to a device used by a user, and includes a smartphone, smart glasses, a head-mounted display, etc. The terminal collects user input and transmits the data to a server. It also presents support and responses transmitted from the server to the user. For example, Microsoft HoloLens 2 (registered trademark) is used as the smart glasses.

[0473] 3. Generative AI The generative AI analyzes the user's input and takes into account the emotional information obtained from the emotion engine to generate presentation support, discussion moderation, and / or product guides in the virtual store. ChatGPT (registered trademark) from OpenAI (registered trademark) is used for the generative AI.

[0474] 4. Emotion Engine The emotion engine uses technology to recognize emotions from the user's facial expressions, tone of voice, input text, etc. This uses the Affectiva SDK (registered trademark). The emotion engine sends the user's emotional information to the server, and the generative AI generates an appropriate response based on that information.

[0475] 5. Virtual Space A virtual world is a three-dimensional virtual environment that users can access to socialize, hold business meetings, or shop. The virtual world is built using Unity® 3D.

[0476] Processing flow

[0477] User Behavior When a user accesses a virtual store through smart glasses and browses products, their facial expressions and voices are collected by the device. For example, if a user is interested in a particular product, that behavioral data is sent to the server.

[0478] Analyzing data and generating responses The server analyzes the received data using a generative AI (ChatGPT®) and an emotion engine (Affectiva SDK®). Based on the user's emotions and input, the generative AI generates useful information such as detailed descriptions of products the user is interested in and other customer reviews.

[0479] Feedback to users The generated responses and supporting information are sent to the terminal in real time, and the user can check the information on the screen of the smart glasses. For example, a message such as "This product looks great on you. It has been highly rated by other customers" may be displayed.

[0480] Examples

[0481] As a specific use case, consider a scenario in which a user is choosing a dress in a virtual fashion store. As the user tries on the chosen dress, the emotion engine judges the user's satisfaction to be high based on the user's facial expressions and voice. Based on this information, the generative AI generates a support message such as "This dress is perfect for you and has been well received by many customers," and displays it on the user's smart glasses.

[0482] Example prompt:

[0483] When the user is choosing tableware in the virtual store, he / she looks a little confused. As a generative AI, please generate a support message that is appropriate for this situation.

[0484] As described above, the system provides emotion-based real-time support to enhance social interactions, business meetings, and shopping experiences in virtual spaces.

[0485] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0486] Step 1: A user accesses a virtual store in a virtual space through a device such as smart glasses. A 3D model of the virtual store is displayed on the user's interface. The user focuses on a specific product and selects it. At this stage, the device collects the user's facial expression data and voice data in real time. Input: User's gaze, facial expression data, voice data Output: 3D model display of virtual space, selected product data

[0487] Step 2: The device transmits the collected data of the user's facial expression and voice to the server. The transmitted data includes the user's facial features, tone of voice, selected product ID, etc. Input: User facial expression data, voice data, product ID from the device Output: Data packet sent to the server

[0488] Step 3: The server analyzes the received data. First, the emotion engine (Affectiva SDK®) analyzes facial features and tone of voice to recognize the user's emotional state (e.g., interest, confusion, satisfaction, etc.). Input: User's facial expression data, voice data Output: User's emotional state data

[0489] Step 4: The server passes the emotional state data and the selected product ID as input to a generative AI (ChatGPT (registered trademark)). The generative AI generates an appropriate support message or response based on the user's emotional state and the product information. Input: User's emotional state data, product ID Output: The generated support message or response.

[0490] Step 5: The generated support messages and responses are sent from the server to the terminal and displayed in real time on the user's smart glasses, for example, "This product looks great on you. Other customers have given it high marks." Input: Generated support message or response Output: Message displayed on the smart glasses

[0491] Step 6: If the user takes any further action (such as asking a question about a product or making a purchase), that information is also sent from the device to the server and is again analyzed by the generative AI and emotion engine, which then generates the next appropriate support or response in real time. Input: User's additional action data Output: The following support message or response:

[0492] Through these steps, the system can provide appropriate support and responses in real time based on the user's emotions, improving the social and shopping experience in the virtual space.

[0493] The specific processing unit 290 transmits the result of the specific processing to the smart glasses 214. In the smart glasses 214, the control unit 46A causes the speaker 240 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits the voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0494] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0495] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the smart glasses 214.

[0496] [Third embodiment]

[0497] FIG. 5 shows an example of the configuration of a data processing system 310 according to the third embodiment.

[0498] 5, the data processing system 310 includes the data processing device 12 and a headset type terminal 314. An example of the data processing device 12 is a server.

[0499] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0500] The headset type terminal 314 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a display 343. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. The microphone 238, the speaker 240, the camera 42, and the display 343 are also connected to the bus 52.

[0501] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0502] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0503] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0504] Fig. 6 shows an example of main functions of the data processing device 12 and the headset type terminal 314. As shown in Fig. 6, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0505] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0506] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0507] In the headset type terminal 314, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0508] Next, a description will be given of the identification process performed by the identification processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as the "server", and the headset type terminal 314 will be referred to as the "terminal".

[0509] The embodiment for carrying out the present invention comprises the following elements.

[0510] 1. Server: Receives user input and processes it to send it to the generative AI. It also receives support and moderation generated by the generative AI and processes it to send it to the terminal.

[0511] 2. Terminal: Displays the information entered by the user, and displays support and moderation received from the generative AI. It also detects the user's dialogue and actions and transmits them to the server.

[0512] 3. Generative AI: Analyzes user input and generates presentation support and discussion moderation in real time. It also takes into account the user's emotions to provide appropriate support and moderation.

[0513] 4. Virtual space: A virtual space for users to participate in, where events such as workshops and seminars are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0514] [Specific example]

[0515] When a user wishes to hold a business meeting in a virtual space, the invention can be implemented in the following manner.

[0516] 1. The user enters a self-introduction, suggestions, requests, etc., and sends them to the server via the terminal.

[0517] 2. The server receives the user's input and sends it to the generative AI.

[0518] 3. Generative AI analyses user input and generates presentation support and discussion moderation in real time.

[0519] 4. The generated support and moderation are sent to the terminal via the server.

[0520] 5. The device displays the support and moderation received and provides it to the user.

[0521] 6. Users can conduct business meetings in the virtual space while referring to the displayed support and moderation.

[0522] In this manner, users can smoothly conduct social gatherings and business meetings in the virtual space. Users can also participate in workshops and seminars in the virtual space and deepen their interactions with other participants.

[0523] The process flow will be explained below.

[0524] Step 1: The user inputs a self-introduction, proposal, request, etc., and sends it to the server via the terminal.

[0525] Step 2: The server receives the user's input and sends it to the generative AI.

[0526] Step 3: The generative AI analyzes user input and generates presentation support and discussion moderation in real time.

[0527] Step 4: The support and moderation generated by the generative AI are sent to the device via the server.

[0528] Step 5: The device displays the support and moderation received and provides it to the user.

[0529] Step 6: The user conducts a business meeting in the virtual space while referring to the displayed support and moderation.

[0530] Step 7: User interactions and actions are detected on the device and sent to the server.

[0531] Step 8: The server sends the user's interactions and actions to the generative AI, which updates them in real time.

[0532] Step 9: Updated support and moderation is sent to the device via the server.

[0533] Step 10: The terminal displays the received update information and provides it to the user.

[0534] Step 11: The user continues the business meeting in the virtual space while referring to the displayed updated information.

[0535] Example 1

[0536] Next, a description will be given of Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0537] In conventional social and business meetings in virtual spaces, users often lacked support to effectively present and discuss in real time. As a result, the quality and efficiency of meetings declined, and the user experience was impaired. In addition, it was difficult to properly analyze the user's input information and provide support and moderation that took the user's emotions into account. This meant that users could not receive appropriate feedback or support, making smooth communication difficult.

[0538] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0539] In this invention, the server includes means for receiving a user's input and generating at least one of a presentation support and a discussion moderation in real time via a generative artificial intelligence, means for transmitting at least one of the support and the moderation to the user's terminal, and means for displaying at least one of the support and the moderation on the user's terminal, thereby enabling the user to effectively socialize or hold a business meeting in a virtual space in real time. In addition, the generative artificial intelligence analyzes the user's input and provides appropriate support and moderation taking into account the user's emotions, improving the user's experience and making communication proceed smoothly.

[0540] "User input" refers to the input of a user's self-introduction, suggestions, opinions, requests, and requests to the other party through a terminal.

[0541] A "generative artificial intelligence" is an artificial intelligence model that analyzes user input and generates presentation support and / or discussion moderation in real time.

[0542] "Generating in real time" means selecting, generating, and outputting presentation support and discussion moderation almost instantly after receiving user input.

[0543] "Presentation support" refers to providing information and materials that enable users to give presentations effectively and smoothly.

[0544] "Discussion moderation" refers to providing advice, questions, feedback, etc. to facilitate discussions among users.

[0545] "Sending to the user's device" means transferring the generated support and moderation content as data from the server to a device such as a PC or smartphone used by the user.

[0546] "Displaying on the user's device" means presenting the submitted support and moderation content on the screen of the device (PC, smartphone, etc.) used by the user.

[0547] "Virtual social and business meetings" refers to activities in which users communicate with other users in a virtual environment created on the Internet, including business meetings and social gatherings.

[0548] The present invention relates to a system for smoothly conducting social interactions and business meetings in a virtual space. The configuration and operation of this system will be described in detail below.

[0549] System Configuration

[0550] The system consists of the following main elements:

[0551] 1. Server

[0552] The server uses a cloud server equipped with a high-performance CPU and memory (e.g. AWS (registered trademark) EC2 (registered trademark)). It receives user input and processes it to send it to the generative AI. It also receives support and moderation generated by the generative AI and processes it to send it to the terminal.

[0553] 2. Terminal

[0554] This refers to the PC or smartphone used by the user (e.g. Windows® PC, iPhone®, Android® device). The terminal displays the user's input and displays support and moderation received from the generative AI. It also detects the user's dialogue and actions and processes them to send them to the server.

[0555] 3. Generative Artificial Intelligence

[0556] Use specialized artificial intelligence tools (e.g., OpenAI ChatGPT, Google BERT). Analyze user input and generate presentation support and discussion moderation in real time. Additionally, provide appropriate support and moderation by taking into account user sentiment.

[0557] 4. Virtual Space

[0558] Use an online event platform (e.g. VRChat (registered trademark), Zoom (registered trademark)). This is a place where users can participate and events such as workshops and seminars are held.

[0559] Example of system operation

[0560] A specific example of operation is shown below.

[0561] 1. User Input The user types into the terminal, "I would like to share my ideas for promoting a new product."

[0562] 2. Server Processing The server receives user input and performs formatting and validation of the data. Send the generative artificial intelligence a prompt: "Generate promotional ideas for a new product."

[0563] 3. Analysis and Generation of Generative Artificial Intelligence Generative artificial intelligence analyses user input and generates relevant ideas and presentation materials in real time. Example prompt: "Generate promotional ideas for a new product."

[0564] 4. Sending the generated results The server transmits the generated ideas and materials to the user's terminal.

[0565] 5. Display of terminal The user's terminal displays the generated promotion ideas and materials on the screen. Users can refer to the displayed information and proceed with the meeting in the virtual space.

[0566] How to use

[0567] When a user holds a business meeting in a virtual space, he or she uses this system in the following steps.

[0568] 1. The user inputs a self-introduction, suggestions, requests, etc. through the terminal. 2. The server receives the user's input and sends it to the generative artificial intelligence. 3. Generative AI analyses user input and generates the necessary support and moderation in real time. 4. The generated support and moderation are sent to the user's device via the server. 5. The device displays the support and moderation received and provides it to the user. 6. The user proceeds with the meeting in the virtual space while referring to the displayed support and moderation.

[0569] This allows users to smoothly conduct social and business meetings in a virtual space and to communicate efficiently with other participants.

[0570] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0571] Program processing steps and detailed explanation

[0572] Step 1: Users use the terminal to input information such as self-introductions, suggestions, and requests. (Input) Text information entered by the user (e.g., "I would like to share my ideas for promoting new products") (Output) User input data that the terminal sends to the server (Specific operation) The user enters an idea into the text box on the terminal and clicks the send button.

[0573] Step 2: The server receives the data input by the user and performs initial processing, checking the format and content of the data and converting it into a format for sending to the generative artificial intelligence. (Input) User input data (Output) Formatted data (format for sending to generative artificial intelligence) (Specific operation) The server checks the string and unifies the format, and also handles errors if necessary.

[0574] Step 3: The server sends the formatted data to the generative AI, which uses prompts to guide the AI ​​in its proper analysis. (Input) Formatted data, prompt (e.g. "Generate promotional ideas for a new product") (Output) Generated ideas and materials (generated by generative artificial intelligence) (Specific operation) When the server sends a prompt sentence to the generative artificial intelligence, it uses an HTTP request and calls the API.

[0575] Step 4: Generative artificial intelligence analyzes user input based on prompt text and generates relevant information and materials. (Input) Prompt statement, user input data (Output) Support materials for generated presentations and moderation of discussions (Specific operation) Generative AI analyzes the user's requests and generates related ideas and specific presentation materials.

[0576] Step 5:

[0577] The server receives the data generated by the generative artificial intelligence and transmits it to the user's device. (Input) Generated supporting materials and moderation data (Output) Data sent to the user's terminal (Specific Operation) The server checks the format of the received data again and sends the data to the user's terminal.

[0578] Step 6: The terminal displays supporting materials and moderation data received from the server. (Input) Data sent from the server (Output) User-visible screen display (Specific Operation) The terminal displays the received data at a specified position on the screen, making it easy for the user to view.

[0579] Step 7: Users can conduct business meetings and discussions in the virtual space while referring to the supporting materials and moderation data displayed on their devices. (Input) Support materials and moderation data displayed on the terminal (Output) Use in user meetings and discussions (Specific operation) The user uses tools in the virtual space to use the generated materials and hold a discussion. For example, the user gives a presentation in a virtual conference room and exchanges opinions with other participants.

[0580] (Application example 1)

[0581] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal."

[0582] We aim to improve the user experience not only in social interactions and business meetings in virtual spaces, but also in virtual stores. In particular, it is required to provide appropriate product recommendations and purchasing support in real time when users have questions or suggestions about products. However, current systems lack support that can immediately respond to user requests, resulting in a decrease in user satisfaction and loss of purchasing opportunities.

[0583] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0584] In this invention, the server includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation in real time via generative AI, a means for presenting the support and the moderation, a means for generating at least one of product recommendations and purchasing support based on the user's questions and suggestions in the virtual environment, and a means for presenting the recommendations and support. This allows the user to obtain appropriate information in real time in the virtual space or virtual store, making it possible to smoothly select and purchase products.

[0585] A "virtual space" is a three-dimensional virtual space generated using computer technology, and is an environment in which users can participate interactively.

[0586] "Generative AI" is an artificial intelligence technology that analyzes user input and generates appropriate responses and support information in real time.

[0587] "Presentation support" is a function that provides users with assistance with creating materials, organizing key points, and giving advice on how to speak when giving a presentation.

[0588] "Discussion moderation" is a function that organizes topics, promotes dialogue, provides feedback on ideas, and more, to ensure that discussions between users proceed smoothly.

[0589] "Product recommendation" is a function that selects and suggests the most suitable products based on the user's interests and requests.

[0590] "Purchase support" is a function that provides the user with the necessary information and procedural assistance when selecting and purchasing a product.

[0591] The "server" is a central processing unit that receives user input, transmits it to the generative AI, and transmits the generated information to the terminal.

[0592] A "terminal" is a device on which a user inputs information and on which information provided by generative AI is displayed. Examples of such devices include smartphones, smart glasses, and head-mounted displays.

[0593] "User input" refers to information such as a self-introduction, a suggestion, an opinion, a request, a question, or a request to a conversation partner that a user gives to the system.

[0594] This invention is a system for smoothly conducting social and business meetings in a virtual space and improving the user experience in a virtual store. The system mainly consists of a server, a terminal, a generative AI, and a virtual space, and operates as follows.

[0595] server

[0596] The server plays a central role in receiving user input and sending it to the generative AI. Specifically, it analyzes the information entered by the user through the terminal in real time and transfers it to the generative AI. It also sends support, moderation, product recommendations, and purchasing support generated by the generative AI to the terminal.

[0597] Terminal

[0598] A terminal is a device that allows users to input and display information. This includes smartphones, smart glasses, and head-mounted displays. Through the terminal, users input suggestions, questions, requests, etc., and receive and display responses from generative AI.

[0599] Generative AI

[0600] The generative AI analyzes user input and generates at least one of the following in real time: presentation support, discussion moderation, product recommendations, and purchasing support. This generated information is sent to the terminal via the server and provided to the user.

[0601] Virtual Space

[0602] A virtual world is a three-dimensional virtual space in which users can participate interactively, where they can socialize, hold business meetings, and enjoy shopping in virtual stores.

[0603] Specific examples of hardware and software used

[0604] Hardware: Servers, smartphones, smart glasses, head-mounted displays

[0605] Software: OpenAI® generative AI models (e.g. GPT® models), virtual environment software

[0606] Examples

[0607] Example 1: Use in business meetings

[0608] When a user holds a business meeting in a virtual space and proposes a new project, the generative AI provides real-time advice on the proposal. For example, if the user asks, "How can I make my presentation materials more visually appealing?" the generative AI will provide specific advice such as, "Add a graph and use color to indicate the points you want to emphasize."

[0609] Example 2: Use in a virtual store

[0610] When a user asks "What is a recommended gift for Father's Day?" through smart glasses in a virtual store, generative AI will generate a list of related products and suggest them to the user. For example, it will make a recommendation such as "This luxury watch has an elegant design and is popular with men."

[0611] Examples of prompt statements

[0612] Prompt: "Virtual storefront question: 'What would you recommend as a gift for Father's Day?' Generate an answer."

[0613] This allows the system to provide users with the most appropriate information in real time in the virtual space or virtual store, and to respond quickly to questions and suggestions.

[0614] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0615] Step 1: The user inputs information through the terminal. Specifically, a user uses a smartphone or smart glasses to input information such as self-introduction, suggestions, questions, requests, etc. in text or voice format. The input information is sent from the terminal to the server. The input data is in text format, and the output is the received data on the server side.

[0616] Step 2: The server receives the user's input and prepares it to be sent to the generative AI. Specifically, the server analyzes the input data it receives and formats it to send an appropriate prompt to the generative AI. Data processing involves analyzing the user's input, extracting the necessary information, and converting it into a prompt. As an output, a prompt is generated in a format that the generative AI can understand.

[0617] Step 3: The server sends the generated prompt to the generation AI. Specifically, the server sends the generated prompt text to the generative AI (e.g., GPT (registered trademark) model) via API. This causes the generative AI to start processing according to the user's input. The input data is the prompt text, and the response content is obtained from the generative AI as output.

[0618] Step 4: A generative AI analyzes the prompt and generates an appropriate response in real time. Specifically, generative AI performs natural language processing based on the prompt sentence and generates responses such as appropriate presentation support, discussion moderation, product recommendations, and purchasing support. Data calculation involves analysis and generation using a multi-layer neural network. The output data is the response content.

[0619] Step 5: The server receives the response from the generative AI and sends it to the terminal. Specifically, the server formats the response received from the generative AI in a format that is easy for the user to understand. After that, a communication process is performed to send it to the terminal. The input data is the response from the generative AI, and the output data is the data that is ready to be sent to the terminal.

[0620] Step 6: The terminal receives the response from the server and displays it to the user. Specifically, the terminal displays the response received from the server on the screen. The user can proceed with their activities in the virtual space based on the displayed information. Data processing involves converting the received data into a display format. The response information that can be viewed by the user is obtained as the output.

[0621] Step 7: The user takes action based on the generated response. Specifically, the user refers to the received information to advance a presentation or discussion in the virtual space, select a product in a virtual store, and complete a purchase procedure. The input data is the displayed response content, and the specific user action occurs as the output.

[0622] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0623] The embodiment for carrying out the present invention comprises the following elements.

[0624] 1. Server: Receives user input and processes it using a combination of generative AI and an emotion engine. The emotion engine recognizes the user's emotions and provides appropriate support and moderation based on them.

[0625] 2. Terminal: Displays the information entered by the user and the emotional information received from the emotion engine, and displays support and moderation received from the generative AI. It also detects the user's dialogue and behavior and transmits them to the server.

[0626] 3. Generative AI: Analyzes user input and generates presentation support and discussion moderation in real time, taking into account the emotional information received from the emotion engine to provide more appropriate support and moderation.

[0627] 4. Emotion Engine: An engine that combines technologies to recognize user emotions, using facial recognition, voice analysis, natural language processing, etc.

[0628] 5. Virtual space: A virtual space for users to participate in, where events such as workshops and seminars are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0629] [Specific example]

[0630] When a user wishes to hold a business meeting in a virtual space, the invention can be implemented in the following manner.

[0631] 1. The user enters a self-introduction, suggestions, requests, etc., and sends them to the server via the terminal.

[0632] 2. The server receives user input and processes it using a combination of generative AI and an emotion engine. The emotion engine recognizes the user's emotions and provides appropriate support or moderation based on them.

[0633] 3. Generative AI analyzes user input and generates presentation support and discussion moderation in real time, taking into account the emotional information received from the emotion engine to provide more appropriate support and moderation.

[0634] 4. The generated support and moderation are sent to the terminal via the server.

[0635] 5. The device displays the support and moderation received and provides it to the user. It also detects the user's interactions and actions and transmits them to the server.

[0636] 6. Users can conduct business meetings in the virtual space while referring to the displayed support and moderation.

[0637] In this manner, users can smoothly conduct social and business meetings in virtual space with support that takes into account emotions. Users can also participate in workshops and seminars in virtual space and deepen their interactions with other participants.

[0638] The process flow will be explained below.

[0639] Step 1: The user inputs a self-introduction, proposal, request, etc., and sends it to the server via the terminal.

[0640] Step 2: The server receives the user's input and processes it to recognize the user's emotion using the emotion engine.

[0641] Step 3: The emotion engine combines technologies such as facial recognition, voice analysis, and natural language processing to recognize the user's emotions and send the results to the server.

[0642] Step 4: The server sends user input and emotion information to the generative AI to generate presentation support and discussion moderation in real time.

[0643] Step 5: The generative AI analyzes the user's input and emotional information and generates appropriate support or moderation.

[0644] Step 6: The generated support and moderation are sent to the terminal via the server.

[0645] Step 7: The device displays the support and moderation received and provides it to the user.

[0646] Step 8: The user conducts a business meeting in the virtual space while referring to the displayed support and moderation.

[0647] Step 9: User interactions and actions are detected on the device and sent to the server.

[0648] Step 10: The server sends the user's interactions and actions to the generative AI for updates.

[0649] Step 11: Updated support and moderation is sent to the device via the server.

[0650] Step 12: The terminal displays the received update information and provides it to the user.

[0651] Step 13: The user continues the business meeting in the virtual space while referring to the displayed updated information.

[0652] Example 2

[0653] Next, a description will be given of Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the headset type terminal 314 will be referred to as a "terminal".

[0654] Conventional social networking services and business meeting tools provide information without considering the user's emotional state, making it difficult to provide appropriate support according to the user's emotions. As a result, users end up engaging in activities in virtual space while feeling tension or anxiety, which leads to problems such as reduced performance and satisfaction.

[0655] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation in real time via a generative AI and an emotion engine, a means for presenting at least one of the support and the moderation, a means for providing support based on the user's emotion information using an emotion engine that analyzes the user's emotion, a means for transmitting the generated support information and moderation to the user's terminal and displaying them on the user's terminal, and a means for the user to interact in the virtual space. This makes it possible to provide appropriate support according to the user's emotion and to smoothly carry out activities in the virtual space.

[0656] "User input" refers to information that a user of the system inputs via a terminal, and includes at least one of a self-introduction, a suggestion, an opinion, a request, and a request to the other party in the conversation.

[0657] "Generative AI" is artificial intelligence that analyzes user input and generates presentation support and discussion moderation in real time.

[0658] An "emotion engine" is a system that combines technologies such as facial recognition, voice analysis, and natural language processing to recognize a user's emotions and provide information about those emotions.

[0659] "Presentation support" is a function that provides information and materials necessary for a user to give a presentation in a virtual space.

[0660] "Discussion moderation" is a function that provides appropriate instructions and adjustments to facilitate smooth discussions and dialogue between users.

[0661] A "virtual space" is a virtual space on the Internet in which users can participate, and where events such as workshops, seminars, and business meetings are held.

[0662] A "terminal" is a device through which a user enters information and views the offered support and moderation, such as a computer or smartphone.

[0663] The "server" is a computer system that receives information sent by users, analyzes it, generates the necessary support information and moderation via a generative AI and emotion engine, and sends it to the user's device.

[0664] "Emotion information" is data about the user's emotional state that the emotion engine obtains through facial recognition, voice analysis, natural language processing, and the like.

[0665] The system for implementing the present invention is composed of the following elements: a server, a terminal, a generative AI, an emotion engine, and a virtual space.

[0666] server

[0667] The server has the function of receiving user input and generating at least one of presentation support and discussion moderation in real time via generative AI and an emotion engine. It also generates support information that takes into account the user's emotion information based on the analysis results of the emotion engine. The server is responsible for receiving and analyzing the information sent by the user, and sending the generated support information and moderation to the user's device. A secure communication protocol (e.g., HTTPS) is used in this process.

[0668] Terminal

[0669] The terminal is a device used by the user to input information and check support information and moderation. The user inputs information through the terminal, and the information is securely transmitted to the server. The terminal displays the support information and moderation transmitted from the server in real time and provides it to the user. It also includes a function to detect the user's interactions and actions and transmit such information to the server.

[0670] Generative AI

[0671] Generative AI analyzes user input and generates presentation support and discussion moderation in real time. Generative AI uses technologies such as natural language processing, speech recognition, and image analysis. In this process, it takes into account emotional information, such as when the user is nervous, to provide appropriate support.

[0672] Emotion Engine

[0673] The emotion engine uses facial recognition, voice analysis, and natural language processing to recognize the user's emotions. For example, it recognizes emotions such as joy, anger, and sadness from the user's facial expressions and tone of voice, and provides that emotional information to the generative AI.

[0674] Virtual Space

[0675] A virtual space is a virtual space on the Internet for users to participate in, where events such as workshops, seminars, business meetings, etc. are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0676] Examples

[0677] When a user wants to hold a business meeting in a virtual space, the system works as follows.

[0678] 1. The user types "I would like to give a presentation on our new product" through the terminal. 2. The server receives the information and passes it to the generative AI, while at the same time analyzing the user's emotional information using the emotion engine. 3. The emotion engine recognizes that the user is a little nervous and provides that information to the generative AI. 4. Generative AI will generate support information tailored to the user, such as slides that highlight the main points of a presentation or advice on how to relax. 5. The server sends the generated support information to the user's terminal. 6. The device displays the received support information and moderation to the user. 7. The user uses the received information to hold a business meeting in the virtual space.

[0679] Examples of prompt statements

[0680] "Please provide support information for a user to give a presentation on a new product. The user seems a bit nervous. Please generate appropriate advice based on the following information:

[0681] Key points from the presentation

[0682] Tips for relaxing

[0683] With the above configuration, the user can smoothly carry out activities in the virtual space while receiving appropriate support that takes into account the user's emotions.

[0684] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0685] Processing Steps

[0686] Step 1: The user enters information into the device The user inputs information such as self-introduction, proposals, requests, etc. via the terminal. This input includes, for example, the text "I would like to give a presentation on a new product." After inputting, a send button to proceed to the next step is displayed on the terminal. Input: User's text information (e.g. "I would like to give a presentation on a new product") Output: The encoded input data (e.g., data in JSON format).

[0687] Step 2: The device sends the input information to the server The terminal transmits the information input by the user to the server. This transmission is performed using a secure protocol (e.g. HTTPS). Input: The encoded input data Output: Data sent to the server

[0688] Step 3: The server receives the input information and analyzes it with the emotion engine The server passes the received user input to the emotion engine, which analyzes the user's emotions using facial recognition, voice analysis, and natural language processing to determine the user's emotional state. Input: Data sent to the server (e.g. user text and corresponding metadata) Data processing / computation: Emotion analysis (e.g., facial recognition, voice analysis, and assessment of emotional state through natural language processing) Output: Data about the user's emotional state (e.g., "The user is nervous")

[0689] Step 4: The server passes the analysis results to the generative AI The server passes the user's input information to the generative AI along with the analysis results obtained from the emotion engine. Input: Data about the user's emotional state, user input data Output: Data passed to the generative AI

[0690] Step 5: Generative AI generates supporting information and moderation The generative AI generates appropriate support information and discussion moderation based on the input information and emotional information received from the server. By taking emotional information into account, advice and materials appropriate for the user are created. Input: Data passed to the generative AI Data processing / computation: Generation of supporting information and moderation (e.g., natural language processing, data analysis) Output: Generated supporting information and moderation (e.g. slides to emphasize key points in the presentation, advice for relaxation)

[0691] Step 6: The server sends the generated results to the terminal. The server transmits the support information and moderation generated by the generative AI to the terminal. This transmission is also done using a secure communication protocol. Input: Generated supporting information and moderation Output: Data sent to the terminal

[0692] Step 7: The terminal displays the results to the user The device displays the received support information and moderation to the user, for example by displaying presentation slides or relaxation techniques on the screen. Input: Data sent to the terminal Output: Support information and moderation displayed to users

[0693] Step 8: Users use the results to take action in the virtual space Users can conduct business meetings and other activities in the virtual space by referring to the support information and moderation displayed on the terminal. Specifically, users can hold discussions with other participants in the virtual space and give presentations. Input: Support information and moderation displayed on the terminal Output: Activity in the virtual space (e.g., a successful presentation, a good discussion)

[0694] As described above, through the specific operations and data flows performed at each processing step, support is provided to enable the user to move smoothly in the virtual space.

[0695] (Application example 2)

[0696] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server", and the headset type terminal 314 will be referred to as a "terminal".

[0697] Conventional user experiences in social interactions, business meetings, and virtual stores in virtual spaces lacked support and responses based on the user's emotions, which led to insufficient customer satisfaction. In addition, real-time support based on the user's input was rarely provided, which often hindered efficient communication and shopping experiences. In particular, due to a lack of technology that properly recognized the user's emotions and provided support based on them, the user's experience tended to be one-way.

[0698] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server, in an application that enables socializing and business meetings in a virtual space, as well as shopping experiences in a virtual store, includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation, and in-store product guide in real time via a generative AI, a means including an emotion engine that recognizes the user's emotions and provides appropriate responses and support based on the emotions, and a means for presenting the support, the moderation, and at least one of the responses. This allows the user to receive appropriate support and moderation that take emotions into account in real time, making it possible to facilitate communication and improve customer satisfaction.

[0699] "Virtual space" refers to a three-dimensional virtual environment constructed using digital technology that enables real-time interaction.

[0700] "Socializing" refers to activities in which people communicate with each other and build social networks.

[0701] "Business meeting" refers to a conference or meeting held for business purposes.

[0702] A "virtual store" refers to a digital store that operates on the Internet or in a virtual space.

[0703] "Generative AI" refers to artificial intelligence that analyzes user input and generates appropriate responses and support in real time based on that input.

[0704] An "emotion engine" refers to a system that combines technologies to analyze a user's facial expressions, voice, and input content and recognize the user's emotions.

[0705] "Presentation support" refers to providing information, materials, and advice in real time to assist a user in making a presentation.

[0706] "Discussion moderation" refers to intervening and adjusting at the appropriate time to ensure that discussions and dialogue between users proceed smoothly.

[0707] "Product Introduction" refers to providing information about products and services within the Virtual Store.

[0708] "User input" refers to various information provided and operations performed by a user on the system.

[0709] "Response" refers to the reply or action that the system generates based on the user's input and emotions.

[0710] "Support" refers to information and advice that the system provides to assist the user in their activities.

[0711] This invention is an advanced system for supporting users' social interactions, business meetings, and shopping experiences in virtual stores in a virtual space. The system is realized mainly using a server, a terminal, a generative AI, an emotion engine, and a virtual space.

[0712] System Configuration

[0713] 1. Server The server receives user input and performs processing that combines generative AI and an emotion engine. Specifically, the server analyzes the input data sent by the user and generates appropriate support and responses in real time. The server uses cloud infrastructure such as Azure Cloud Services (registered trademark).

[0714] 2. Terminal The terminal refers to the device used by the user, and includes smartphones, smart glasses, head-mounted displays, etc. The terminal collects the user's input and sends the data to the server. It also presents support and responses sent from the server to the user. For example, Microsoft HoloLens 2 is used as the smart glasses.

[0715] 3. Generative AI The generative AI analyzes the user's input and takes into account the emotional information obtained from the emotion engine to generate presentation support, discussion moderation, and / or product guides in the virtual store. ChatGPT (registered trademark) from OpenAI (registered trademark) is used for the generative AI.

[0716] 4. Emotion Engine The emotion engine uses technology to recognize emotions from the user's facial expressions, tone of voice, input text, etc. This uses the Affectiva SDK (registered trademark). The emotion engine sends the user's emotional information to the server, and the generative AI generates an appropriate response based on that information.

[0717] 5. Virtual Space A virtual world is a three-dimensional virtual environment that users can access to socialize, hold business meetings, or shop. The virtual world is built using Unity® 3D.

[0718] Processing flow

[0719] User Behavior When a user accesses a virtual store through smart glasses and browses products, their facial expressions and voices are collected by the device. For example, if a user is interested in a particular product, that behavioral data is sent to the server.

[0720] Analyzing data and generating responses The server analyzes the received data using a generative AI (ChatGPT®) and an emotion engine (Affectiva SDK®). Based on the user's emotions and input, the generative AI generates useful information such as detailed descriptions of products the user is interested in and other customer reviews.

[0721] Feedback to users The generated responses and supporting information are sent to the terminal in real time, and the user can check the information on the screen of the smart glasses. For example, a message such as "This product looks great on you. It has been highly rated by other customers" may be displayed.

[0722] Examples

[0723] As a specific use case, consider a scenario in which a user is choosing a dress in a virtual fashion store. As the user tries on the chosen dress, the emotion engine judges the user's satisfaction to be high based on the user's facial expressions and voice. Based on this information, the generative AI generates a support message such as "This dress is perfect for you and has been well received by many customers," and displays it on the user's smart glasses.

[0724] Example prompt:

[0725] When the user is choosing tableware in the virtual store, he / she looks a little confused. As a generative AI, please generate a support message that is appropriate for this situation.

[0726] As described above, the system provides emotion-based real-time support to enhance social interactions, business meetings, and shopping experiences in virtual spaces.

[0727] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0728] Step 1: A user accesses a virtual store in a virtual space through a device such as smart glasses. A 3D model of the virtual store is displayed on the user's interface. The user focuses on a specific product and selects it. At this stage, the device collects the user's facial expression data and voice data in real time. Input: User's gaze, facial expression data, voice data Output: 3D model display of virtual space, selected product data

[0729] Step 2: The device transmits the collected data of the user's facial expression and voice to the server. The transmitted data includes the user's facial features, tone of voice, selected product ID, etc. Input: User facial expression data, voice data, product ID from the device Output: Data packet sent to the server

[0730] Step 3: The server analyzes the received data. First, the emotion engine (Affectiva SDK®) analyzes facial features and tone of voice to recognize the user's emotional state (e.g., interest, confusion, satisfaction, etc.). Input: User's facial expression data, voice data Output: User's emotional state data

[0731] Step 4: The server passes the emotional state data and the selected product ID as input to a generative AI (ChatGPT (registered trademark)). The generative AI generates an appropriate support message or response based on the user's emotional state and the product information. Input: User's emotional state data, product ID Output: The generated support message or response.

[0732] Step 5: The generated support messages and responses are sent from the server to the terminal and displayed in real time on the user's smart glasses, for example, "This product looks great on you. Other customers have given it high marks." Input: Generated support message or response Output: Message displayed on the smart glasses

[0733] Step 6: If the user takes any further action (such as asking a question about a product or making a purchase), that information is also sent from the device to the server and is again analyzed by the generative AI and emotion engine, which then generates the next appropriate support or response in real time. Input: User's additional action data Output: The following support message or response:

[0734] Through these steps, the system can provide appropriate support and responses in real time based on the user's emotions, improving the social and shopping experience in the virtual space.

[0735] The specific processing unit 290 transmits the result of the specific processing to the headset type terminal 314. In the headset type terminal 314, the control unit 46A causes the speaker 240 and the display 343 to output the result of the specific processing. The microphone 238 acquires voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0736] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0737] In the above embodiment, an example was given in which the specific processing is performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific processing may be performed by the headset type terminal 314.

[0738] [Fourth embodiment]

[0739] FIG. 7 shows an example of the configuration of a data processing system 410 according to the fourth embodiment.

[0740] 7, a data processing system 410 includes a data processing device 12 and a robot 414. An example of the data processing device 12 is a server.

[0741] The data processing device 12 includes a computer 22, a database 24, and a communication I / F 26. The computer 22 is an example of a "computer" according to the technology of the present disclosure. The computer 22 includes a processor 28, a RAM 30, and a storage 32. The processor 28, the RAM 30, and the storage 32 are connected to a bus 34. The database 24 and the communication I / F 26 are also connected to the bus 34. The communication I / F 26 is connected to a network 54. Examples of the network 54 include a wide area network (WAN) and / or a local area network (LAN).

[0742] The robot 414 includes a computer 36, a microphone 238, a speaker 240, a camera 42, a communication I / F 44, and a control target 443. The computer 36 includes a processor 46, a RAM 48, and a storage 50. The processor 46, the RAM 48, and the storage 50 are connected to a bus 52. In addition, the microphone 238, the speaker 240, the camera 42, and the control target 443 are also connected to the bus 52.

[0743] The microphone 238 receives instructions and the like from the user 20 by receiving voice uttered by the user 20. The microphone 238 captures the voice uttered by the user 20, converts the captured voice into audio data, and outputs it to the processor 46. The speaker 240 outputs the voice according to instructions from the processor 46.

[0744] Camera 42 is a small digital camera equipped with an optical system including a lens, an aperture, and a shutter, and an imaging element such as a CMOS (Complementary Metal-Oxide-Semiconductor) image sensor or a CCD (Charge Coupled Device) image sensor, and captures images of the surroundings of user 20 (e.g., an imaging range defined by an angle of view equivalent to the width of the field of vision of an average healthy person).

[0745] The communication I / F 44 is connected to a network 54. The communication I / Fs 44 and 26 are responsible for transmitting and receiving various types of information between the processor 46 and the processor 28 via the network 54. The transmission and reception of various types of information between the processor 46 and the processor 28 using the communication I / Fs 44 and 26 is performed in a secure state.

[0746] The control target 443 includes a display device, LEDs in the eyes, and motors for driving the arms, hands, legs, etc. The posture and behavior of the robot 414 are controlled by controlling the motors of the arms, hands, legs, etc. Some of the emotions of the robot 414 can be expressed by controlling these motors. In addition, the facial expressions of the robot 414 can also be expressed by controlling the light emission state of the LEDs in the eyes of the robot 414.

[0747] Fig. 8 shows an example of main functions of the data processing device 12 and the robot 414. As shown in Fig. 8, in the data processing device 12, a specific process is performed by the processor 28. A specific process program 56 is stored in the storage 32.

[0748] The specific processing program 56 is an example of a "program" according to the technology of the present disclosure. The processor 28 reads the specific processing program 56 from the storage 32, and executes the read specific processing program 56 on the RAM 30. The specific processing is realized by the processor 28 operating as a specific processing unit 290 in accordance with the specific processing program 56 executed on the RAM 30.

[0749] The storage 32 stores a data generation model 58 and an emotion identification model 59. The data generation model 58 and the emotion identification model 59 are used by the identification processing unit 290.

[0750] In the robot 414, the reception output process is performed by the processor 46. A reception output program 60 is stored in the storage 50. The processor 46 reads the reception output program 60 from the storage 50, and executes the read reception output program 60 on the RAM 48. The reception output process is realized by the processor 46 operating as the control unit 46A in accordance with the reception output program 60 executed on the RAM 48.

[0751] Next, a description will be given of the specific processing by the specific processing unit 290 of the data processing device 12. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[0752] The embodiment for carrying out the present invention comprises the following elements.

[0753] 1. Server: Receives user input and processes it to send it to the generative AI. It also receives support and moderation generated by the generative AI and processes it to send it to the terminal.

[0754] 2. Terminal: Displays the information entered by the user, and displays support and moderation received from the generative AI. It also detects the user's dialogue and actions and transmits them to the server.

[0755] 3. Generative AI: Analyzes user input and generates presentation support and discussion moderation in real time. It also takes into account the user's emotions to provide appropriate support and moderation.

[0756] 4. Virtual space: A virtual space for users to participate in, where events such as workshops and seminars are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0757] [Specific example]

[0758] When a user wishes to hold a business meeting in a virtual space, the invention can be implemented in the following manner.

[0759] 1. The user enters a self-introduction, suggestions, requests, etc., and sends them to the server via the terminal.

[0760] 2. The server receives the user's input and sends it to the generative AI.

[0761] 3. Generative AI analyses user input and generates presentation support and discussion moderation in real time.

[0762] 4. The generated support and moderation are sent to the terminal via the server.

[0763] 5. The device displays the support and moderation received and provides it to the user.

[0764] 6. Users can conduct business meetings in the virtual space while referring to the displayed support and moderation.

[0765] In this manner, users can smoothly conduct social gatherings and business meetings in the virtual space. Users can also participate in workshops and seminars in the virtual space and deepen their interactions with other participants.

[0766] The process flow will be explained below.

[0767] Step 1: The user inputs a self-introduction, proposal, request, etc., and sends it to the server via the terminal.

[0768] Step 2: The server receives the user's input and sends it to the generative AI.

[0769] Step 3: The generative AI analyzes user input and generates presentation support and discussion moderation in real time.

[0770] Step 4: The support and moderation generated by the generative AI are sent to the device via the server.

[0771] Step 5: The device displays the received support and moderation and provides it to the user. Step 6: The user conducts a business meeting in the virtual space while referring to the displayed support and moderation.

[0772] Step 7: User interactions and actions are detected on the device and sent to the server.

[0773] Step 8: The server sends the user's interactions and actions to the generative AI, which updates them in real time.

[0774] Step 9: Updated support and moderation is sent to the device via the server.

[0775] Step 10: The terminal displays the received update information and provides it to the user.

[0776] Step 11: The user continues the business meeting in the virtual space while referring to the displayed updated information.

[0777] Example 1

[0778] Next, a description will be given of Example 1. In the following description, the data processing device 12 is referred to as a "server" and the robot 414 is referred to as a "terminal."

[0779] In conventional social and business meetings in virtual spaces, users often lacked support to effectively present and discuss in real time. As a result, the quality and efficiency of meetings declined, and the user experience was impaired. In addition, it was difficult to properly analyze the user's input information and provide support and moderation that took the user's emotions into account. This meant that users could not receive appropriate feedback or support, making smooth communication difficult.

[0780] The specific processing by the specific processing unit 290 of the data processing device 12 in the first embodiment is realized by the following means.

[0781] In this invention, the server includes means for receiving a user's input and generating at least one of a presentation support and a discussion moderation in real time via a generative artificial intelligence, means for transmitting at least one of the support and the moderation to the user's terminal, and means for displaying at least one of the support and the moderation on the user's terminal, thereby enabling the user to effectively socialize or hold a business meeting in a virtual space in real time. In addition, the generative artificial intelligence analyzes the user's input and provides appropriate support and moderation taking into account the user's emotions, improving the user's experience and making communication proceed smoothly.

[0782] "User input" refers to the input of a user's self-introduction, suggestions, opinions, requests, and requests to the other party through a terminal.

[0783] A "generative artificial intelligence" is an artificial intelligence model that analyzes user input and generates presentation support and / or discussion moderation in real time.

[0784] "Generating in real time" means selecting, generating, and outputting presentation support and discussion moderation almost instantly after receiving user input.

[0785] "Presentation support" refers to providing information and materials that enable users to give presentations effectively and smoothly.

[0786] "Discussion moderation" refers to providing advice, questions, feedback, etc. to facilitate discussions among users.

[0787] "Sending to the user's device" means transferring the generated support and moderation content as data from the server to a device such as a PC or smartphone used by the user.

[0788] "Displaying on the user's device" means presenting the submitted support and moderation content on the screen of the device (PC, smartphone, etc.) used by the user.

[0789] "Virtual social and business meetings" refers to activities in which users communicate with other users in a virtual environment created on the Internet, including business meetings and social gatherings.

[0790] The present invention relates to a system for smoothly conducting social interactions and business meetings in a virtual space. The configuration and operation of this system will be described in detail below.

[0791] System Configuration

[0792] The system consists of the following main elements:

[0793] 1. Server

[0794] The server uses a cloud server equipped with a high-performance CPU and memory (e.g. AWS (registered trademark) EC2 (registered trademark)). It receives user input and processes it to send it to the generative AI. It also receives support and moderation generated by the generative AI and processes it to send it to the terminal.

[0795] 2. Terminal

[0796] This refers to the PC or smartphone used by the user (e.g. Windows® PC, iPhone®, Android® device). The terminal displays the user's input and displays support and moderation received from the generative AI. It also detects the user's dialogue and actions and processes them to send them to the server.

[0797] 3. Generative Artificial Intelligence

[0798] Use specialized artificial intelligence tools (e.g., OpenAI ChatGPT, Google BERT). Analyze user input and generate presentation support and discussion moderation in real time. Additionally, provide appropriate support and moderation by taking into account user sentiment.

[0799] 4. Virtual Space

[0800] Use an online event platform (e.g. VRChat (registered trademark), Zoom (registered trademark)). This is a place where users can participate and events such as workshops and seminars are held.

[0801] Example of system operation

[0802] A specific example of operation is shown below.

[0803] 1. User Input The user types into the terminal, "I would like to share my ideas for promoting a new product."

[0804] 2. Server Processing The server receives user input and performs formatting and validation of the data. Send the generative artificial intelligence a prompt: "Generate promotional ideas for a new product."

[0805] 3. Analysis and Generation of Generative Artificial Intelligence Generative artificial intelligence analyses user input and generates relevant ideas and presentation materials in real time. Example prompt: "Generate promotional ideas for a new product."

[0806] 4. Sending the generated results The server transmits the generated ideas and materials to the user's terminal.

[0807] 5. Display of terminal The user's terminal displays the generated promotion ideas and materials on the screen. Users can refer to the displayed information and proceed with the meeting in the virtual space.

[0808] How to use

[0809] When a user holds a business meeting in a virtual space, he or she uses this system in the following steps.

[0810] 1. The user inputs a self-introduction, suggestions, requests, etc. through the terminal. 2. The server receives the user's input and sends it to the generative artificial intelligence. 3. Generative AI analyses user input and generates the necessary support and moderation in real time. 4. The generated support and moderation are sent to the user's device via the server. 5. The device displays the support and moderation received and provides it to the user. 6. The user proceeds with the meeting in the virtual space while referring to the displayed support and moderation.

[0811] This allows users to smoothly conduct social and business meetings in a virtual space and to communicate efficiently with other participants.

[0812] The flow of the identification process in the first embodiment will be described with reference to FIG.

[0813] Program processing steps and detailed explanation

[0814] Step 1: Users use the terminal to input information such as self-introductions, suggestions, and requests. (Input) Text information entered by the user (e.g., "I would like to share my ideas for promoting new products") (Output) User input data that the terminal sends to the server (Specific operation) The user enters an idea into the text box on the terminal and clicks the send button.

[0815] Step 2: The server receives the data input by the user and performs initial processing, checking the format and content of the data and converting it into a format for sending to the generative artificial intelligence. (Input) User input data (Output) Formatted data (format for sending to generative artificial intelligence) (Specific operation) The server checks the string and unifies the format, and also handles errors if necessary.

[0816] Step 3: The server sends the formatted data to the generative AI, which uses prompts to guide the AI ​​in its proper analysis. (Input) Formatted data, prompt (e.g. "Generate promotional ideas for a new product") (Output) Generated ideas and materials (generated by generative artificial intelligence) (Specific operation) When the server sends a prompt sentence to the generative artificial intelligence, it uses an HTTP request and calls the API.

[0817] Step 4: Generative artificial intelligence analyzes user input based on prompt text and generates relevant information and materials. (Input) Prompt statement, user input data (Output) Support materials for generated presentations and moderation of discussions (Specific operation) Generative AI analyzes the user's requests and generates related ideas and specific presentation materials.

[0818] Step 5: The server receives the data generated by the generative artificial intelligence and transmits it to the user's device. (Input) Generated supporting materials and moderation data (Output) Data sent to the user's terminal (Specific Operation) The server checks the format of the received data again and sends the data to the user's terminal.

[0819] Step 6: The terminal displays supporting materials and moderation data received from the server. (Input) Data sent from the server (Output) User-visible screen display (Specific Operation) The terminal displays the received data at a specified position on the screen, making it easy for the user to view.

[0820] Step 7: Users can conduct business meetings and discussions in the virtual space while referring to the supporting materials and moderation data displayed on their devices. (Input) Support materials and moderation data displayed on the terminal (Output) Use in user meetings and discussions (Specific operation) The user uses tools in the virtual space to use the generated materials and hold a discussion. For example, the user gives a presentation in a virtual conference room and exchanges opinions with other participants.

[0821] (Application example 1)

[0822] Next, a description will be given of Application Example 1. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal."

[0823] We aim to improve the user experience not only in social interactions and business meetings in virtual spaces, but also in virtual stores. In particular, it is required to provide appropriate product recommendations and purchasing support in real time when users have questions or suggestions about products. However, current systems lack support that can immediately respond to user requests, resulting in a decrease in user satisfaction and loss of purchasing opportunities.

[0824] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 1 is realized by the following means.

[0825] In this invention, the server includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation in real time via generative AI, a means for presenting the support and the moderation, a means for generating at least one of product recommendations and purchasing support based on the user's questions and suggestions in the virtual environment, and a means for presenting the recommendations and support. This allows the user to obtain appropriate information in real time in the virtual space or virtual store, making it possible to smoothly select and purchase products.

[0826] A "virtual space" is a three-dimensional virtual space generated using computer technology, and is an environment in which users can participate interactively.

[0827] "Generative AI" is an artificial intelligence technology that analyzes user input and generates appropriate responses and support information in real time.

[0828] "Presentation support" is a function that provides users with assistance with creating materials, organizing key points, and giving advice on how to speak when giving a presentation.

[0829] "Discussion moderation" is a function that organizes topics, promotes dialogue, provides feedback on ideas, and more, to ensure that discussions between users proceed smoothly.

[0830] "Product recommendation" is a function that selects and suggests the most suitable products based on the user's interests and requests.

[0831] "Purchase support" is a function that provides the user with the necessary information and procedural assistance when selecting and purchasing a product.

[0832] The "server" is a central processing unit that receives user input, transmits it to the generative AI, and transmits the generated information to the terminal.

[0833] A "terminal" is a device on which a user inputs information and on which information provided by generative AI is displayed. Examples of such devices include smartphones, smart glasses, and head-mounted displays.

[0834] "User input" refers to information such as a self-introduction, a suggestion, an opinion, a request, a question, or a request to a conversation partner that a user gives to the system.

[0835] This invention is a system for smoothly conducting social and business meetings in a virtual space and improving the user experience in a virtual store. The system mainly consists of a server, a terminal, a generative AI, and a virtual space, and operates as follows.

[0836] server

[0837] The server plays a central role in receiving user input and sending it to the generative AI. Specifically, it analyzes the information entered by the user through the terminal in real time and transfers it to the generative AI. It also sends support, moderation, product recommendations, and purchasing support generated by the generative AI to the terminal.

[0838] Terminal

[0839] A terminal is a device that allows users to input and display information. This includes smartphones, smart glasses, and head-mounted displays. Through the terminal, users input suggestions, questions, requests, etc., and receive and display responses from generative AI.

[0840] Generative AI

[0841] The generative AI analyzes user input and generates at least one of the following in real time: presentation support, discussion moderation, product recommendations, and purchasing support. This generated information is sent to the terminal via the server and provided to the user.

[0842] Virtual Space

[0843] A virtual world is a three-dimensional virtual space in which users can participate interactively, where they can socialize, hold business meetings, and enjoy shopping in virtual stores.

[0844] Specific examples of hardware and software used

[0845] Hardware: Servers, smartphones, smart glasses, head-mounted displays

[0846] Software: OpenAI® generative AI models (e.g. GPT® models), virtual environment software

[0847] Examples

[0848] Example 1: Use in business meetings

[0849] When a user holds a business meeting in a virtual space and proposes a new project, the generative AI provides real-time advice on the proposal. For example, if the user asks, "How can I make my presentation materials more visually appealing?" the generative AI will provide specific advice such as, "Add a graph and use color to indicate the points you want to emphasize."

[0850] Example 2: Use in a virtual store

[0851] When a user asks "What is a recommended gift for Father's Day?" through smart glasses in a virtual store, generative AI will generate a list of related products and suggest them to the user. For example, it will make a recommendation such as "This luxury watch has an elegant design and is popular with men."

[0852] Examples of prompt statements

[0853] Prompt: "Virtual storefront question: 'What would you recommend as a gift for Father's Day?' Generate an answer."

[0854] This allows the system to provide users with the most appropriate information in real time in the virtual space or virtual store, and to respond quickly to questions and suggestions.

[0855] The flow of the specific process in the application example 1 will be described with reference to FIG.

[0856] Step 1: The user inputs information through the terminal. Specifically, a user uses a smartphone or smart glasses to input information such as self-introduction, suggestions, questions, requests, etc. in text or voice format. The input information is sent from the terminal to the server. The input data is in text format, and the output is the received data on the server side.

[0857] Step 2: The server receives the user's input and prepares it to be sent to the generative AI. Specifically, the server analyzes the input data it receives and formats it to send an appropriate prompt to the generative AI. Data processing involves analyzing the user's input, extracting the necessary information, and converting it into a prompt. As an output, a prompt is generated in a format that the generative AI can understand.

[0858] Step 3: The server sends the generated prompt to the generation AI. Specifically, the server sends the generated prompt text to the generative AI (e.g., GPT (registered trademark) model) via API. This causes the generative AI to start processing according to the user's input. The input data is the prompt text, and the response content is obtained from the generative AI as output.

[0859] Step 4: A generative AI analyzes the prompt and generates an appropriate response in real time. Specifically, generative AI performs natural language processing based on the prompt sentence and generates responses such as appropriate presentation support, discussion moderation, product recommendations, and purchasing support. Data calculation involves analysis and generation using a multi-layer neural network. The output data is the response content.

[0860] Step 5: The server receives the response from the generative AI and sends it to the terminal. Specifically, the server formats the response received from the generative AI in a format that is easy for the user to understand. After that, a communication process is performed to send it to the terminal. The input data is the response from the generative AI, and the output data is the data that is ready to be sent to the terminal.

[0861] Step 6: The terminal receives the response from the server and displays it to the user. Specifically, the terminal displays the response received from the server on the screen. The user can proceed with their activities in the virtual space based on the displayed information. Data processing involves converting the received data into a display format. The response information that can be viewed by the user is obtained as the output.

[0862] Step 7: The user takes action based on the generated response. Specifically, the user refers to the received information to advance a presentation or discussion in the virtual space, select a product in a virtual store, and complete a purchase procedure. The input data is the displayed response content, and the specific user action occurs as the output.

[0863] In addition, an emotion engine that estimates the user's emotion may be further combined. That is, the identification processing unit 290 may estimate the user's emotion using the emotion identification model 59, and perform identification processing using the user's emotion.

[0864] The embodiment for carrying out the present invention comprises the following elements.

[0865] 1. Server: Receives user input and processes it using a combination of generative AI and an emotion engine. The emotion engine recognizes the user's emotions and provides appropriate support and moderation based on them.

[0866] 2. Terminal: Displays the information entered by the user and the emotional information received from the emotion engine, and displays support and moderation received from the generative AI. It also detects the user's dialogue and behavior and transmits them to the server.

[0867] 3. Generative AI: Analyzes user input and generates presentation support and discussion moderation in real time, taking into account the emotional information received from the emotion engine to provide more appropriate support and moderation.

[0868] 4. Emotion Engine: An engine that combines technologies to recognize user emotions, using facial recognition, voice analysis, natural language processing, etc.

[0869] 5. Virtual space: A virtual space for users to participate in, where events such as workshops and seminars are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0870] [Specific example]

[0871] When a user wishes to hold a business meeting in a virtual space, the invention can be implemented in the following manner.

[0872] 1. The user enters a self-introduction, suggestions, requests, etc., and sends them to the server via the terminal.

[0873] 2. The server receives user input and processes it using a combination of generative AI and an emotion engine. The emotion engine recognizes the user's emotions and provides appropriate support or moderation based on them.

[0874] 3. Generative AI analyzes user input and generates presentation support and discussion moderation in real time, taking into account the emotional information received from the emotion engine to provide more appropriate support and moderation.

[0875] 4. The generated support and moderation are sent to the terminal via the server.

[0876] 5. The device displays the support and moderation received and provides it to the user. It also detects the user's interactions and actions and transmits them to the server.

[0877] 6. Users can conduct business meetings in the virtual space while referring to the displayed support and moderation.

[0878] In this manner, users can smoothly conduct social and business meetings in virtual space with support that takes into account emotions. Users can also participate in workshops and seminars in virtual space and deepen their interactions with other participants.

[0879] The process flow will be explained below.

[0880] Step 1: The user inputs a self-introduction, proposal, request, etc., and sends it to the server via the terminal.

[0881] Step 2: The server receives the user's input and processes it to recognize the user's emotion using the emotion engine.

[0882] Step 3: The emotion engine combines technologies such as facial recognition, voice analysis, and natural language processing to recognize the user's emotions and send the results to the server.

[0883] Step 4: The server sends user input and emotion information to the generative AI to generate presentation support and discussion moderation in real time.

[0884] Step 5: The generative AI analyzes the user's input and emotional information and generates appropriate support or moderation.

[0885] Step 6: The generated support and moderation are sent to the terminal via the server.

[0886] Step 7: The device displays the support and moderation received and provides it to the user.

[0887] Step 8: The user conducts a business meeting in the virtual space while referring to the displayed support and moderation.

[0888] Step 9: User interactions and actions are detected on the device and sent to the server.

[0889] Step 10: The server sends the user's interactions and actions to the generative AI for updates.

[0890] Step 11: Updated support and moderation is sent to the device via the server.

[0891] Step 12: The terminal displays the received update information and provides it to the user.

[0892] Step 13: The user continues the business meeting in the virtual space while referring to the displayed updated information.

[0893] Example 2

[0894] Next, a description will be given of Example 2. In the following description, the data processing device 12 is referred to as a "server" and the robot 414 is referred to as a "terminal."

[0895] Conventional social networking services and business meeting tools provide information without considering the user's emotional state, making it difficult to provide appropriate support according to the user's emotions. As a result, users end up engaging in activities in virtual space while feeling tension or anxiety, which leads to problems such as reduced performance and satisfaction.

[0896] The specific processing by the specific processing unit 290 of the data processing device 12 in the second embodiment is realized by the following means. In this invention, the server includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation in real time via a generative AI and an emotion engine, a means for presenting at least one of the support and the moderation, a means for providing support based on the user's emotion information using an emotion engine that analyzes the user's emotion, a means for transmitting the generated support information and moderation to the user's terminal and displaying them on the user's terminal, and a means for the user to interact in the virtual space. This makes it possible to provide appropriate support according to the user's emotion and to smoothly carry out activities in the virtual space.

[0897] "User input" refers to information that a user of the system inputs via a terminal, and includes at least one of a self-introduction, a suggestion, an opinion, a request, and a request to the other party in the conversation.

[0898] "Generative AI" is artificial intelligence that analyzes user input and generates presentation support and discussion moderation in real time.

[0899] An "emotion engine" is a system that combines technologies such as facial recognition, voice analysis, and natural language processing to recognize a user's emotions and provide information about those emotions.

[0900] "Presentation support" is a function that provides information and materials necessary for a user to give a presentation in a virtual space.

[0901] "Discussion moderation" is a function that provides appropriate instructions and adjustments to facilitate smooth discussions and dialogue between users.

[0902] A "virtual space" is a virtual space on the Internet in which users can participate, and where events such as workshops, seminars, and business meetings are held.

[0903] A "terminal" is a device through which a user enters information and views the offered support and moderation, such as a computer or smartphone.

[0904] The "server" is a computer system that receives information sent by users, analyzes it, generates the necessary support information and moderation via a generative AI and emotion engine, and sends it to the user's device.

[0905] "Emotion information" is data about the user's emotional state that the emotion engine obtains through facial recognition, voice analysis, natural language processing, and the like.

[0906] The system for implementing the present invention is composed of the following elements: a server, a terminal, a generative AI, an emotion engine, and a virtual space.

[0907] server

[0908] The server has the function of receiving user input and generating at least one of presentation support and discussion moderation in real time via generative AI and an emotion engine. It also generates support information that takes into account the user's emotion information based on the analysis results of the emotion engine. The server is responsible for receiving and analyzing the information sent by the user, and sending the generated support information and moderation to the user's device. A secure communication protocol (e.g., HTTPS) is used in this process.

[0909] Terminal

[0910] The terminal is a device used by the user to input information and check support information and moderation. The user inputs information through the terminal, and the information is securely transmitted to the server. The terminal displays the support information and moderation transmitted from the server in real time and provides it to the user. It also includes a function to detect the user's interactions and actions and transmit such information to the server.

[0911] Generative AI

[0912] Generative AI analyzes user input and generates presentation support and discussion moderation in real time. Generative AI uses technologies such as natural language processing, speech recognition, and image analysis. In this process, it takes into account emotional information, such as when the user is nervous, to provide appropriate support.

[0913] Emotion Engine

[0914] The emotion engine uses facial recognition, voice analysis, and natural language processing to recognize the user's emotions. For example, it recognizes emotions such as joy, anger, and sadness from the user's facial expressions and tone of voice, and provides that emotional information to the generative AI.

[0915] Virtual Space

[0916] A virtual space is a virtual space on the Internet for users to participate in, where events such as workshops, seminars, business meetings, etc. are held. Users can interact with other participants in the virtual space and deepen their relationships.

[0917] Examples

[0918] When a user wants to hold a business meeting in a virtual space, the system works as follows.

[0919] 1. The user types "I would like to give a presentation on our new product" through the terminal. 2. The server receives the information and passes it to the generative AI, while at the same time analyzing the user's emotional information using the emotion engine. 3. The emotion engine recognizes that the user is a little nervous and provides that information to the generative AI. 4. Generative AI will generate support information tailored to the user, such as slides that highlight the main points of a presentation or advice on how to relax. 5. The server sends the generated support information to the user's terminal. 6. The device displays the received support information and moderation to the user. 7. The user uses the received information to hold a business meeting in the virtual space.

[0920] Examples of prompt statements

[0921] "Please provide support information for a user to give a presentation on a new product. The user seems a bit nervous. Please generate appropriate advice based on the following information: Key points from the presentation Tips for relaxing

[0922] With the above configuration, the user can smoothly carry out activities in the virtual space while receiving appropriate support that takes into account the user's emotions.

[0923] The flow of the identification process in the second embodiment will be described with reference to FIG.

[0924] Processing Steps

[0925] Step 1: The user enters information into the device The user inputs information such as self-introduction, proposals, requests, etc. via the terminal. This input includes, for example, the text "I would like to give a presentation on a new product." After inputting, a send button to proceed to the next step is displayed on the terminal. Input: User's text information (e.g. "I would like to give a presentation on a new product") Output: The encoded input data (e.g., data in JSON format).

[0926] Step 2: The device sends the input information to the server The terminal transmits the information input by the user to the server. This transmission is performed using a secure protocol (e.g. HTTPS). Input: The encoded input data Output: Data sent to the server

[0927] Step 3: The server receives the input information and analyzes it with the emotion engine The server passes the received user input to the emotion engine, which analyzes the user's emotions using facial recognition, voice analysis, and natural language processing to determine the user's emotional state. Input: Data sent to the server (e.g. user text and corresponding metadata) Data processing / computation: Emotion analysis (e.g., facial recognition, voice analysis, and assessment of emotional state through natural language processing) Output: Data about the user's emotional state (e.g., "The user is nervous")

[0928] Step 4: The server passes the analysis results to the generative AI The server passes the user's input information to the generative AI along with the analysis results obtained from the emotion engine. Input: Data about the user's emotional state, user input data Output: Data passed to the generative AI

[0929] Step 5: Generative AI generates supporting information and moderation The generative AI generates appropriate support information and discussion moderation based on the input information and emotional information received from the server. By taking emotional information into account, advice and materials appropriate for the user are created. Input: Data passed to the generative AI Data processing / computation: Generation of supporting information and moderation (e.g., natural language processing, data analysis) Output: Generated supporting information and moderation (e.g. slides to emphasize key points in the presentation, advice for relaxation)

[0930] Step 6: The server sends the generated results to the terminal. The server transmits the support information and moderation generated by the generative AI to the terminal. This transmission is also done using a secure communication protocol. Input: Generated supporting information and moderation Output: Data sent to the terminal

[0931] Step 7: The terminal displays the results to the user The device displays the received support information and moderation to the user, for example by displaying presentation slides or relaxation techniques on the screen. Input: Data sent to the terminal Output: Support information and moderation displayed to users

[0932] Step 8: Users use the results to take action in the virtual space Users can conduct business meetings and other activities in the virtual space by referring to the support information and moderation displayed on the terminal. Specifically, users can hold discussions with other participants in the virtual space and give presentations. Input: Support information and moderation displayed on the terminal Output: Activity in the virtual space (e.g., a successful presentation, a good discussion)

[0933] As described above, through the specific operations and data flows performed at each processing step, support is provided to enable the user to move smoothly in the virtual space.

[0934] (Application example 2)

[0935] Next, a description will be given of Application Example 2. In the following description, the data processing device 12 will be referred to as a "server" and the robot 414 will be referred to as a "terminal".

[0936] Conventional user experiences in social interactions, business meetings, and virtual stores in virtual spaces lacked support and responses based on the user's emotions, which led to insufficient customer satisfaction. In addition, real-time support based on the user's input was rarely provided, which often hindered efficient communication and shopping experiences. In particular, due to a lack of technology that properly recognized the user's emotions and provided support based on them, the user's experience tended to be one-way.

[0937] The specific processing by the specific processing unit 290 of the data processing device 12 in the application example 2 is realized by the following means. In this invention, the server, in an application that enables socializing and business meetings in a virtual space, as well as shopping experiences in a virtual store, includes a means for receiving a user's input and generating at least one of presentation support and discussion moderation, and in-store product guide in real time via a generative AI, a means including an emotion engine that recognizes the user's emotions and provides appropriate responses and support based on the emotions, and a means for presenting the support, the moderation, and at least one of the responses. This allows the user to receive appropriate support and moderation that take emotions into account in real time, making it possible to facilitate communication and improve customer satisfaction.

[0938] "Virtual space" refers to a three-dimensional virtual environment constructed using digital technology that enables real-time interaction.

[0939] "Socializing" refers to activities in which people communicate with each other and build social networks.

[0940] "Business meeting" refers to a conference or meeting held for business purposes.

[0941] A "virtual store" refers to a digital store that operates on the Internet or in a virtual space.

[0942] "Generative AI" refers to artificial intelligence that analyzes user input and generates appropriate responses and support in real time based on that input.

[0943] An "emotion engine" refers to a system that combines technologies to analyze a user's facial expressions, voice, and input content and recognize the user's emotions.

[0944] "Presentation support" refers to providing information, materials, and advice in real time to assist a user in making a presentation.

[0945] "Discussion moderation" refers to intervening and adjusting at the appropriate time to ensure that discussions and dialogue between users proceed smoothly.

[0946] "Product Introduction" refers to providing information about products and services within the Virtual Store.

[0947] "User input" refers to various information provided and operations performed by a user on the system.

[0948] "Response" refers to the reply or action that the system generates based on the user's input and emotions.

[0949] "Support" refers to information and advice that the system provides to assist the user in their activities.

[0950] This invention is an advanced system for supporting users' social interactions, business meetings, and shopping experiences in virtual stores in a virtual space. The system is realized mainly using a server, a terminal, a generative AI, an emotion engine, and a virtual space.

[0951] System Configuration

[0952] 1. Server The server receives user input and performs processing that combines generative AI and an emotion engine. Specifically, the server analyzes the input data sent by the user and generates appropriate support and responses in real time. The server uses cloud infrastructure such as Azure Cloud Services (registered trademark).

[0953] 2. Terminal The terminal refers to a device used by a user, and includes a smartphone, smart glasses, a head-mounted display, etc. The terminal collects user input and transmits the data to a server. It also presents support and responses transmitted from the server to the user. For example, Microsoft HoloLens 2 (registered trademark) is used as the smart glasses.

[0954] 3. Generative AI The generative AI analyzes the user's input and takes into account the emotional information obtained from the emotion engine to generate presentation support, discussion moderation, and / or product guides in the virtual store. ChatGPT (registered trademark) from OpenAI (registered trademark) is used for the generative AI.

[0955] 4. Emotion Engine The emotion engine uses technology to recognize emotions from the user's facial expressions, tone of voice, input text, etc. This uses the Affectiva SDK (registered trademark). The emotion engine sends the user's emotional information to the server, and the generative AI generates an appropriate response based on that information.

[0956] 5. Virtual Space A virtual world is a three-dimensional virtual environment that users can access to socialize, hold business meetings, or shop. The virtual world is built using Unity® 3D.

[0957] Processing flow

[0958] User Behavior When a user accesses a virtual store through smart glasses and browses products, their facial expressions and voices are collected by the device. For example, if a user is interested in a particular product, that behavioral data is sent to the server.

[0959] Analyzing data and generating responses The server analyzes the received data using a generative AI (ChatGPT®) and an emotion engine (Affectiva SDK®). Based on the user's emotions and input, the generative AI generates useful information such as detailed descriptions of products the user is interested in and other customer reviews.

[0960] Feedback to users The generated responses and supporting information are sent to the terminal in real time, and the user can check the information on the screen of the smart glasses. For example, a message such as "This product looks great on you. It has been highly rated by other customers" may be displayed.

[0961] Examples

[0962] As a specific use case, consider a scenario in which a user is choosing a dress in a virtual fashion store. As the user tries on the chosen dress, the emotion engine judges the user's satisfaction to be high based on the user's facial expressions and voice. Based on this information, the generative AI generates a support message such as "This dress is perfect for you and has been well received by many customers," and displays it on the user's smart glasses.

[0963] Example prompt:

[0964] When the user is choosing tableware in the virtual store, he / she looks a little confused. As a generative AI, please generate a support message that is appropriate for this situation.

[0965] As described above, the system provides emotion-based real-time support to enhance social interactions, business meetings, and shopping experiences in virtual spaces.

[0966] The flow of the specific process in the application example 2 will be described with reference to FIG.

[0967] Step 1: A user accesses a virtual store in a virtual space through a device such as smart glasses. A 3D model of the virtual store is displayed on the user's interface. The user focuses on a specific product and selects it. At this stage, the device collects the user's facial expression data and voice data in real time. Input: User's gaze, facial expression data, voice data Output: 3D model display of virtual space, selected product data

[0968] Step 2: The device transmits the collected data of the user's facial expression and voice to the server. The transmitted data includes the user's facial features, tone of voice, selected product ID, etc. Input: User facial expression data, voice data, product ID from the device Output: Data packet sent to the server

[0969] Step 3: The server analyzes the received data. First, the emotion engine (Affectiva SDK®) analyzes facial features and tone of voice to recognize the user's emotional state (e.g., interest, confusion, satisfaction, etc.). Input: User's facial expression data, voice data Output: User's emotional state data

[0970] Step 4: The server passes the emotional state data and the selected product ID as input to a generative AI (ChatGPT (registered trademark)). The generative AI generates an appropriate support message or response based on the user's emotional state and the product information. Input: User's emotional state data, product ID Output: The generated support message or response.

[0971] Step 5: The generated support messages and responses are sent from the server to the terminal and displayed in real time on the user's smart glasses, for example, "This product looks great on you. Other customers have given it high marks." Input: Generated support message or response Output: Message displayed on the smart glasses

[0972] Step 6: If the user takes any further action (such as asking a question about a product or making a purchase), that information is also sent from the device to the server and is again analyzed by the generative AI and emotion engine, which then generates the next appropriate support or response in real time. Input: User's additional action data Output: The following support message or response:

[0973] Through these steps, the system can provide appropriate support and responses in real time based on the user's emotions, improving the social and shopping experience in the virtual space.

[0974] The specific processing unit 290 transmits the result of the specific processing to the robot 414. In the robot 414, the control unit 46A causes the speaker 240 and the control target 443 to output the result of the specific processing. The microphone 238 acquires a voice indicating a user input for the result of the specific processing. The control unit 46A transmits voice data indicating the user input acquired by the microphone 238 to the data processing device 12. In the data processing device 12, the specific processing unit 290 acquires the voice data.

[0975] The data generation model 58 is a so-called generative AI (Artificial Intelligence). An example of the data generation model 58 is ChatGPT (registered trademark) (Internet search<URL: https: / / openai.com / blog / chatgpt> ), Gemini (registered trademark) (Internet search <url: https: gemini.google.com ?hl="ja">) and other generation AIs. The data generation model 58 is obtained by making a neural network perform deep learning. A prompt including an instruction is input to the data generation model 58, and inference data such as voice data indicating a voice, text data indicating a text, and image data indicating an image is input. The data generation model 58 performs inference on the input inference data according to the instruction indicated by the prompt, and outputs the inference result in a data format such as voice data and text data. Here, inference refers to, for example, analysis, classification, prediction, and / or summarization.

[0976] In the above embodiment, an example was given in which the specific process was performed by the data processing device 12, but the technology of the present disclosure is not limited to this, and the specific process may be performed by the robot 414.

[0977] Each of the above systems allows users to socialize and conduct business meetings in a virtual space. Users can input self-introductions, proposals, opinions, requests, and requests to their interlocutors, and receive presentation support and discussion moderation in the form of voice, text, images, etc. in real time through generative AI. The generative AI supports and moderates the dialogue taking into account the user's emotions. Users can participate in workshops and seminars in the virtual space and deepen their interactions with experts and peers from all over the world.

[0978] For example, say a user wants to hold a business meeting in a virtual space. The user inputs their suggestions, opinions, and requests, and receives real-time support from the generative AI. The generative AI provides appropriate presentation support and discussion moderation based on the user's comments and emotions. The user can interact with other participants in the virtual space and share business ideas.

[0979] The system has the unique ability to facilitate social interactions and business meetings in virtual space. Users can deepen their interactions with experts and peers from around the world while receiving real-time support through generative AI. Future work will involve further development of the system, including more advanced emotion recognition and improved dialogue.

[0980] The emotion identification model 59 as an emotion engine may determine the emotion of the user according to a specific mapping. Specifically, the emotion identification model 59 may determine the emotion of the user according to an emotion map (see FIG. 9), which is a specific mapping. Similarly, the emotion identification model 59 may determine the emotion of the robot, and the identification processing unit 290 may perform identification processing using the emotion of the robot.

[0981] FIG. 9 is a diagram showing an emotion map 400 on which multiple emotions are mapped. In the emotion map 400, emotions are arranged in concentric circles radiating from the center. The closer to the center of the concentric circles, the more primitive emotions are arranged. The more outside the concentric circles, the more emotions that represent states and actions that arise from a state of mind are arranged. Emotions are a concept that includes emotions and mental states. On the left side of the concentric circles, emotions that are generally generated from reactions that occur in the brain are arranged. On the right side of the concentric circles, emotions that are generally induced by situational judgment are arranged. On the upper and lower sides of the concentric circles, emotions that are generally generated from reactions that occur in the brain and are induced by situational judgment are arranged. In addition, on the upper side of the concentric circles, emotions of "pleasure" are arranged, and on the lower side, emotions of "discomfort" are arranged. In this way, in the emotion map 400, multiple emotions are mapped based on the structure in which emotions are generated, and emotions that tend to occur simultaneously are mapped close to each other.

[0982] These emotions are distributed in the three o'clock direction of emotion map 400 and usually fluctuate between relief and anxiety. In the right half of emotion map 400, situational awareness dominates over internal sensations, resulting in a sense of calm.

[0983] The inside of emotion map 400 represents what is going on inside one's mind, and the outside of emotion map 400 represents behavior, so the further out on emotion map 400 you go, the more visible the emotions become (the more they are expressed in behavior).

[0984] Here, human emotions are based on various balances such as posture and blood sugar level, and when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. Emotions can also be created for robots, cars, motorcycles, etc., based on various balances such as posture and battery level, so that when these balances are far from the ideal, it indicates an unpleasant state, and when they are close to the ideal, it indicates a pleasant state. The emotion map may be generated, for example, based on the emotion map of Dr. Mitsuyoshi (Research on speech emotion recognition and emotion brain physiological signal analysis system, Tokushima University, doctoral dissertation: https: / / ci.nii.ac.jp / naid / 500000375379). On the left half of the emotion map, emotions belonging to an area called "reaction" where sensation is dominant are lined up. On the right half of the emotion map, emotions belonging to an area called "situation" where situation recognition is dominant are lined up.

[0985] The emotion map defines two emotions that promote learning. The first is the negative emotion around the middle of "repentance" or "remorse" on the situation side. In other words, this is when the robot experiences negative emotions such as "I never want to feel this way again" or "I don't want to be scolded again." The other is the positive emotion around "desire" on the response side. In other words, this is when the robot has positive feelings such as "I want more" or "I want to know more."

[0986] The emotion identification model 59 inputs the user input to a pre-trained neural network, obtains emotion values ​​indicating each emotion shown in the emotion map 400, and determines the emotion of the user. This neural network is pre-trained based on multiple learning data that are combinations of the user input and emotion values ​​indicating each emotion shown in the emotion map 400. Furthermore, this neural network is trained so that emotions that are located close to each other have similar values, as in the emotion map 900 shown in Fig. 10. Fig. 10 shows an example in which multiple emotions, "relief," "calm," and "encouraging," have similar emotion values.

[0987] Although the system according to the present disclosure has been described above mainly with respect to the functions of the data processing device 12, the system according to the present disclosure is not necessarily implemented in a server. The system according to the present disclosure may be implemented as a general information processing system. The present disclosure may be implemented, for example, as a software program that runs on a personal computer, or an application that runs on a smartphone or the like. The method according to the present disclosure may be provided to a user in the form of SaaS (Software as a Service).

[0988] In the above embodiment, an example is given in which the specific processing is performed by one computer 22, but the technology of the present disclosure is not limited to this, and the specific processing may be distributed and performed by a plurality of computers including the computer 22. For example, the data generation model 58 may be provided in an external device of the data processing device 12, and the external device may generate data according to input data.

[0989] In the above embodiment, an example in which the specific processing program 56 is stored in the storage 32 has been described, but the technology of the present disclosure is not limited to this. For example, the specific processing program 56 may be stored in a portable computer-readable non-transitory storage medium such as a Universal Serial Bus (USB) memory. The specific processing program 56 stored in the non-transitory storage medium is installed in the computer 22 of the data processing device 12. The processor 28 executes the specific processing in accordance with the specific processing program 56.

[0990] In addition, the specific processing program 56 may be stored in a storage device such as a server connected to the data processing device 12 via the network 54, and the specific processing program 56 may be downloaded and installed on the computer 22 upon request from the data processing device 12.

[0991] It is not necessary to store all of the specific processing program 56 in a storage device such as a server connected to the data processing device 12 via the network 54, or to store all of the specific processing program 56 in the storage 32; only a portion of the specific processing program 56 may be stored.

[0992] As the hardware resource for executing the specific process, various processors as shown below can be used. An example of the processor is a CPU, which is a general-purpose processor that functions as a hardware resource for executing the specific process by executing software, i.e., a program. Another example of the processor is a dedicated electric circuit, which is a processor having a circuit configuration designed exclusively for executing the specific process, such as a Field-Programmable Gate Array (FPGA), a Programmable Logic Device (PLD), or an Application Specific Integrated Circuit (ASIC). Each processor has a built-in or connected memory, and each processor executes the specific process by using the memory.

[0993] The hardware resource that executes the specific process may be one of these various processors, or may be a combination of two or more processors of the same or different types (for example, a combination of multiple FPGAs, or a combination of a CPU and an FPGA). Also, the hardware resource that executes the specific process may be a single processor.

[0994] As an example of a configuration using one processor, first, one processor is configured by combining one or more CPUs and software, and this processor functions as a hardware resource that executes a specific process. Second, there is a configuration using a processor that realizes the functions of the entire system including multiple hardware resources that execute a specific process on a single IC chip, as typified by SoC (System-on-a-chip). In this way, a specific process is realized using one or more of the above-mentioned various processors as hardware resources.

[0995] Furthermore, more specifically, the hardware structure of these various processors can be an electric circuit that combines circuit elements such as semiconductor elements. The specific processes described above are merely examples. It goes without saying that unnecessary steps may be deleted, new steps may be added, or the order of processes may be changed without departing from the spirit of the invention.

[0996] The above description and illustrations are detailed descriptions of the parts related to the technology of the present disclosure, and are merely an example of the technology of the present disclosure. For example, the above description of the configuration, function, action, and effect is an example of the configuration, function, action, and effect of the parts related to the technology of the present disclosure. Therefore, it goes without saying that unnecessary parts may be deleted, new elements may be added, or replacements may be made to the above description and illustrations, within the scope of the gist of the technology of the present disclosure. In addition, in order to avoid confusion and to facilitate understanding of the parts related to the technology of the present disclosure, the above description and illustrations omit explanations of technical common sense that do not require explanation in order to enable the implementation of the technology of the present disclosure.

[0997] All publications, patent applications, and standards mentioned in this specification are herein incorporated by reference to the same extent as if each individual publication, patent application, or standard was specifically and individually indicated to be incorporated by reference.

[0998] The following is further disclosed regarding the above embodiment.

[0999] (Claim 1) In social networking applications that allow social interactions and business meetings in virtual space, means for receiving user input and generating, via a generative AI, at least one of presentation support and discussion moderation in real time; means for providing at least one of said support and said moderation; A system including:

[1000] (Claim 2) The user's input includes at least one of a self-introduction, a suggestion, an opinion, a request, and a request to a dialogue partner; The system of claim 1.

[1001] (Claim 3) The generative AI analyzes the user's input and generates at least one of support for the presentation and moderation of the discussion in at least one form of audio, text, and images; A system according to claim 1 or claim 2.

[1002] (Claim 4) The present invention is characterized in that it combines an emotion engine that recognizes the user's emotions. The system of claim 1.

[1003] (Claim 5) Recognizing the user's emotion from at least one of the user's speech content, facial expression, and tone of voice, and generating at least one of the support and the moderation based on a result of the recognition; The system of claim 1.

[1004] (Claim 6) The emotion engine uses at least one of facial recognition, speech analysis, and natural language processing in recognizing the emotion of the user. The system of claim 4.

[1005] (Claim 7) The emotion engine transmits the result of recognizing the emotion of the user to a terminal via a server, and displays the result on the terminal. A system according to claim 4 or claim 6.

[1006] (Claim 8) The emotion engine transmits the result of recognizing the emotion of the user to the generative AI, and generates at least one of the support and the moderation more appropriately in accordance with the emotion of the user. A system according to claim 4, claim 6 or claim 7.

[1007] "Example 1" (Claim 1) means for receiving user input and generating, via a generative AI, at least one of presentation support and discussion moderation in real time; means for transmitting at least one of the support and the moderation to a user terminal; A means for displaying at least one of the support and the moderation on a user's terminal; A means for sharing and discussing information based on the generated support and moderation in a virtual space in which users participate; A means for providing appropriate support and moderation taking into account user sentiment; A system including:

[1008] (Claim 2) The user's input includes at least one of a self-introduction, a suggestion, an opinion, a request, and a request to a dialogue partner; 2. The system of claim 1.

[1009] (Claim 3) The generative AI analyzes the user's input and generates at least one of support for the presentation and moderation of the discussion in at least one form of audio, text, and images; 2. The system of claim 1.

[1010] "Application example 1" (Claim 1) In a system that allows social interactions and business meetings in virtual space, means for receiving user input and generating, via a generative AI, at least one of presentation support and discussion moderation in real time; means for providing at least one of said support and said moderation; means for generating at least one of product recommendations and purchasing assistance based on user questions and suggestions within the virtual environment; means for presenting said recommendations and support; A system including:

[1011] (Claim 2) The user input includes at least one of a self-introduction, a suggestion, an opinion, a request, a request for a dialogue partner, a product-related question, and a suggestion; 2. The system of claim 1.

[1012] (Claim 3) The generative AI analyzes the user's input and generates at least one of the presentation support and discussion moderation, the product recommendation, and the purchase support in at least one form of voice, text, and image; 2. The system of claim 1.

[1013] "Example 2 of combining emotion engines" (Claim 1) means for receiving user input and generating presentation support and / or discussion moderation in real time via a generative AI and an emotion engine; means for providing at least one of said support and said moderation; A means for providing support based on the user's emotion information by utilizing an emotion engine that analyzes the user's emotion; A means for transmitting the generated support information and moderation to a user's terminal and displaying the same on the user's terminal; A means for users to interact in the virtual space; A system including:

[1014] (Claim 2) The system of claim 1 , wherein the user input includes at least one of a self-introduction, a suggestion, an opinion, a request, and a request to a dialogue partner.

[1015] (Claim 3) 2. The system of claim 1, wherein the generative AI analyzes the user's input and generates at least one of support for the presentation and moderation of the discussion in at least one of the forms of audio, text, and images.

[1016] "Application example 2 when combining emotion engines" (Claim 1) Applications that enable social and business meetings in virtual spaces, as well as shopping experiences in virtual stores. means for receiving user input and generating, via generative AI, in real time, presentation support and / or discussion moderation and / or in-store product guidance; A means including an emotion engine for recognizing an emotion of a user and providing an appropriate response or support based thereon; means for providing at least one of said support and said moderation, and said response; A system including:

[1017] (Claim 2) The user's input includes at least one of a self-introduction, a suggestion, an opinion, a request, a request to a dialogue partner, and a selection or browsing of a product; 2. The system of claim 1.

[1018] (Claim 3) The generative AI analyzes the user's input and generates at least one of the presentation support, discussion moderation, and product guide in at least one form of audio, text, and image. 2. The system of claim 1. [Explanation of symbols]

[1019] 10, 210, 310, 410 Data Processing Systems 12 Data Processing Device 14 Smart Devices 214 Smart Glasses 314 Headset-type terminal 414 Robot< / url:> < / url:> < / url:> < / url:>

Claims

1. A social networking application that enables social interactions and / or business meetings in a virtual space, means for receiving user input and recognizing emotions of said user using an emotion engine, and using said user input and said user emotions to generate presentation support and / or discussion moderation in real time via a generative AI; means for presenting at least one of the generated support and the moderation; A system including:

2. The user's input includes at least one of a self-introduction, a suggestion, an opinion, a request, and a request to a dialogue partner; The system of claim 1 .

3. The generative AI analyzes the user's input and the user's sentiment and generates at least one of support for the presentation and moderation of the discussion in at least one form of voice, text, and image. A system according to claim 1 or claim 2.

4. The SNS application enables a shopping experience in a virtual store, The generating means further generates at least one of a product recommendation and a purchasing support by further using at least one of a question and a suggestion from the user in the virtual space; The presenting means further presents at least one of the generated product recommendation and the purchase support. A system according to claim 1 or claim 2.

Images